Reporting quality of the Delphi technique in reporting guidelines: a protocol for a systematic analysis of the EQUATOR Network Library

Abstract

Introduction

Reporting guidelines are important tools for improving the quality of medical research. The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network’s Library contains a comprehensive and up-to-date database of reporting guidelines relevant to health research. Only 31% of reporting guidelines published from 2010 to 2014 reported using the Delphi technique, and the reporting quality of the Delphi technique in reporting guidelines is unknown even though the use of the Delphi technique was recommended in the guidance for reporting guidelines. We will assess the quality reports of the Delphi technique or modified Delphi technique in reporting guidelines.

Methods and analysis

The present study is a systematic analysis of the EQUATOR Network Library. We will include all reporting guidelines in the EQUATOR Network that used the Delphi technique or modified Delphi technique, published since 1 January 2011 and registered in the EQUATOR on or before 31 May 2018. Our primary outcome is the reporting quality of the Delphi technique, measured by the quality score (each item) in the Delphi technique. We will also examine the relationship between the reporting quality score (each item) of the Delphi technique and year of publication, number of authors, impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published in open access policy.

Ethics and dissemination

Ethics approval will not be applicable for this study. This protocol has been registered in the University Hospital Medical Information Network Clinical Trials Registry. We will publish our findings in a peer-reviewed journal and may also present them at conferences.

Trial registration number

UMIN000032685.

Strengths and limitations of this study.

• This is the first study to investigate the factors associated with each item in the quality score of the Delphi technique in reporting guidelines.

• We will perform an independent assessment for reporting guidelines.

• Applicability will be limited because the analyses investigating the quality score of the Delphi technique will include only reporting guidelines registered in the Enhancing the QUAlity and Transparency Of health Research Network Library and will not include other, possibly low-quality, reporting guidelines.

• We will not investigate whether the reporting quality of the Delphi technique in reporting guidelines affects the reporting quality of individual studies referring to these reporting guidelines.

Introduction

Critical appraisal and effective dissemination of research is hindered by insufficient reporting of a study’s methodology and findings.¹ Additionally, insufficient reporting impedes the applicability and misrepresents results used by patients and practitioners.¹ To improve the quality of research, experts have developed reporting guidelines.² Reporting guidelines are important tools for improving the quality of medical research.² The number of reporting guidelines in the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network’s Library has increased. The Network’s site contains a comprehensive database of reporting guidelines for health research.³ There are almost 400 reporting guidelines in the EQUATOR Network.⁴

The three main formal consensus methods used in the health field are the Delphi technique, Nominal Group Technique (NGT) and consensus development conferences.⁵ The Delphi technique is widely applied in order to obtain input from a group of experts.^6–8 The method is characterised by anonymity between members with structured feedback.^{6 9} Participants may regulate their initial ratings based on feedback from the group in a number of accompanying loops.^{6 9} The Delphi technique consists of any type of self-administered questionnaire with no meetings, whereas the modified Delphi technique consists of the use of a self-administered questionnaire, combined with a physical meeting of the experts, to discuss the results or rate the indicators.^{10 11} The Delphi method and the NGT are associated with obtaining a group decision from a suite of experts.⁵ On the other hand, consensus development conferences have the further aim of preparing a public forum for discussion.⁵

The NGT and consensus development conferences have limitations. The NGT has a smaller number of participants than does the Delphi technique, with the potential for dominant participants to inordinately affect the group.¹² Consensus development conferences depend on implicit methods (qualitative or simple quantitative methods such as majority voting), while the Delphi method and the NGT practice explicit, statistical integration in order to combine the judgements of experts.⁵

A previous study suggests the reporting of consensus methods in reporting guidelines should be improved.² Exercising the Delphi technique in reporting guideline development is important because of its potential to add participants in the reporting guideline development process³ and reduce variance of opinion within the group between the two rounds,¹³ in addition to having higher between-group reliability ratings than NGT.¹⁴ Therefore, the technique will improve the quality of guidelines.^{3 13 14} Only 13% of reporting guidelines published from 2010 to 2014 included the use of the Delphi technique² even though the use of the Delphi consensus method was recommended in the guidance for reporting guidelines.³ The study, however, did not assess the reporting quality of the Delphi technique among the reporting guidelines, and this aspect remains unknown.²

Objectives

We will assess the quality of reports of the Delphi technique or the modified Delphi technique in reporting guidelines.

Methods and analysis

Types of studies to be included

We will include reporting guidelines in the EQUATOR Network that used the Delphi technique or modified Delphi technique, were published since 1 January 2011, and registered in the EQUATOR Network on or before 31 May 2018. We will select reporting guidelines published on and after January 2011 because a previous study that recommends using the Delphi technique in reporting guideline development was published in February 2010.³ We will only include the most recent versions of reporting guidelines in order to eliminate duplication in the guidelines. We will check for the recent versions of guidelines by screening for the data item ‘previous versions of this guideline/guideline history’ in each of the reporting guidelines in the EQUATOR Network. We will also check whether meetings were held between Delphi rounds. We will regard the reporting guidelines as ‘Delphi’ when meetings were not held between Delphi rounds and as ‘modified Delphi’ when meetings were held between Delphi rounds.

Search methods

We will search the EQUATOR Network Library after 31 May 2018. The search will be subjected to English language restrictions.

Study selection

One of three authors (MB, YT and YK) will assess the eligibility based on a full-text review of reporting guidelines identified by the initial search and another author (MB, YT or YK) will confirm the contents. We will search using the terms ‘Delphi’ or ‘modified Delphi’ in the text and check whether the Delphi technique or modified Delphi technique was used. We will resolve disagreements by discussion between the authors (MB, YT and YK).

Data extraction and assessment

For each of the included reported guidelines, one author (MB) will extract the Delphi technique information and another author (YT or YK) will confirm the contents. We will resolve disagreements by discussion between the authors (MB, YT and YK).

We will extract data for the four quality score items in the Delphi technique.¹⁵ A recent study proposed a quality score in the Delphi technique after assessing quality in reports of the Delphi technique, published in 2000–2009.¹⁵ The four items are as follows:

1. Were criteria for participants reproducible? (Yes or No): The method to select and exclude participants is stated. Number and type of participant subgroups (eg, patients, generalists and experts) are needed.

2. Was the number of rounds to be performed stated? (Yes or No): We will categorise this as ‘Yes’ when the number of rounds is stated in the methods or results. We will categorise this as ‘Yes’ when researchers report the actual number of Delphi rounds in the results.

3. Were criteria for dropping items clear? (Yes, No or Not applicable): The prespecified criteria for dropping items at each round are reported.

4. Are stopping criteria, other than rounds, specified? (Yes or No): The prespecified criteria for stopping the Delphi process, other than a statement of the number of rounds, are reported. For example, the prespecified criteria are related to the consensus or stability of responses.

We will score ‘Yes’ and ‘Not applicable’ as 1 and score ‘No’ as 0, as done in the previous study.¹⁵ Two authors (MB and YT or YK) will independently assess the score for each reporting guideline. We will adopt the median of the quality score in the earliest full publication in case of multiple publications with the same guideline.

We will extract the following factors potentially associated with the reporting quality of the Delphi technique: year of publication, number of authors, impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published according to open access policy. One author (MB) will extract data for the impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published according to open access policy and another author (YT or YK) will confirm the contents. We will resolve disagreements by discussion between the authors (MB, YT and YK). YK will perform web scraping from PubMed and Google Scholar with Python V.3.6 (Python Software Foundation) and collect data for the year of publication, and number of authors. We will record the number of editors as the number of authors if some of the reporting guidelines are books or handbooks. We will define funding as the receipt of any supporting funds for conducting the research. We will regard sources of funding as ‘industry’ when funds are received from an industry (for example, pharmaceutical companies).¹⁶ We will regard sources of funding as ‘non-industry’ when the funds are from government, other academic or non-profit organisations.¹⁶ We will define multiple publications as publications in multiple journals.³ We will regard translated versions of original guidelines as multiple publications if the translated versions are published in journals. We will check for multiple publications by screening for the data item ‘full bibliographic reference’ of each of the reporting guidelines in the EQUATOR Network. The year of publication will be the oldest year when there are multiple publications for reporting guidelines. We will extract the impact factors determined by the 2018 Journal Citation Reports. An impact factor will be a mean value of multiple publications when there are multiple publications for reporting guidelines. We will deem that the guidelines are published according to open access policy when at least one full-text of the guidelines is available on the web (whether the full-text is downloadable will not be considered). We will check the official sites of 15 reporting guidelines, which are highlighted as ‘Reporting guidelines for main study types’ in the EQUATOR Network and collect additional information about year of publication, impact factor and multiple publications, as well as whether the guidelines are published according to open access policy. We will contact the corresponding authors of the reporting guidelines for additional information if necessary.

Our primary outcome of interest will be the reporting quality of the Delphi technique (each item) in the reporting guidelines. We will also examine the relationship between the reporting quality score (each item) of the Delphi technique and year of publication, number of authors, impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published according to open access policy.

Patient and public involvement

We evolved the study protocol without patient participation. This study will use only public data without patient recruitment. We will spread the results via web sites and social network services to patients and the public.

Sample size

The sample size calculation for a primary outcome will not be applicable because the sample size of the reporting guidelines is determined beforehand.

Data analysis

We will report the frequency of the reporting quality score (each item) of the Delphi technique as the descriptive analysis. We will explore correlations, using Fisher’s exact test, between each item of the quality score (Yes, No or Not applicable) and the following possible predictors, defined a priori: year of publication, number of authors, impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published according to open access policy. We will conduct prespecified sensitivity analyses by repeating the analysis and excluding additional data from the official websites of the 15 reporting guidelines.

All p values will be two-sided. P values will be considered statistically significant if less than 0.05. We will not perform an adjustment of the alpha level for multiple comparisons because our study is an exploratory study. Alpha level adjustment is not essential in exploratory analyses.¹⁷ All statistical analyses will be performed with EZR (Saitama Medical Center, Jichi Medical University, Saitama, Japan), which is a graphical user interface for R (The R Foundation for Statistical Computing, Vienna, Austria).¹⁸ More precisely, it is a modified version of R commander designed to add statistical functions frequently used in biostatistics.

Ethics and dissemination

The planned completion date of the present study is 31 December 2019. We will publish our findings in a peer-reviewed journal and may also present them at conferences.

Discussion

This is the first study to investigate the factors associated with each item in the quality score of the Delphi technique in reporting guidelines.

This study will reveal the quality of reports of the Delphi technique in reporting guidelines. Problems with the quality of reports of the Delphi technique will be detected. Therefore, this study will be potentially used to improve the quality of reports of the Delphi technique in reporting guidelines. Improved reporting guidelines will result in better health research.¹⁹ Therefore, this study has the potential to alter the quality of reporting guidelines and provide useful resources in developing reporting guidelines. This study may also result in new recommendations about the quality of reports of the Delphi technique in the development of reporting guidelines.

There are several expected limitations for this study. First, the applicability will be limited because the analyses investigating the quality score of the Delphi technique include only reporting guidelines registered in the EQUATOR Network Library. The Library contains a comprehensive database of reporting guidelines relevant to health research.³ However, other, possibly low-quality, reporting guidelines may be missing. Second, we will not investigate whether the reporting quality of the Delphi technique in reporting guidelines affects the reporting quality of individual studies referring to these reporting guidelines. However, reporting guidelines created in good order may affect the reporting quality of individual studies because a type of reporting guideline, the Preferred Reporting Items for Systematic Reviews and Meta-Analysis, has increased the quality of reporting in individual systematic reviews and meta-analyses.²⁰ Third, there is no information on the reliability and validity of the quality score. However, we will use the score because we contend it represents a necessary first step for assessing the reporting quality of the Delphi technique in the absence of other measures. Fourth, this study is based on an exploratory analysis and will provide information rather than recommendations.

In conclusion, this study will provide a comprehensive investigation about the reporting quality of the Delphi technique in reporting guidelines using the EQUATOR Network. The expected findings will show the needs and key factors for improving the reporting quality of the Delphi technique in reporting guidelines.