Reporting Quality of Randomized Controlled Trials in Restless Legs Syndrome Based on the CONSORT Statement

Abstract

Background

Randomized controlled trials (RCTs) are the cornerstone of modern medical research, and their reporting may not always be optimal. The Consolidated Standards of Reporting Trials (CONSORT) statement is an evidence-based means to improve the quality of RCTs’ reporting by providing a checklist of recommended items.

The aim of this study was to assess the reporting quality of published RCTs on the restless legs syndrome (RLS), based on a checklist arising from the CONSORT statement.

Methods

Medical electronic databases were searched for RCTs involving patients with RLS. Inclusion criteria were follows: articles must have been published in English and RLS patients must have been randomized into a minimum of two treatment cohorts of different medicinal orientations. CONSORT-recommended items were marked as “reported” or “not reported,” and an overall CONSORT compliance metric was calculated. Comparisons among different time periods, CONSORT-endorsing and non-endorsing, and different levels of impact factor journals were made.

Results

Fifty-four eligible trials, published in 21 different scientific journals, were found. The average CONSORT compliance score was 56.6% (23.68–84.21%). CONSORT-endorsing journals had a mean CONSORT compliance of 58.47%, whereas non-endorsing journals had a mean CONSORT compliance of 50.4%. The median CONSORT compliance for articles published in low- (IF<2), medium- (IF 2-7), and high-ranked (IF>7) journals was 52.63, 56.57, and 59.21%, respectively. Only 14 of the 38 CONSORT items (36.8%) were reported in >75% of the articles.

Discussion

This study shows that the reporting of RLS-related RCTs is suboptimal, regardless of the time period, the quality of the publishing journal, and the endorsing or non-endorsing of the CONSORT statement.

Introduction

In the era of evidence-based medicine, randomized controlled trials (RCTs) are the highest ranked means of research and are considered as “the most powerful tool in modern clinical research.”¹ Even if recent research methods, such as meta-analyses and umbrella meta-analyses, provide more accurate data, the quality of RCTs remains central, as they represent the structural element of the aforementioned research methodologies. The process of randomization in clinical trials guarantees that the significant findings in the group comparisons regarding the matter under examination can be accredited to the intervention and not to other confounding factors,² thus minimizing bias. However, available information in biomedical journals may be concealing a wide range of biases, such as publication, selection, and funding biases.^3,4,5 It is crucial for readers and other researchers to be aware of the quality standards of RCTs so as to properly assess their strong and weak points.^6,7

It is clear that evidence-based healthcare relies, among others, on the quality assessment of RCTs. The reporting of RCTs, as a type of medical literature, has in general got a certain form and adopted a basic structure that is more or less used by the majority of authors. However, several strategic methodological characteristics may be overlooked in published reports and, therefore, the assessment of reporting quality may be used as a surrogate for trial quality.⁸ A trial with inadequate reporting quality may be misclassified, while an unclear and inaccurate description may be attempting to conceal erroneous methods.^7,9,10

Time and time again, evidence has stated that the reporting quality of RCTs is not optimal.^11,12 Addressing these concerns, at the end of the 20th century, an international group of trial methodology experts, including statisticians, epidemiologists, science journal editors, and clinical trialists, created the Consolidated Standards of Reporting Trials (CONSORT) Statement, which was last updated in March 2010.This statement is an evidence-based set of recommendations, including a checklist and a flow diagram, for the reporting of RCTs; its aim is to promote the thorough and transparent reporting of trials while simultaneously assisting in their critical evaluation and understanding.¹¹ Recommendations concern the title, introduction, methods, results, and other information that should be reported and cover virtually every aspect of conducting an RCT according to Good Clinical Practice. Separate items cover the methodology of the trials (randomization methods, blinding, etc.), the statistical analysis, the outcomes, as well as other information like the funding source and the online access of the study protocol. CONSORT stresses the importance of integrality, preciseness, and transparency of reporting, which are, all in all, indicative of the actual trial plan and conduct.¹³

A number of publications have examined the reporting quality of RCTs in several subspecialties of medicine.^{14,15,16,17,18,19,20,21} In a recent study by our team, concerning multiple sclerosis, the mean CONSORT compliance was found to be considerably low, since only 30% of the studies reported more than 75% of the CONSORT items.²¹ Similar studies in polycystic ovary syndrome, urology, and other surgical specialties also found the CONSORT compliance to be suboptimal,^16,20 reflecting the intersectionality of the issue in the medical field.

Restless legs syndrome (RLS) is a relatively common neurological disease,^22,23 which can be primary or secondary.^{24, 25, 26} The unpleasant sensation of RLS and the subsequent sleep disturbance have a significant socioeconomic impact on patients.^{27, 28} Following the ongoing series of reporting quality analyses in neurological diseases, in the present study, we analyzed the reporting quality of RCTs involving patients with RLS (primary or secondary), using the items of the revised CONSORT 2010 statement checklist.¹³ The period covered by this report extends from 1998 to 2017.

Methods

Data sources and search strategies

We searched the PubMed database for RCTs on patients with RLS published between the January 1, 1998 and December 31, 2017. As a search criterion, the phrase “Restless Legs Syndrome” or “RLS” or “Willis–Ekbom disease” was used. We used the “Randomized Controlled Trial” type of article, “English” language, and “Humans” for species as filters.

Eligibility of studies

Trials were eligible for inclusion if the participants had been randomly assigned to at least two medicinal treatment arms (including iron and vitamin supplementation) and included patients with RLS regardless of the etiology (primary or secondary). Reports of trials without RLS-relevant symptoms as end points, dose comparison studies, those with non-medicinal regimens, and small pilot studies, alongside any article with information resulting from a previously conducted trial (such as post hoc or sub-group analysis and various sub-studies), were excluded. Crossover studies were also excluded due to their specific design.²⁹

Reporting assessment tool

The CONSORT 2010 checklist consists of 25 items, which, alongside their sub-items, reach a total number of 37 items. This checklist is recommended to be used by authors during the reporting of RCTs to promote thorough work and transparency in their presentation. Alongside the explanation and elaboration document, CONSORT provides instructions for every section of an RCT article (title, introduction, methods, results, and funding) and covers every aspect of a well-conducted clinical trial (Supplementary File 2). We have included an additional item (No 13), whether an article included a participant flow diagram or not, in order to pertain to the authors’ (strong) recommendation of using one. The CONSORT explanation and elaboration document (available at the CONSORT website) was used to guide our evaluation process. Out of the total 54 eligible trials, 28 were published before 2010 and 26 were published after 2010, with 2010 being the year when the CONSORT Statement was revised. We used the revised CONSORT version for all the extracted articles. The full CONSORT checklist can be accessed online at http://www.consort-statement.org.

Evaluation – analysis

During the evaluation process, each author reviewed the selected articles one by one and assigned a positive (“yes”) or a negative (“no”) response to each CONSORT item, according to whether it was reported or not. Additionally, the following procedures were followed: (1) all of the checklist items were searched for into the published trials in terms of whether they were reported, and not if they were actually performed during the trial. In cases where a methodology followed by the trials’ authors was insinuated in the results or other sources, but there was no lucid reference in the article (or the supplementary files), the CONSORT item was marked as “non-reported.” For example, if a trial stated on its title that it is a “double-blinded” trial but there was no clear reference in the methodology section as for how it was blinded, the respective items were checked as “non-reported.” (2) In cases where it was not obvious whether the criterion of reporting was met, after consensus among all authors, the item was assigned a negative mark (non-reported), since they may conceal important information. (3) In cases where a procedure of the trial was not mentioned in the main manuscript of the trial but there was a reference to it in a supplementary file which addressed this information, the procedure was considered as adequately reported. This rule was not applied to item 8a, where the CONSORT Explanation and Elaboration Document specifically requires that “information on the process of randomization is included in the body of the main article and not as a separate supplementary file; where it can be missed by the reader.”³⁰ Evaluation was made by two independent authors (D.R. and E.D.), and discrepancies were resolved by consensus among all authors. The articles were divided into three groups depending on the time of publication: from 1998 to 2004, from 2005 to 2009, and from 2010 to 2017. We also categorized the reported items into five groups as follows: (1) Title/Abstract and Introduction, (2) Methods, (3) Results, (4) Discussion, and (5) Other information.

The basic quality-of-reporting metric was the “CONSORT compliance,” meaning the percentage of the 38 CONSORT items that each article addressed. We calculated the greater than 75% compliance with the CONSORT statement items, as compliance with the CONSORT items to the extent of greater than 75% was considered an adequate cutoff in a number of other studies.^14,31 This metric was considered as a measure of each article’s reporting quality. It must be noted at this point that items 3b (changes to methods), 6b (changes to trial outcomes), 7b (interim analyses and stopping guidelines), and 14b (why the trial had to be ended or terminated) are items that were not applicable in most of the trials. Item 17b (effect sizes of binary outcomes), though an important item in RCTs in general, may not be applicable to RLS trials where the most commonly used outcomes are categorical (e.g., IRLS SG score and polysomnographic data); hence, it is also justifiably under-reported. By removing the five aforementioned items, a total of 86.6% compliance was considered to be the maximum that an article could reach.

The comparison of the >75% compliance among the different time period groups was made using the Pearson chi-square statistic for a 2×3 table. We calculated the percentage of items per group (title and abstract, methods, results, discussion, and other) that was reported in at least 75% of the articles for the total and individual time periods. This metric is an indicator of which CONSORT items were adequately reported or under-reported by the articles in comparison to the compliance, which is a measure of each article’s reporting quality. We further calculated the median CONSORT compliance of the articles published in journals with a current impact factor greater than 7, between 2 and 7, and lower than 2, and additionally performed a Kruskal–Wallis non-parametric test to compare the three groups. Journal impact factor was accessed through the Thomson Reuters Journal Citation Reports website,^32,33 and all statistical analyses were made using the IBM SPSS v.21 package.

Results

The evaluation process was carried out in four steps, as can be seen in the search flow chart (Figure 1). Initial search for entries meeting the set criteria returned 307 related articles, whose titles were scrutinized. One hundred and eighty articles were excluded due to irrelevance, use of non-medicinal intervention (behavioral treatment, exercise, herbal), or not referring to randomized trials. The abstracts of the remaining 127 articles were reviewed and an additional 52 articles were excluded for the same reasons. The remaining 75 articles were retrieved in full text, 21 of which were found ineligible for reasons explained before, and finally 54 articles were included in the study. A list of these 54 RCTs, which included a total of 10,427 randomized patients, can be found in the Supplementary file.

Figure 1.

Systematic Review Flowchart.

Of the total 54 eligible trials, 9 were published between 1998 and 2004, 19 between 2005 and 2009, and 26 between 2010 and 2017, when the revised CONSORT version had been made available. The drugs in comparison against placebo or another active component were gabapentin (8 studies), cabergolide (2 studies), bupropion (1 study), iron (7 studies), oxycodone (1 study), pergolide (2 studies), pramipexole (12 studies), pregabaline (3 studies), ropinirole (9 studies), rotigotine (6 studies), sumanirole (1 study), vitamins (1 study), and xp13512 (1 study).

Twenty-one different scientific journals hosted the included articles. Nine of them are currently CONSORT-endorsing, corresponding to 76% (41) of the articles. The mean CONSORT compliance of articles published in these journals reached 58.47%. The remaining 13 articles, published in 12 non-endorsing journals, had a mean CONSORT compliance of 50%. The median CONSORT compliance scores of articles published in low (IF<2), medium (IF 2-7), and high-ranked (IF >7) journals were 52.63, 56.57, and 59.21%, respectively. These percentages were not found to be significantly different (Kruskal-Wallis (K-W) p = 0.272).The eligible RCTs were relatively small, with an average count of randomized patients of 196. Fifteen of them (27.7 %) randomized fewer than 100 patients.

Table 1 shows the percentage of articles that reported each individual item divided into publishing period groups. The numbers and percentages of CONSORT items reported by >75% of the articles by time period and by group are shown in Table 2.

Table 1.

Compliance with the CONSORT Checklist per Item and Time Period

Data Item	Combined 1998–2016 (n = 54)	1998–2004 (n = 9)	2005–2009 (n = 19)	2010–2016 (n = 26)
Abstract/Title
1a	0.70	0.44	0.74	0.77
1b	0.81	0.78	0.79	0.85
Introduction
2a	1.00	1.00	1.00	1.00
2b	0.98	1.00	1.00	0.96
Methods
3a	0.78	0.56	0.84	0.81
3b	0.04	0.11	0.00	0.04
4a	0.94	0.89	0.89	1.00
4b	0.37	0.56	0.42	0.27
5	0.74	0.89	0.68	0.73
6a	0.69	0.44	0.79	0.69
6b	0.02	0.11	0.00	0.00
7a	0.69	0.44	0.58	0.85
7b	0.04	0.00	0.05	0.04
8a	0.46	0.33	0.53	0.46
8b	0.50	0.44	0.58	0.46
9	0.30	0.22	0.37	0.27
10	0.26	0.11	0.21	0.35
11a	0.39	0.44	0.37	0.38
11b	0.33	0.44	0.32	0.31
12a	0.96	1.00	1.00	0.92
12b	0.50	0.44	0.63	0.42
Results
13	0.70	0.44	0.68	0.81
13a	0.85	0.67	0.89	0.88
13b	0.78	0.56	0.74	0.88
14a	0.44	0.33	0.37	0.54
14b	0.02	0.00	0.05	0.00
15	0.81	0.89	0.79	0.81
16	0.54	0.11	0.53	0.69
17a	0.94	0.89	0.95	0.96
17b	0.13	0.11	0.16	0.12
18	0.41	0.22	0.47	0.42
19	0.87	0.78	0.89	0.88
Discussion
20	0.76	0.78	0.68	0.81
21	0.63	0.33	0.74	0.65
22	0.93	0.78	0.95	0.96
Other information
23	0.39	0.11	0.26	0.58
24	0.00	0.00	0.00	0.00
25	0.78	0.56	0.79	0.85

Table 2.

>75% Compliance with the COSNORT Checklist per Group of Items and Time Period Abbreviations: RCTs, randomized controlled trials.

Period/Checklist Items	1998–2016 (54 RCTs) N(%)	1998–2004 (9 RCTs) N(%)	2005–2009 (19 RCTs) N(%)	2010–2016 (26 RCTs) N(%)
Overall (38)	14 (36.8)	11 (28.9)	13 (34.2)	17 (44.7)
Title/abstract – intro (4)	3 (75)	3 (75)	3 (75)	4 (100)
Methods (17)	3 (17.6)	3 (17.6)	4 (23.5)	4 (23.5)
Results (11)	5 (45.5)	3 (27.3)	4 (36.4)	6 (54.5)
Discussion (3)	2 (66.6)	2 (66.6)	1 (33.3)	2 (66.6)
Other information (3)	1 (33.3)	0 (0)	1 (33.3)	1 (33.3)

Abbreviations: RCTs, randomized controlled trials.

During the whole examined period of 1998–2016, 14 items (36.8%) were found reported in >75% of the articles, whereas the respective numbers for the 5-year periods were 11 (28.9%) in 1998–2004, 13 (34.2%) in 2005–2009, and 17 (44.7%) in 2010–2016, showing a very small, insignificant increase in the reporting of CONSORT items (p = 0.344).

The mean CONSORT compliance score was 56.5% (23.68–84.21%). The RCTs that covered more than 75% of the CONOSRT items (by time period) were as follows: 5 (9.25%); 1998–2004: 0; 2005–2009: 3(15.79%); 2010–2016: 2 (7.69%).

Discussion

The conclusion that this study seems to reach is that the reporting of RLS-related RCTs is far from optimal. Furthermore, it provides a general view of RLS RCTs and its methodological components.

The number and size of RLS RCTs is smaller than that of other neurological diseases. We included 54 RCT articles referring to 10,427 randomized patients. In comparison to a recent similar study carried out by our group on the subject of multiple sclerosis RCTs, we found 103 studies regarding 42,031 patients²¹ for about the same time period (2000–2015). This is also made evident by the mean number of patients randomized in these 53 studies, which was less than 200 (193).

Only 14 of the 38 checklist items (36.8%) were addressed in 75% or more of the studies published between 1998 and 2016. There was a slight increase in checklist item reporting between the three time periods. Some of the underreported items were –in most cases – non-applicable (3b: important changes to methods; 6b: changes to trial outcomes; 7b: interim analyses and stopping guidelines). Other items underreported, though, contain important methodological information regarding randomization (9: allocation concealment method – 30%; 10: implementation – 26%). The allocation concealment method (concealment of the allocated intervention at the time of enrollment) and implementation (who generated the allocation sequence, who enrolled participants, and who assigned participants to trial groups) is an important piece of methodological information since trials in which the allocation sequence had been inadequately or unclearly reported yielded larger estimates of treatment effects than trials in which authors adequately reported allocation concealment.^13,32,34 Item 24 (where the full trial protocol can be accessed, if available) was the most underreported item (0%). The overall CONSORT compliance was generally found to be poor as well, with the mean compliance calculated to be 56.6%, which is considerably low and, sadly, not showing signs of improvement over time. When considering the five usually not applicable items in RLS RCTs, an adjusted overall CONSORT compliance would be 71%, which is similar with the one found in our study regarding multiple sclerosis (average CONSORT compliance 68.2%). This finding, though, should be regarded with some degree of skepticism since not all studies included non-applicable items.

There are 585 journals currently endorsing CONSORT. In our study, 9 out of 21 journals that published RLS RCTs were CONSORT endorsers, and those articles’ compliance was larger – though insignificantly – than their non-endorsing counterparts (58.27% vs. 50.4%). At the same time, high impact factor journals had a better reporting, but it still did not reach the significance threshold.

The results of this study must be interpreted with caution and some points need to be addressed. First, we used the 2010 revised CONSORT checklist for all RCTs regardless of their publication date. Furthermore, the allocation of a negative or positive response on the used checklist has not always been clear and straightforward, making it susceptible to subjectivity. In addition, most of the CONSORT items have a title alongside a large paragraph of explanation and information that should be used while reporting an RCT. The CONSORT checklist was made to help researchers report studies in a more thorough and detailed manner rather than being used as an evaluation tool. Consequently, the evaluation of reporting is a more critical and delicate procedure.

Taking into consideration the aforementioned limitations and the data presented here, it is our feeling that RLS-related RCTs are in general badly reported, regardless of the time period, the quality of the publishing journal, and its CONSORT endorsement. To the extent that reporting reflects study design and conducting, along with the fact that RLS RCTs are relatively small and few in number, while also regarding many different medicinal products, this study raises important concerns. RLS is a disease whose management is troublesome and tricky, and the related research seems to have many methodological pitfalls; hence, designing, performing, and reporting RCTs all need to be as detailed and accurate as possible.