Assessment of the quality of reporting of randomized clinical trials in paediatric dentistry: A comparative systematic review

Abstract

Objectives

This systematic review aimed to determine the improvement in quality of the reporting of randomized clinical trials (RCTs) in paediatric dentistry. The quality of reporting during the period 2014–2015 was compared with the quality of reporting during 1985–2006.

Methods

This systematic review compared the scientific quality of RCTs in paediatric dentistry published in five paediatric dentistry journals during the defined periods. The Consolidated Standards for Reporting Trials (CONSORT) checklist of 2010 was used to evaluate the quality of reporting. The inter-reviewers’ agreement was assessed by calculating the kappa score, and disagreements between reviewers were resolved by consequent discussion. The p values and percentages were used to test for significant differences between the two reviews (1985–2006 and 2014–2015).

Results

A total of 40 articles were included. Although the quality of reporting showed considerable heterogeneity, the overall quality of reporting by RCTs was satisfactory and had improved over the years.

Conclusions

Using CONSORT checklist, this study showed general improvement in the quality of reporting of RCTs published in pediatric dentistry journals in all article's sections.

Introduction

‘Evidence based medicine is defined as the integration of best research evidence with clinical expertise and patient values. Although the evidence-based approach was originally developed in medicine, its principles can be applied to all fields in healthcare including dentistry. In dentistry, the approach is known as evidence based dentistry (EBD)’.¹ ‘The American dental association has defined evidence based dentistry as an approach to oral health care that requires the judicious integration of systematic assessments of clinically relevant scientific evidence, relating to the patient's oral and medical condition and history, together with the dentist's clinical expertise and the patient's treatment needs and preferences'.²

The evidence of research is critical to EBD because it permits experts to choose which mediations are the most effectual.³ As the RCT is high up in the pyramid of evidence, obviously the design of these studies will have an impact on their quality.⁴ As well as design, the reporting of these types of studies will also have a bearing on the quality of outcomes.⁴ Given the importance of reporting evidence, particularly data from RCT, ‘a checklist has been developed for authors to follow before publishing their research to improve the quality of RCTs reporting. This checklist is called the Consolidated Standards of Reporting Trials (CONSORT) statement, which was developed by an international group of clinical trialists, statisticians, epidemiologists, and biomedical editors. It was first published in 1996’⁵ and updated in 2010.⁵ ‘In 2008, the CONSORT Group developed an extension to the original statement that addressed methodological issues specific to trials of nonpharmacological treatments (NPTs), such as surgery, rehabilitation, or psychotherapy’.⁶

In medicine, the CONSORT checklist is used frequently to evaluate reporting of RCTs, while there is less evidence in dentistry. We were one of the first to do this by examining the quality of studies in paediatric dentistry through the analysis of published RCTs from 1985 to 2006 using the CONSORT checklist¹; this previous study indicated that the general quality of clinical trials reporting was poor and inadequate for researchers to accurately evaluate the strength of the trials.¹ Therefore, the aim of this review was to calculate if there has been progress in the condition of RCTs reporting ten years after the publication of the first review. Two steps were completed: Evaluation of the quality of reporting of Randomized Clinical Trials (RCTs) published in paediatric dental journals from 2014 to 2015 inclusive, and comparing the conclusions with the findings of the first review which examined RCTs published in paediatric dental journals from 1985 to 2006.

Materials and Methods

A systematic review using the PICOS method (Participants, Interventions, Comparison, Outcome, and Study) of design was performed as follows: The participants consisted of five paediatric dentistry journals; the intervention was the evaluation of the quality of reporting of RCTs published from January 2014 to December 2015; the comparison was our previous assessment which evaluated the quality of reporting of RCTs published in 1985–2006¹; the outcomes were comparisons of the results of both reviews; and the study design was a systematic review. The CONSORT 2010 checklist was used to assess the quality of RCTs’ reporting for this systematic review; the reporting of this article followed the PRISMA 2009 checklist as shown in Table 1.

Table 1.

Comparison of compliance of articles against the CONSORT checklist.

Article Section and Topic	Item	The percentage of reporting (1985–2006)	The percentage of reporting (2014–2015)
Title and abstract	1	71	92.5
Background	2	98.8	100
Participants	3	90.2	89.5
	4	79.2	95
Interventions	5	96.0	100
Objectives	6	90.2	90
Outcomes	7	87.3	91.4
	8	38.7	47.5
Sample size	9	4.6	55
	10	0.6	10
Randomisation generation	11	28	60
	12	5.8	15
Randomisation concealment	13	5.2	40
Randomisation implementation	14	5.8	40
	15	6.4	37.5
	16	9.2	42.5
Blinding	17	34	40
	18	27.7	37.5
	19	58	58.5
	20	11.6	15
Statistic	21	83.8	92.5
	22	25.4	2.5
Participant flow	23	45.1	52.5
	24	13.9	17.5
Recruitment	25	32.9	55
Baseline data	26	60.1	77.5
Analysis	27	61.3	87.5
	28	1.2	20
Outcomes	29	57.2	62.5
Ancillary analysis	30	21.4	2.5
Adverse events	31	57.8	35
Discussion	32	97.1	98.5
Generalisability	33	98.3	98.8
Overall evidence	34	96.5	97

The term ‘first review’ will be used to report the 1985–2006 results, and the term ‘this review’ will be used to report the results of the current article (2014–2015 inclusive).

The first stage of this study was an electronic screening of the participating journals to establish the RCTs that could be involved in the evaluation based on the following inclusion criteria: the studies were RCTs; the trials were reported in English; the participants were children of age 16 years or under; articles were those published between January 2014 and December 2015 in one of the following five paediatric dental journals:

• 1.International Journal of Pediatric Dentistry (IJPD)
• 2.Journal of Dentistry for Children (JDFC)
• 3.Pediatric Dentistry (PD)
• 4.European Archives of Pediatric Dentistry (EAPD)
• 5.The Journal of Clinical Pediatric Dentistry (JCPD)

The initial search process was performed by the author (AA). Any RCTs performed in vitro, or on animals or adults, were excluded. Other reasons for exclusion were if the study was a case report, a review, or an observational study. In addition, all cross-sectional, cohort, longitudinal, case–control, or survey studies reported as observational studies were not included in the assessment. The screening of titles, abstracts, and full-text articles was completed twice by both authors at the UCL Eastman Dental Institute in London, UK.

Assessments of compliance with the CONSORT 2010 checklist of included trials were undertaken twice by both authors. The CONSORT 2010 checklist has 25 questions, which were transformed into an operational list of 34 questions, consisting of the same questions in the original list but, instead of multiple sections under one question, numbered separately. Each item on the list was scored as Yes, No, or Not Applicable.

The average kappa score for compliance of the articles was used to assess the inter-reviewer's agreement of the 34 questions; disagreements between reviewers were resolved by consequent discussion. Chi-square tests and percentages were used to compare proportions of articles that complied with the 34 questions in the CONSORT checklist between those that were published in the first review and those published in this review; and only the main questions of the CONSORT checklist were reported (questions 1, 11, 13, 17, 18, 19) as they had been used in the first review. Results were gathered and analyzed by Statistical Package for Social Science-SPSS 24.0 (SPSS Inc., Chicago Ill, USA).

Results

A total of 567 published articles were identified in the initial screening phase, most of which were not RCTs; 525 were excluded because they were either reviews, editorials, case reports, observational studies, or short communications. Only 56 (9.8%) of the 567 published articles were RCTs, of which 40 were eligible for inclusion. The remaining 16 articles were not included as they were either RCTs performed on adults (4 articles), or in vitro (8 articles), while four articles were incorrectly published as RCT where the random/randomization was mentioned in the abstract but RCT methods were not used; three of these were published in the IJPD, and one article was published in the EAPD. The CONSORT flowchart of articles throughout the study is shown in Figure 1.

Figure 1.

The CONSORT flowchart of articles throughout the study.

Over the period of two years (2014 and 2015) the EAPD had the highest number of published RCTs, 12 out of 40 (30%) of all included journals; followed by PD and JCPD, both of which published 11 RCTs; the IJPD published 5 RCTs; and the JDFC published only one RCT.

Chi-square test and percentages of reported trials were used to assess compliance with the main questions on the CONSORT checklist (questions 1, 11, 13, 17, 18, 19) used in the first review. There was considerable variety between articles on the separate parts of the CONSORT checklist. Question 1 specifically asked if the words ‘random allocation’, ‘randomized’, or ‘randomly assigned’ were mentioned in the title or abstract. The result revealed a significant improvement from 71% in the first review to 92.5% in this review (P = 0.005). Question 11 questioned if the method used to generate the random allocation sequence was reported. The results revealed that there was a significant difference between the first review (28%) and this review (60%), (P = 0.04). Question 13 was about the concealment of randomization allocation and the result revealed that there was significant improvement between the first and current reviews: 5.8% and 40%, respectively (P = 0.008).

Question 17 considered if authors were reporting whether the participants were blinded to group assignment. The result showed that there was no significant difference (P = 0.13). However, there was a slight increase in the reporting percentage from 34% in the first review to 40% in this review.

Question 18 asked if the authors had reported whether those administering the intervention were blinded to group assignment. The result showed that there was no significant difference (P = 0.08). However, there was a slight increase in the reporting percentage from 27.7% in the first review to 37.5% in this review. Question 19 questioned if authors reported whether assessors were blinded to group assignment. The result showed that there was no significant difference (P = 1.00) with a slight increase in reporting from 58% to 58.5%.

A comparison of the percentage of fulfilment of articles versus the CONSORT checklist that were published in 1985–2006 and 2014–2015 (inclusive) is shown in Table 2.

Table 2.

PRISMA checklist.

Section/topic	#	Checklist item	Reported on page #
TITLE
Title	1	Identify the report as a systematic review, meta-analysis, or both.	1
ABSTRACT
Structured summary	2	Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number.	2
INTRODUCTION
Rationale	3	Describe the rationale for the review in the context of what is already known.	3
Objectives	4	Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS).	4
METHODS
Protocol and registration	5	Indicate if a review protocol exists, if and where it can be accessed (e.g. Web address) and, if available, provide registration information including registration number.	N/A
Eligibility criteria	6	Specify study characteristics (e.g. PICOS, length of follow-up) and report characteristics (e.g. years considered, language, publication status) used as criteria for eligibility, giving rationale.	5
Information sources	7	Describe all information sources in the search (e.g. databases with dates of coverage, contact with study authors to identify additional studies) and date last searched.	5
Search	8	Present full electronic search strategy for at least one database, including any limits used, such that it can be repeated.	5
Study selection	9	State the process for selecting studies (i.e. screening, eligibility, included in systematic review and, if applicable, included in the meta-analysis).	5
Data collection process	10	Describe method of data extraction from reports (e.g. piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators.	6
Data items	11	List and define all variables for which data were ought (e.g. PICOS, funding sources) and any assumptions and simplifications made.	5
Risk of bias in individual studies	12	Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level) and how this information is to be used in any data synthesis.	6
Summary measures	13	State the principal summary measures (e.g. risk ratio, difference in means).	N/A (the study was not a meta-analysis)
Synthesis of results	14	Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g. I392) for each meta-analysis.	N/A (the study was not a meta-analysis)
Risk of bias across studies	15	Specify any assessment of risk of bias that may affect the cumulative evidence (e.g. publication bias, selective reporting within studies).	N/A (the study was not a meta-analysis)
Additional analyses	16	Describe methods of additional analyses if done (e.g. sensitivity or subgroup analyses, meta-regression), indicating which were pre-specified.	N/A (the study was not a meta-analysis)
RESULTS
Study selection	17	Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.	7,15
Study characteristics	18	For each study, present characteristics for which data were extracted (e.g. study size, PICOS, follow-up period) and provide citations.	The number of articles included was high, citations are available on request.
Risk of bias within studies	19	Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12).	N/A (the study was not a meta-analysis)
Results of individual studies	20	For all outcomes considered (benefits or harms), present for each study: (a) simple summary data for each intervention group, and (b) effect estimates and confidence intervals (ideally with a forest plot).	7
Synthesis of results	21	Present results of each completed meta-analysis, including confidence intervals and measures of consistency.	N/A (the study was not a meta-analysis)
Risk of bias across studies	22	Present results of any assessment of risk of bias across studies (see item 15).	N/A (the study was not a meta-analysis)
Additional analysis	23	Give results of additional analyses, if done (e.g. sensitivity or subgroup analyses, meta-regression [see item 16]).	N/A (the study was not a meta-analysis)
DISCUSSION
Summary of evidence	24	Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g. healthcare providers, users, and policymakers).	8,9
Limitations	25	Discuss limitations at study and outcome levels (e.g. risk of bias), and at review level (e.g. incomplete retrieval of identified research, reporting bias).	10
Conclusions	26	Provide a general interpretation of the results in the context of other evidence and the implications for future research.	10,11
FUNDING
Funding	27	Describe sources of funding for the systematic review and other support (e.g. supply of data); the role of funders for the systematic review.	11

Disputes between reviewers were solved by subsequent discussion. The inter-reviewers’ agreement was calculated by measuring the average kappa scores for the 34 items of CONSORT, and the kappa score was 0.88.

Discussion

Apart from comparing results of this study with the first review,¹ the decision to include the period 2014–2015 in this review was based on a previous study published in the EAPD that also included only two years for the review.⁷ The CONSORT 2010 checklist was used as the CONSORT 2017 extended checklist had not yet been released and is only for nonpharmacological trials, while we included all types of trials.

The trial report is the only information available to evaluate the quality of a trial. It is important that this is evaluated as poor quality can lead to an overestimation of the impact of mediations by 30–41%.⁸ Good research should be of a high quality to back authors' decisions, whether they draw a positive or negative conclusion. When comparing the quality of RCTs reporting between the first review (1985–2006) and this review (2014–2015), there was a noticeable enhancement in the RCTs reporting generally. In the introduction sections, there was a significant improvement in precise reporting (P = 0.005) of whether the words ‘random allocation’, ‘randomized’, or ‘randomly assigned’ were mentioned in the title or abstract (71% in the first review,¹ and 92.5% in this review). This percentage was only at 45% before the publication of the CONSORT checklist.¹

In the methods sections, there were dramatic increases in the quality of reporting between the first review and the current one (this review), especially in the method used to produce the sequence of random allocation; the reporting for this was 28% in the first review¹ and 60% in this review (P = 0.04). There was a significant improvement in the randomization allocation concealment between the first and this review, 5.8% and 40%, respectively (P = 0.008), while conversely, a previous study found that the method and the trial conduct were inadequately reported.⁹

On the other hand, there was no significant difference between the two reviews in the following aspects: reporting whether participants were blinded to group assignment; reporting whether authors administering the intervention were blinded to group assignment; and if authors reported whether assessors were blinded to group assignment. The reporting percentages of the above items in the current review had marginal increases.

A total of 40 RCTs were included. Most of the published articles in paediatric dentistry journals were not RCTs; most were reviews, observational studies, or case reports. Even though the percentage of published RCTs was only 9.8%, this is still promising as it represents a large increase over the 3% from the previous review.¹ However, more RCT studies are still required in the field of paediatric dentistry.

In the first review, it was found that ‘the quality of RCTs published in pediatric dentistry journals for the period of 1985–2006 was generally poor; even after the publication of the CONSORT checklist, and the quality of reporting was only improved negligibly in titles, abstracts and discussion sections’.¹ Another study stated that ‘the general quality of RCTs reporting in pediatric dentistry journals was inadequate’⁷; the same result was found in the field of dental public health.¹⁰ On the other hand, this review found that the quality of RCTs reporting in paediatric dentistry journals was generally improved, especially in the methods sections as well as in titles, abstracts, and discussion sections; parallel results were found in implant dentistry.¹¹

We were the first to raise awareness to adapt the CONSORT checklist, making it one of the requirements for publication in the paediatric dentistry field. In 2006, ‘letters were sent to journals that were included in the review, to determine their status regarding the adoption of the CONSORT checklist, only two of the five journals responded to the letter, the European Archives of Paediatric Dentistry, which stated that they have not yet adopted the CONSORT checklist, but they were considering adopting it in volume 9 of 2008; and the International Journal of Paediatric Dentistry who were considering adopting it in 2008’.¹

Even though it has been more than 20 years since the initial publication of CONSORT, its potential benefits are not being fully utilized by either researchers or publishers. The reasons for not adopting the CONSORT guidelines are unclear, but both researchers and publishers should demand vociferously that it is adopted so as to enhance the quality of reporting of trials in paediatric dentistry.

This study has a potential limitation in that a broader period of publication years could be considered in future studies. Apparently, including RCTs published in languages other than English, and in dental journals other than those relevant to paediatric dentistry would increase the number of articles, however, the decision to restrict this search to paediatric dentistry journals published in English was to make this review practicable. However, including articles published in other languages other than English, and in dental journals other than those relevant to paediatric dentistry is recommended for future studies. Furthermore, researchers might be more stimulated to adhere to the CONSORT checklist when reporting a trial, and all paediatric dental journals should adopt the CONSORT checklist and make it an essential requirement for any article considered for publication. The protocol of this study could be used for future studies whether in paediatric dentistry or other specialities.

Conclusions

Considering the comparison of compliance of articles that were published in 1985–2006 and in 2014–2015 (inclusive) against the CONSORT checklist, the quality of reporting of RCTs published in paediatric dentistry journals has improved generally in all sections.

Source of funding

I declare that this research was conducted at the UCL Eastman Dental Institute, London and funded by a British aerospace postdoctoral research award of 5000 GBP.

Conflict of interest

There is no conflict of interest.

Ethical approval

I declare that this study does not require an ethical approval.

Authors contributions

PFA conceived and designed the study. AAA conducted research, provided research materials, and collected and organized data. PFA and AAA analyzed and interpreted data. AAA wrote initial and final draft of article, and provided logistic support. All authors have critically reviewed and approved the final draft and are responsible for the content and similarity index of the manuscript.