Quality of randomised controlled trials in dentistry

Abstract

Randomised controlled trials (RCTs) are regarded as the best study designs to test the efficacy of medical and dental intervention. Many reports, however, have shown that at the moment the quality of dental RCT reports is still poor, and further efforts to improve it are necessary. It has been suggested that trials that are not well designed provide biased estimates of the treatment effects and that a journal’s impact factor is not related to the quality of RCTs published. For these reasons, before trusting RCT reports, a careful assessment of the study selected is needed. Randomisation, blinding, allocation concealment, drop outs analysis are essential quality components of RCTs. Many systems for RCTs quality assessment are available. In this report the concept of quality of RCTs will be critically evaluated and the most commonly used instruments available for quality assessment of RCTs in dental research will be reviewed.

Evidence Based Medicine (EBM) is the procedure by which the outcomes of the scientific research are applied to daily clinical practice¹. Currently, this concept is popular among dentists and for some years has been known as evidence based dentistry. Randomised controlled trials (RCTs) represent the gold standard for testing the efficacy of medical interventions2., 3. and are generally regarded as the most reliable source of ‘scientific evidence’ in healthcare⁴.

The reliability of the results coming from clinical trials strongly depends on a sound methodology and proper management of research. Indeed, biases resulting from poorly conducted trials could mislead clinicians and address questionable clinical decisions. In recent years the need for predictable and effective dental treatments has lead clinicians to look for therapeutic approaches validated by means of experimental trials but, unfortunately, the quality of many published medical and dental RCTs is variable⁵. Also, no correlation between the quality of the trials and the journal impact factor has been noted⁵. This may cause incorrect evaluations by readers who may assess quality of trials by the impact factor of the journal in which they are published.

Moreover, basing therapeutic approaches on trials that are not well designed might be a risk for patients and, from a legal perspective for dentists also. For this reason, it is important that clinicians are enabled to recognise good-quality evidence and to use only those studies in support of their daily practice. In this report the concept of quality of RCTs will be critically evaluated and the most commonly used instruments available for quality assessment of RCTs in dental research will be described.

RANDOMISED CONTROLLED TRIALS

Clinical trials are scientific experiments involving subjects or/and patients designed to find out the most effective therapy for treatment of a disease². Generally the results coming from clinical trials help to set-up a treatment protocol for populations who are affected by the same condition of the research sample or to choose among the different therapies available². The efficacy of the treatment tested in a trial is determined by comparing the findings obtained from a treated group, on which the treatment protocol object of the research is tested, with those obtained from a control group, which is treated with a different treatment protocol or is not treated at all.

Without using a control group, the efficacy of the treatment tested cannot be assessed adequately being generally overestimated. The allocation of the experimental sample to both groups should be randomised. Randomisation prevents selection biases during group allocation and allows that control and treated subjects will have similar characteristics at baseline.

Another important aspect of reliable clinical trials is blinding. This is the process by which people involved in research or patients (single blinding), or both (double blinding) are not aware of one of more aspects of the research process. Single or double blinding should be considered when planning clinical trials because they can drastically increase the internal validity of a trial and reduce the risk of examiner’s or patient’s biases.

Quality of randomised controlled trials: concepts and tools

The assessment of the methodological quality of a trial is essential as quality can considerably influence the scientific outcomes and the clinical interpretation of the research. The definition of the construct ‘quality’ is the first needed but difficult step for evaluating the quality of a trial. Quality has been defined as the likelihood of the trial design to generate unbiased results, sufficiently valid that they can be applied in clinical practice³. The quality of a trial quantifies the likelihood that the experimental outcomes are valid estimates of the truth.

Thus far, some issues have been considered when assessing the quality of RCTs⁶:

• •The clinical relevance of the research question
• •The internal validity of the trial (the degree to which the trial design, conduct and statistical analysis have minimised biases during comparison of the interventions)
• •The external validity (the validity of the results in real-world settings)
• •Correct data analysis and presentation
• •The ethical implications of the intervention.

When talking about quality, two different issues should be considered. The methodological quality of the trial itself that is related to its internal and external validity, and the reporting quality, that concerns the reporting of the research design, conduct, and data analysis. Since the only instrument available to readers for assessing the quality of a trial is the published manuscript, RCT reports must provide the most accurate information about the trial design, conduct and data analysis⁷.

Quality scales and checklists may be used for assessing RCT quality. Scales provide a quantitative assessment of the quality of a trial by giving a quality numerical score, while checklists do not provide a quantitative score. Quality scales and checklists consist of lists of methodological issues that should be included when performing RCTs. Generally, the tools available include methodological issues that have been selected by experts in the field, such as statisticians, or include items that have been considered as ‘accepted criteria’ in textbooks.

However, the most common issues considered of these tools are randomisation, allocation concealment, blinding, withdrawals and drop-out analysis. These issues have been reported to strongly influence the outcomes of a trial and the efficacy of the treatments tested. For instance, a clinically and statistically 30–50% exaggeration of treatment efficacy in lower quality trials has been shown⁸ and larger estimates of treatment effects have been reported in trials in which allocation concealment was unclear or inadequate9., 10.. Therefore quality assessment tools such as scales and checklists are needed before assessing published RCTs.

In 2001 the number of quality scales and checklists available in literature was estimated to be between fifty and sixty³. An extensive review of scales can be found in Moher et al.¹¹ and Olivo et al.¹². The scales available differ for the number of items incorporated and for the importance given to each of them. Although these tools provide an overall summary score of the quality of a trial, they should be first tested for their validity and reliability. In many cases the scales available are the result of modifications of primary quality scales used for other disciplines and cannot be considered valid and reliable unless they are tested.

Among the RCTs quality assessment tools, the Jadad Scale¹³ is the most commonly used, which provides an easy score for a quantitative evaluation of the quality of a trial (Figure 1). Randomisation, blinding, and the account for withdrawals and drop-outs are considered within this scale. The quality assessment of research reports is easy and quick. For each item a unit score is added or detracted from an initial given score (0). Score range is −5 to 5. Low quality trials result in a score lower than three.

Figure 1.

Jadad Scale. Modified from Jadad et al⁷.

The inter-rater reliability of the Jadad scale has been reported to be high in many studies, ranging from 0.48 to 1.00 (Intraclass correlation coefficients) and 0.37 to 0.89 (Kappa values)12., 14., 15., 16., 17., 18., 19., 20.. The test retest reliability was shown to be 0.98²¹. Nevertheless, the high simplicity of the scale and the large influence of blinding on the overall summary score might be questionable since in some disciplines blinding is not always applicable because of the nature of the clinical interventions. For these reasons a reassessment of the validity of the Jadad Scale has been recently claimed¹².

The Delphi list²² (Figure 2) is a checklist devised in 1998 by statisticians and experts in RCTs. With this tool, a consensus among experts about a core of items for quality assessment of RCTs was achieved. The Delphi method represents the first step toward a standard in RCT quality assessment. After three sessions, a set of items was chosen to be included in the Delphi list. Inter-observer reliability was reported to range from 0.54²³ to 0.85²⁴. Although several efforts were made in order to reach a consensus, this instrument has not been as popular as the Jadad scale.

Figure 2.

Delphi list. Modified from Verhagen et al²².

The PEDro scale²⁵ (Figure 3) has been developed for the Physiotherapy Evidence Database (PEDro). This instrument comprises a set of 11 items, and consists of a modification of the primary 9-item Delphi list, with two items more (items 8 and 10). The inter-rater reliability has been reported to range from −0.61 to 0.8826., 27.. The PEDro scale was developed in order to help consultants of the PEDro database to quickly identify RCTs with internal validity and with sufficient statistical information among the uploaded RCTs. The scoring of PEDro scale is significantly influenced by blinding since different types of blinding are scored by three items. Therefore the quality score of RCTs related to disciplines in which blinding cannot be completely achieved is underestimated with this tool.

Figure 3.

PEDro Scale. Modified from PEDro Physiotherapy Evidence Database: http://www.pedro.org.au/²⁵

The question of which scale to use is a difficult one since a gold standard is lacking and the choice is not reliable28., 29.. The availability of various scales implies that the quantitative evaluation of the quality of a trial is different according to the system used. Thus, the use of scales developed with a consensus should be largely promoted. However, it is important to note that only the research report (i.e. the manuscript) is the object of the quality assessment. Hence, a trial designed with many biases, but well reported, can get a high score. Conversely, well designed trials poorly reported may receive a low score. As a consequence of this, systems have been developed to increase the quality of reporting trials. The introduction of CONSORT Statement³⁰ in publication process has improved the reporting quality of RCTs, allowing more accurate and reproducible assessments³¹.

The CONSORT (Consolidation of the Standards of Reporting trials) statement was developed in 1996 to improve the standard of quality of RCTs and currently offers a standard for preparing reports of trials. A 25-item checklist and a flowchart were attached to this document and uploaded on-line with the intent of enhancing reporting of RCTs. The checklist items focus on trial design while the flow chart guides the progress of all participants through the research phases.

Currently a growing number of journals support the CONSORT statement and invite authors to follow its guidelines before submitting papers. The use of CONSORT has been shown to significantly increase the quality of reporting of RCTs in different medical fields in the recent past. Currently, modified versions of the CONSORT checklist have been used for quality assessment of trials in different medical fields. However, better reporting of RCTs is still claimed in nursing journals³², orthopaedic surgery³³ and plastic surgery³⁴. Better reporting of clinical trials in the five leading Chinese medical journals has been also requested³⁵.

QUALITY OF RANDOMISED CONTROLLED TRIALS IN DENTISTRY

So far, the quality of randomised controlled trials has been assessed in few dental disciplines and has been object of only few studies. In a report concerning RCTs in paediatric dentistry³⁶, the quality of trials published between 1985 and 1997, and between 1998 and 2006 were compared to assess the influence of CONSORT introduction on quality of published RCTs. On average, the reporting quality of the published reports was poor. Also in a study by Dumbrigue et al.³⁷ the quality of implant dentistry RCTs was assessed. Using a scheme developed through the Cochrane Collaboration, two examiners assessed the quality of 67 articles and concluded that accounting of randomisation procedures and blinding for most implant RCTs was inadequate. The quality of prosthodontic RCTs was evaluated by Jokstad et al.³⁸. Randomisation and procedures for concealment allocation were not described in 70% of the articles. The random allocation sequence was not clear in 94% of papers. Blinding was not reported in 72% of cases.

The quality of published RCTs has been examined also in relation to Journal Impact Factor® in different areas of dental research by Sjogren and Halling⁵. The Jadad scale was used primary. The randomisation process was judged to be adequate in only 22% of cases, and blinding was reported in just 28% of papers. The accounting for withdrawals/dropouts was present in 35% of cases. Finally, a high quality score was reached by only one third of the RCTs examined and those on oral surgery had significantly higher median quality scores, compared with all other areas grouped together. No correlation between the journal impact factor and the quality of RCTs was found. This result has also been confirmed by Barbui et al.³⁹.

So far, the reporting of orthodontic clinical trials has been judged to be insufficient to help readers to assess the validity of RCTs and the median quality of published trials has been estimated to be poor. In a survey by Harrison⁴⁰, the accounting of withdrawals and allocation concealment was mostly evaded by research reports. Only 2.6% of the trials analysed presented double blinding. It must be noted that the reporting of blinding of trial personnel is frequently missing in clinical trials⁴¹ and a major concern arises when talking about blinding in certain disciplines such as orthodontics. Indeed, since orthodontic appliances have commonly different shapes or configurations, clinicians are often aware of which intervention patients are receiving, especially when testing multibracket fixed appliances. In some cases, for instance while performing trials concerning functional removable appliance, a second clinician blinded should take the measurements after patient has removed the appliance and the clinical chart has been hidden.

CONCLUSIONS

Since it is proved that RCTs provide the most reliable and valid estimates of the efficacy of the treatments tested, clinicians should focus their daily clinical practice on RCTs outcomes. Before trusting the outcomes resulting from published RCTs it is necessary to assess their quality. The number of dental RCTs has increased enormously during the last decade but, unfortunately, the quality of the trials published is variable and further efforts to promote high quality studies are needed.

Clinicians should be able to recognise high quality trials among those published every year by using the systems available. At the present time, the tools available for quality assessment have improved the quality of RCTs published and have contributed to the standardisation process in RCT quality assessment. Assessing quality of research can be quite challenging, particularly for readers who are unacquainted with research methodology. Further to the quality tools reviewed in this report, it is always important to evaluate an article by thorough and careful reading as well as by critical appraisal of all of its parts, especially the materials and methods sections. This may be difficult and time-consuming but remains probably the best way to make an in-depth evaluation of the quality of clinical research.