Risk of bias and magnitude of effect in orthodontic randomized controlled trials: a meta-epidemiological review

Summary

Aim:

To assess the risk of bias (RoB) in a subset of randomized controlled trials (RCTs) published in orthodontic journals using the Cochrane RoB tool and to identify associations between domain RoB assessment and treatment effect estimates.

Materials and methods:

Fifty consecutive issues of four major orthodontic journals were electronically searched to identify RCTs. The quality of the included studies was assessed using the Cochrane RoB tool, which involves seven domains rated as ‘low’, ‘unclear’ or ‘high’: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting, and other threats to internal validity. Estimates and confidence intervals (CIs) were recorded or calculated where possible for binary and continuous outcome measures. Meta-regression models were employed to assess the impact of RoB per domain on the magnitude of treatment effect.

Results:

One hundred and one eligible studies involving 128 pair-wise comparisons were retrieved. Blinding of outcome assessors and incomplete outcome data were frequently judged as ‘high’ for RoB both for studies with binary and continuous outcome (42.9 and 48.8 per cent, respectively). For binary outcomes, high RoB regarding random sequence generation [odds ratio (OR): 5.97, 95% CI: 2.03, 17.63, P-value: 0.002] and incomplete outcome data (OR: 4.07, 95% CI: 1.03, 16.15, P-value: 0.05) were more likely to provide exaggerated effect estimates.

Conclusions:

There is a need for improved clinical trial methodology and reporting, in order to avoid inflated associations and erroneous conclusions.

Introduction

The importance of randomized controlled trials (RCTs) in informing clinical practice and formulating clinical decision-making is widely recognized. The value of a clinical trial in guiding clinical decisions is dependent on a series of reporting characteristics, providing insight to the methodology and any potential risk of bias (RoB) (1). However, methodological and reporting deficiencies have been repeatedly exposed in RCTs within medicine, dentistry, and orthodontics (2–6).

Systematic reviews (SRs) and meta-analyses concerning the effectiveness of interventions combine the results from RCTs leading to a more precise summary estimate and ultimately a recommendation for practice. Consequently, RoB inherent in RCTs is likely to influence the quality of the evidence derived from SRs (7–9). A variety of tools have been used to assess the reporting/methodological quality or RoB within RCTs (10, 11). The Cochrane Collaboration’s RoB tool includes assessment for six types of bias: selection bias, performance bias, detection bias, attrition bias, reporting bias, and other bias within seven domains (11). Each type of bias may arise at different stages of the trial and corresponds to different domains. For example, in respect of selection bias the relevant domains are random number generation and allocation concealment.

Bias identified in RCTs stems from threats to the internal validity of the study which might lead to distorted treatment effects and may ultimately impact on treatment choices. The magnitude of the effect size and its precision is an important parameter, therefore, in treatment choices. Previous biomedical literature has exposed associations between study internal validity and effect sizes; however, the findings were not always consistent (12–14). A recent SR concluded that effect sizes are exaggerated by 11–13 per cent for inappropriate random number generation or allocation concealment and lack of blinding especially if outcomes were subjective (15). There is currently only one assessment of this possible association within oral health literature, whereby in periodontology research no association between allocation concealment or blinding of assessors and corresponding effect sizes was noted (16). Thus, the aim of the present cross-sectional survey was to assess the RoB of RCTs published in orthodontic literature using the Cochrane RoB tool and to assess potential associations between RoB domains and observed treatment effect estimates.

Materials and methods

RCTs published in major orthodontic journals were included in the present study. The contents of the most recent 50 issues of the American Journal of Orthodontics and Dentofacial Orthopedics (AJODO), the Angle Orthodontist (Angle), the European Journal of Orthodontics (EJO), and the Journal of Orthodontics (JO) were electronically searched up to November 2013 by two authors (DK and EL). Only RCTs were eligible for inclusion. Initially the title and abstract were screened; if randomization was apparent or the prospective nature of the study was verified, the full text was accessed to clarify trial design. Terminology such as ‘random allocation’, ‘random assignment’, ‘randomly divided’ or similar were chosen as indicators of a randomized design.

The quality of the RCTs included was assessed according to the Cochrane RoB tool, incorporating the following domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting and other potential threats to validity. RoB assessment may result in ‘low’, ‘unclear’ or ‘high’ RoB. Details concerning population characteristics (mean and standard deviation or number of events) and sample size per group were recorded. Estimates and confidence intervals (CIs) for continuous or dichotomous/time-to-event outcomes were recorded or calculated where possible.

Fifty percent of the papers were scored independently by two authors (DK and EL) to arrive to a consensus assessment, as part of a calibration procedure. The rest of the papers were divided between the investigators and scored independently. Discrepancies were resolved through consultation with a third author (NP).

To assess the impact of RoB on the estimates, a series of simple meta-regression models were undertaken for each domain: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, and other potential threats to validity. Continuous and binary data were analysed separately. For the analysis, only ‘high’ and ‘unclear’ RoB levels were combined converting each domain into a binary variable, as recommended also in the Cochrane Handbook (17). The dependent variable was the trial results presented on an odds ratio (OR) scale for binary outcomes or as standardized mean difference for continuous outcomes. The independent variable was the dichotomized bias score: low RoB and unclear/high RoB. Low RoB was used as the reference level.

Among the included studies, some multi-arm studies were identified; incorporation of multi-arm studies in a joint model was attempted but the results were imprecise. Consequently, we decided to ‘split’ the control group into two (for the 3-arm studies) or more groups with smaller sample size in order to include each pair-wise comparison separately without inappropriately inflating sample size. Particularly, in the binary dataset both the number of events and the total number of patients were separated, whereas in the continuous dataset only the total number of participants was split up keeping the means and standard deviations as reported (17).

The following confounding parameters were planned for inclusion in the meta-regression models: year of publication, recruitment country, availability of informed consent documentation, number of authors involved in the publication, the number of recruiting centres, the study design and whether a methodologist was involved in the analysis. The involvement of a methodologist was assessed either from reported affiliations and/or acknowledgements in the manuscripts. To prevent the risk of false positive conclusions due to the presence of many covariates, a Monte Carlo permutation test using 10 000 permutations was applied, in order to select the statistically significant covariates (18). In the aforementioned meta-regression models, statistical significance was inferred at a significance level of 5 per cent. All analyses were performed in Stata 13.0 (Stata Corp, College Station, Texas, USA).

Results

A total of 128 studies were identified. Of these, 24 (18.8 per cent) studied a binary outcome (i.e. event or non-event) and the remaining 104 involved a continuous outcome. Among the 24 studies with binary outcome, 3 (12.5 per cent) did not provide any outcome data. Similarly, among the 104 studies with continuous outcomes, 24 (23.1 per cent) did not provide any outcome data. Two trials with binary outcome studies evaluated three treatments, whereas 17 trials with continuous outcome studied three treatments and 1 trial studied five treatments. Overall, 101 RCTs were eligible for inclusion, with 119 comparisons (Supplementary Material online). The majority of studies were published after 2007 (77/101; 76.2 per cent), were single-centred (69/101; 68.3 per cent), and were of European authorship (62/101; 61.4 per cent). Informed consent was documented in 80 studies (79.2 per cent). A considerable amount of studies involved less than four authors (38/101; 37.6 per cent), whereas only in one in five studies a statistician for the statistical analysis was involved (21/101; 20.8 per cent). A breakdown to years of the included issues per journal revealed studies published in the AJODO within the years 2009–13, in the Angle and the EJO between 2006 and 2012 and in the JO from 2001 to 2013.

Binary outcome studies

As shown in Table 1, most of the studies with binary outcome were parallel-group (66.7 per cent), with the most frequently studied effect measure being the hazard ratio followed by the risk difference (38.1 and 28.6 per cent, respectively).

Table 1.

Distribution of several study characteristics, by outcome type (n = 101 studies).

Characteristic	Frequency (%)*
	Binary	Continuous
Study type
Parallel	14 (66.67)	67 (83.75)
Split mouth	7 (33.33)	8 (10.00)
Crossover	0	5 (6.25)
Effect estimate
Mean/median diff	NA	80 (100.00)
Risk ratio	4 (19.05)	NA
Odds ratio	3 (14.29)	NA
Hazard ratio	8 (38.10)	NA
Risk difference	6 (28.57)	NA
Total	21 (100)	80 (100)

Mean/median diff: mean or median difference; NA: not applicable.

*The results are presented as counts and row percentage in the parentheses by type of outcome.

The distribution of the RoB in the individual domains assessed with the Cochrane Collaboration’s ‘RoB’ tool is illustrated in Table 2. More than half trials presenting binary outcomes reported insufficient details pertaining to allocation concealment (66.7 per cent), blinding of participants and personnel (85.7 per cent), incomplete outcome data (57.1 per cent), and selective outcome reporting (90.5 per cent), and as a result, the judgments on the degree of bias were unclear. Only detection bias and other sources of bias were judged to be high risk (42.9 and 57.1 per cent, respectively). Finally, according to the reports random sequence generation was judged as appropriate in the majority of these trials (66.7 per cent).

Table 2.

Distribution of the risk of bias across type of outcome (n = 101 studies).

Source of bias	Binary*			Continuous*
	Low	Unclear	High	Low	Unclear	High
Random sequence generation	14 (66.67%)	5 (23.81%)	2 (9.52%)	46 (57.50%)	32 (40.00%)	2 (2.50%)
Allocation concealment	7 (33.33%)	14 (66.67%)	0 (0.0%)	20 (25.00%)	58 (72.50%)	2 (2.50%)
Blinding of participants and personnel	3 (14.29%)	18 (85.71%)	0 (0.0%)	10 (12.50%)	66 (82.50%)	4 (5.00%)
Blinding of outcome assessment	5 (23.81%)	7 (33.33%)	9 (42.86%)	34 (42.50%)	7 (8.75%)	39 (48.75%)
Incomplete outcome data	5 (23.81%)	12 (57.14%)	4 (19.05%)	22 (27.50%)	36 (45.00%)	22 (27.50%)
Selective outcome reporting	0 (0.0%)	19 (90.48%)	2 (9.52%)	5 (6.25%)	74 (92.50%)	1 (1.25%)
Other sources of bias	3 (14.29%)	6 (28.57%)	12 (57.14%)	11 (13.75%)	36 (45.00%)	33 (41.25%)

*The results are presented as counts and row percentage in the parentheses by type of outcome.

A Monte Carlo permutation test was employed to adjust the type I error due to employing a series of single covariate meta-regression models. Table 3 provides the P-value results for each single covariate meta-regression model after 10 000 permutations. Without type I error adjustment, there is no evidence of association for any covariate (minimum unadjusted P-value = 0.292). After accounting for multiplicity, all P-values increased considerably. The standard error of the calculated P-values was estimated at 0.005 indicating that 10 000 random permutations were enough to provide precise results.

Table 3.

Monte Carlo permutation results on P-value.

Model	Binary*		Continuous**
	Unadjusted	Adjusted	Unadjusted	Adjusted
Publication year	0.292	0.903	0.743	1.000
Country	0.313	0.922	0.114	0.562
Informed consent	0.807	1.000	0.056	0.338
Number of authors	0.843	1.000	0.483	0.993
Statistician	0.602	0.999	0.766	1.000
Number of centres	0.513	0.993	0.692	1.000
Type of study	0.178	0.747	0.695	1.000

*Largest Monte Carlo standard error of P-value equal to 0.005 for binary outcome.

**Largest Monte Carlo standard error of P-value equal to 0.005 for continuous outcome.

Results from a series of single covariate meta-regression models are given in Table 4; each covariate reflects a specific domain of the RoB tool. A study is more likely to provide larger treatment effect when there is unclear or high RoB regarding random sequence generation (OR: 5.97, 95% CI: 2.03, 17.63, P-value: 0.002) and incomplete outcome data (OR: 4.07, 95% CI: 1.03, 16.15, P-value: 0.05) and the findings are statistically significant. However, the results are based only on a set of 23 comparisons which may explain the wide CIs.

Table 4.

Single covariate meta-regression results (n = 119 comparisons). OR, odds ratio; CI, confidence interval; SMD, standardized mean difference; Ref, reference.

Covariate	Risk of bias	Binary			Continuous
		OR	95% CI	P-value	SMD	95% CI	P-value
Random sequence generation	Low	Ref			Ref
	Unclear/high	5.97	2.03 to 17.63	0.002	−0.26	−0.81 to 0.29	0.35
Allocation concealment	Low	Ref			Ref
	Unclear/high	2.54	0.68 to 9.44	0.17	0.04	−0.58 to 0.67	0.89
Blinding participants and personnel	Low	Ref			Ref
	Unclear/high	0.57	0.08 to 3.88	0.55	−0.005	−0.77 to 0.76	0.99
Blinding outcome assessment	Low	Ref			Ref
	Unclear/high	0.63	0.13 to 3.08	0.55	−0.002	−0.55 to 0.54	0.99
Incomplete outcome data	Low	Ref			Ref
	Unclear/high	4.07	1.03 to 16.15	0.05	0.05	−0.58 to 0.67	0.89
Selective outcome reporting*	Low	Ref			Ref
	Unclear/high	NA	NA	NA	−0.27	−1.25 to 0.70	0.58
Other sources of bias	Low	Ref			Ref
	Unclear/high	1.92	0.31 to 12.13	0.47	−0.31	−1.07 to 0.45	0.42

Statistically significant results are highlighted in bold and italic. NA, not applicable.

*Selective outcome reporting for binary outcomes was dropped by the model because of collinearity (all comparisons were rated as unclear/high risk of bias).

Continuous outcome studies

Similar to the studies with binary outcomes, the majority of studies with continuous outcome were parallel-group (83.8 per cent), (Table 1).

More than 70 per cent of the trials reported insufficient details pertaining to allocation concealment (72.5 per cent, Table 2), blinding of participants and personnel (82.5 per cent) and selective outcome reporting (92.5 per cent), whereas only blinding of outcome assessment and other sources of bias were judged to be at high RoB in a considerable number of the studies (48.8 and 41.3 per cent, respectively). For almost half of the studies attrition bias and other sources of bias were judged to be unclear. Only random sequence generation was judged to be of low RoB in the majority of these trials (57.5 per cent).

The Monte Carlo permutation results on P-value for each single covariate meta-regression model are outlined in Table 3. Without type I error adjustment, there is weak evidence that informed consent may have an impact on the study results (smaller P-value = 0.056). After accounting for multiplicity, all P-values increase considerably (smaller P-value = 0.338). Similar to studies with binary outcome, the standard error of the calculated P-values was estimated at 0.005 indicating that 10 000 random permutations were enough to provide precise results.

The results from a series of single covariate meta-regression models on the standardized mean difference scale are presented in Table 4. The first arm (active intervention/new intervention) is more likely to provide smaller treatment effect than the comparator (control/standard intervention) when there is high/unclear RoB regarding random sequence generation, blinding of participants and personnel, blinding of outcome assessment, selective outcome reporting, and other potential threats to study validity (SMD: −0.26, −0.005, −0.002, −0.27, and −0.31, respectively). However, all CIs include the zero value reflecting no difference between the arms for high/unclear and low RoB. In addition, the impact of blinding of personnel and blinding of outcome assessment on the treatment effects was almost negligible (−0.005 and −0.002, respectively).

Discussion

RoB assessment in a representative subset of RCTs published in major orthodontic journals resulted in judgment as ‘unclear’ for more than half of the studies in at least four of the RoB domains including allocation concealment, blinding of participants and personnel, incomplete outcome data, and selective outcome reporting. RoB was judged as ‘unclear’ when there was insufficient information available to permit judgment of ‘high’ or ‘low’ RoB. Blinding of participants or personnel involved in an orthodontic trial may be inapplicable due to the nature of the interventions provided. Inapplicability of blinding does not necessarily indicate low methodological quality; however, the potential RoB due to lack of blinding cannot be ignored. Moreover, blinding of outcome assessors was frequently judged to be of high RoB whereas random sequence generation was frequently judged to be of low RoB. These findings underscore not only the compromised internal validity of RCTs in orthodontic research, but also the apparent weaknesses in trial reporting which is not new to biomedical literature (5, 19–23). Suboptimal reporting and deficient adherence to reporting guidelines of medical and dental RCTs remains a problem, despite the widespread adoption of the Consolidated Standards for Reporting of Trials statement by journals and editorial policies (3, 5, 24).

Significant associations between treatment effects and RoB domains were identified solely in studies assessing binary outcomes in the present cross-section of RCTs. Reports with high/unclear RoB in random sequence generation and incomplete outcome data were more likely to provide an exaggerated treatment effect compared to those with low RoB scores. However, RCTs bearing results for continuous outcomes did not reveal any evidence between RoB and treatment effects. In a similar study undertaken in periodontology, trials from five Cochrane SRs were explored in terms of two domains from the Cochrane RoB tool, and neither allocation concealment nor examiner blinding could be associated with the magnitude of treatment outcomes (16). Conversely, evidence from biomedical literature in a breadth of clinical trials contributing to meta-analyses has raised awareness with regard to the influence of reported study design characteristics on the intervention effect estimates from RCTs (15). Among others, treatment effects of trials with inadequate or unclear random sequence generation, allocation concealment and ‘double blinding’ have appeared exaggerated by a factor of 7 to 13 per cent, and have been exposed as inflated by up to 15–17 per cent in a subgroup of studies addressing subjective outcomes. Evidence from pregnancy and childbirth research has revealed exaggerated ORs by as much as 41 per cent, when concealment of allocation was inadequate (13).

The effects of other sources of bias such as attrition bias or reporting bias in the results from clinical trials have not been previously assessed. An association of attrition bias and exaggerated treatment effects was recorded in the present study. Losses to follow up or incomplete accounting for dropouts may risk inflated effect sizes to either direction and erroneous study conclusions due to between-group discrepancies (25). This is of particular importance in the clinical context of orthodontics or other medical fields where treatment effectiveness is largely dependent on the cooperation/compliance of trial participants to the allocated intervention (19).

A subset of orthodontic journals was searched to provide a representative cross-section of orthodontic RCTs in the present study; these represent the major journals within the clinical field of orthodontics. While it is likely that some RCTs over the study period have not been identified, these journals are known to publish the highest frequency of RCTs (20) with adequate reporting profiles (4). No attempt was made to contact authors of included studies for further clarification with regard to study design and conduct which may have had an effect on our judgment on internal validity of the primary study. However, contact with authors is notorious for leading to post hoc conclusions that lack credibility (26). Moreover, in the analysis a decision was made to combine ‘high’ and ‘unclear’ RoB levels allowing each domain to be converted into a binary variable. While this approach may have resulted in loss of data, it has been recommended (17) with the majority of empirical studies undertaking this to facilitate comparison with studies having low RoB. In addition, we decided to convert multi-arm studies into two-arm studies by ‘splitting’ the control group into two or more groups with smaller. We recognize that employing such an approach may also result in some loss of data.

Conclusions

The RoB in a large subset of orthodontic clinical trials was often adjudged to be either high or unclear with particular shortcomings including blinding of outcome assessors. High RoB RCTs regarding random sequence generation and incomplete outcome data were liable to provide exaggerated treatment effects. Improved clinical trial methodology and greater clarity of research reporting within dental trials are required to allow quality assessment and better use of research findings. Active adoption of reporting guidelines by clinical trial methodologists and researchers, as well as greater editorial awareness and contribution to identify malpractices, will guarantee improved conduct and reporting of clinical trials and facilitate evidence-based decision-making.

Supplementary material

Supplementary material is available at European Journal of Orthodontics online.

Funding

No funding received for this research.