Abstract
Objective
To determine if adequacy of randomisation and allocation concealment is associated with changes in effect sizes (ES) when comparing physical therapy (PT) trials with and without these methodological characteristics.
Design
Meta-epidemiological study.
Participants
A random sample of randomised controlled trials (RCTs) included in meta-analyses in the PT discipline were identified.
Intervention
Data extraction including assessments of random sequence generation and allocation concealment was conducted independently by two reviewers. To determine the association between sequence generation, and allocation concealment and ES, a two-level analysis was conducted using a meta-meta-analytic approach.
Primary and secondary outcome measures
association between random sequence generation and allocation concealment and ES in PT trials.
Results
393 trials included in 43 meta-analyses, analysing 44 622 patients contributed to this study. Adequate random sequence generation and appropriate allocation concealment were accomplished in only 39.7% and 11.5% of PT trials, respectively. Although trials with inappropriate allocation concealment tended to have an overestimate treatment effect when compared with trials with adequate concealment of allocation, the difference was non-statistically significant (ES=0.12; 95% CI −0.06 to 0.30). When pooling our results with those of Nuesch et al, we obtained a pooled statistically significant value (ES=0.14; 95% CI 0.02 to 0.26). There was no difference in ES in trials with appropriate or inappropriate random sequence generation (ES=0.02; 95% CI −0.12 to 0.15).
Conclusions
Our results suggest that when evaluating risk of bias of primary RCTs in PT area, systematic reviewers and clinicians implementing research into practice should pay attention to these biases since they could exaggerate treatment effects. Systematic reviewers should perform sensitivity analysis including trials with low risk of bias in these domains as primary analysis and/or in combination with less restrictive analyses. Authors and editors should make sure that allocation concealment and random sequence generation are properly reported in trial reports.
Keywords: allocation concealment, sequence generation, physical therapy, meta-epidemiological, risk of bias
Strengths and limitations of this study.
To the best of our knowledge, this is the first meta-epidemiological study using continuous outcomes in allied health disciplines such as physical therapy (PT).
This large meta-epidemiological study, which was based on 43 Cochrane reviews, 393 trials and 44 622 patients makes an important contribution to the field.
This study provides with novel evidence regarding the association between adequacy of randomisation and allocation concealment and treatment estimates in PT.
We restricted our analysis to Cochrane systematic reviews in PT and results might not be applicable to all Cochrane reviews or other reviews conducted in other areas of research.
For determining the association between random sequence generation and allocation concealment, we limited our analysis to trials describing a true control group, or placebo intervention, reducing our statistical power.
Introduction
Randomisation and allocation concealment have been extensively investigated in the medical literature as key methodological characteristics of randomised controlled trials (RCTs) in medical research.1–3 Allocation concealment is the most commonly evaluated quality characteristic in reviews of RCTs due to its crucial role as a key marker of internal validity of RCT.4 Moreover, it is firmly established the conventional wisdom that adequate randomisation sequence generation is an essential part of a valid clinical trial.5
There has been an extensive body of empirical research looking at the influence of random sequence generation and allocation concealment on treatment effect estimates of health RCTs. Several meta-epidemiological studies investigating the association between trial characteristics and treatment effects have found an association between inadequate randomisation and/or allocation concealment and an overestimation of treatment effects.1–3 6–9 For example, inadequate random sequence generation can overestimate treatment effects by 12% (Ratio of ORs (ROR) 0.88, 95% CI 0.79 to 0.99).10 Inadequate allocation concealment can overestimate treatment effects on average by 18%.3 8 11 In addition, Pildal et al12 found that 2/3 of the conclusions from meta-analyses were no longer supported if only trials with adequate allocation concealment were included and that 69% of meta-analyses lost statistical significance when trials with unclear or inadequate allocation concealment were excluded.12 Over-estimates or underestimates of treatment effects can lead to biased or inaccurate results and conclusions in systematic reviews and meta-analyses.3 12–15 These factors can ultimately have repercussions on decision-making and quality of patient care since different assessments could lead to different decisions for clinical practice.
Most of the empirical evidence regarding the relationship between trial components, and specifically random sequence generation and allocation concealment, and treatment effect estimates comes from RCTs in medicine and is based mainly on dichotomous outcomes.3 8 11 No such studies using continuous outcomes have been conducted in other health areas such as the allied health professions, including physical therapy (PT). Furthermore, although it has been reported that the quality of reporting of PT trials has improved over time,16 a finer analysis of types of random sequence generation and allocation concealment in PT trials is lacking.
Therefore, it is necessary to determine the extent to which sequence generation and allocation concealment affect treatment effect estimates in PT trials to provide accurate results to the PT clinical community. This information is urgently needed to develop guidelines for designing, conducting and implementing PT trials as well as providing clear benchmarks to assess the quality and/or risk of bias of PT trials in systematic reviews and meta-analyses and ultimately the strength of evidence for decision-making in PT. Therefore, our research questions were: (1) are random sequence generation and allocation concealment adequately used and reported in RCTs of PT; (2) do random sequence generation and allocation concealment have an effect on estimates of treatment in PT trials; and (3) do effects sizes on PT trials differ depending on some characteristics of the meta-analyses analysed such as magnitude of the effect size, meta-analysis heterogeneity, type of outcome (subjective or objective) and whether the meta-analysis involved the musculoskeletal area.
Method
Design
Meta-epidemiological approach.
Study selection
A random sample of RCTs included in meta-analyses in the PT discipline were identified by searching the Cochrane Database of Systematic Reviews from January 2005 to 25 May 2011 on PT interventions. The search strategy can be found elsewhere and can be provided on request.17 18 Meta-analyses were included if they met the following eligibility criteria: (1) the meta-analysis included at least three RCTs comparing at least two interventions, with at least one of the interventions being part of PT scope of practice according to the World Confederation for Physical Therapy (WCPT);19 and (2) the main outcome or the outcome of the meta-analysis with the largest number of trials conducted in the review was continuous.
Assessment of random sequence generation and allocation concealment domains
Assessments of random sequence generation and allocation concealment domains were performed by two independent reviewers following the Cochrane collaboration guidelines.20
Random sequence generation assessment
In addition to the general evaluation of adequacy of random sequence generation, different methods of random sequence generation were determined. We grouped these methods into four categories as follows: Category 1 included trials where random sequence generation was unclear or not reported; category 2 included trials that had adequate randomisation (eg, use of a computer software, random number table and minimisation); category 3 included trials using acceptable methods of randomisation, but less efficient than the previous category (eg, drawing lots, envelopes, shuffling cards, throwing a dice); category 4 involved trials using inappropriate methods of sequence generation (eg, date of birth, day of admission, hospital record number). Categorisation into one of these four categories was conducted in duplicate. To facilitate comparison with previous meta-analyses, these four categories were combined to create two main groups: an “adequate sequence generation” group (combining categories 2 and 3) and an “inadequate or unclear sequence generation” group (combining categories 1 and 4).
Allocation concealment assessment
We divided the methods of allocation concealment used in the trials into the following categories: Category 1 comprised trials that used any type of central randomisation (eg, a remote telephone service or a central office); category 2 comprised trials that used sequentially numbered, opaque and sealed envelopes; category 3 comprised trials that used sealed envelopes without reporting any further details; and category 4 comprised trials where allocation was clearly not hidden (eg, being based on an open list, odd or even days of the week, participant's birth date or the team on duty at enrolment); and category 5 comprised trials were concealment of allocation was not reported or unclear. Categorisation into one of these five categories was conducted by two independent assessors. To facilitate comparison with previous meta-analyses, these five categories were combined to create two main groups: an “adequate concealment” group (combining categories 1 and 2) and an “inadequate or unclear concealment” group (combining categories 3, 4 and 5).
Data extraction of treatment estimates and trial characteristics
Two independent reviewers extracted specific data (eg, type of interventions, type of outcomes (ie, objective, subjective), PT area) for all trials included in the meta-analyses as well as data on means, SDs and sample sizes. The primary outcome chosen for the analysis was the main outcome of interest reported in the review or determined from the meta-analysis that contained the largest number of trials in the review. Details on the reviewers’ panel and training process can be found elsewhere.17 18
Data analysis
Data on sequence generation and allocation concealment were analysed descriptively based on the categories described previously. In order to determine whether random sequence generation and allocation concealment domains affect treatment effect estimates, a two-level analysis was conducted using a meta-meta-analytic approach with a random-effects model to allow for within and between meta-analyses heterogeneity as suggested by Sterne.21
The first level analysis (within meta-analysis) was as follows: we derived effect sizes (ES) for each trial by dividing the between-group difference in mean values by the pooled SD.22 A negative ES indicates a beneficial effect of the experimental intervention. If some required data were unavailable, we used approximations as previously described.23 The data from each trial were obtained from the meta-analyses included in our study. We followed the classification used in the Cochrane reviews to classify the treatment arms as the experimental treatment of interest or as the control group. In the case of studies appearing in more than one review, the study was only considered once in the meta-analysis with the fewer number of overall studies. We then calculated two pooled ES for each meta-analysis: one corresponding to the pooled effect size from studies having the characteristic of interest (eg, allocation concealment) and the other for studies that did not (eg, no or unclear allocation concealment). We used standard random-effects meta-analyses to combine ES across trials and calculated the DerSimonian and Laird estimate of the variance to determine heterogeneity between trials.1 24 Then, for each meta-analysis, we derived the difference between pooled ES estimates from trials with and without the characteristic of interest (eg, allocation concealment). A negative difference in ES indicates that trials without the characteristic of interest show a more beneficial effect for the experimental group.
The second level analysis (between meta-analyses) involved pooling the results of the previous analysis to describe the effect of each trial component across all meta-analyses. The ES were also combined at this stage using the DerSimonian and Laird random-effects models25 to allow for between meta-analysis heterogeneity.
Formal tests of interaction between adequate sequence generation and concealment of allocation and estimated treatment benefits were performed separately for each meta-analysis based on Z scores using the estimated difference in ES between trials with and without adequate sequence and concealment of allocation and the corresponding SE.
We additionally stratified analyses accompanied by interaction tests according to the prespecified characteristics as reported by Nuesch et al1 as follows: treatment benefit in overall meta-analysis: small (ES greater than −0.5) versus large (ES≤ to −0.5); between-trial heterogeneity in overall meta-analysis (low (τ2<0.06) vs high (τ2≥0.06)), nature of the outcome (subjective or objective) and if the intervention was classified as musculoskeletal or other PT area.
In order to evaluate the effect of random sequence generation and allocation concealment on treatment effect estimates, we limited the analyses to studies describing a true control group, or placebo intervention as well as studies in which the direction of the expected treatment effect was evident (ie, standard care vs standard care plus active intervention; and active intervention 1 plus active intervention 2 vs active intervention 1).
Finally, results of our analyses were pooled with results from the meta-epidemiological study performed by Nuesch et al,1 which also investigated the effect of concealment of allocation on treatment effects on osteoarthritis measured on a continuous scale. Stata statistical software V.12 was used to perform these analyses.
Results
Selection and characteristics of meta-analyses and RCTs
The search identified 3901 Cochrane reviews, with 271 reviews potentially relevant to PT. Of these, 68 reviews included a meta-analysis of at least three studies of PT interventions and used a continuous outcome. We randomly selected 44 meta-analyses but excluded one 26 because it used follow-up data from the same group rather than a control group for comparison (figure 1). Forty-three meta-analyses including 393 trials and analysing 44 622 patients contributed to this study. Table 1 summarises the characteristics of the 43 Cochrane reviews. Briefly, the reviews were published between 2008 and 2011 and included meta-analyses of the effectiveness of PT interventions for musculoskeletal (22 reviews),27–35 cardiorespiratory (9 reviews),36–44 neurological (6 reviews)45–51 and other areas of physical therapy (6 reviews).51–56 A median number of six trials were included in the meta-analyses (IQR 5–8). Most trials were parallel group trials (367; 93.4%), single-centre studies (298; 76%) and had active control interventions (362; 92%). The most common intervention was exercise (n=282, 71.8%). Online Supplementary table S1 (appendix S1) lists the characteristics of each of the 43 meta-analyses.
Table 1.
Musculoskeletal | Cardio respiratory | Neurology | Other | |
---|---|---|---|---|
Characteristics | ||||
Number of meta-analyses | 22 | 9 | 6 | 6 |
Median year of publication | 2009 | 2010 | 2009 | 2009 |
Total number of trials included | 194 | 78 | 52 | 69 |
Total number of patients included | 19 861 | 9015 | 2138 | 13 608 |
Main intervention | ||||
Exercise | 13 | 7 | 3 | 5 |
Physical agents | 1 | 0 | 1 | 0 |
Acupuncture | 2 | 0 | 0 | 0 |
Manual therapy | 1 | 0 | 0 | 0 |
Other | 1 | 2 | 2 | 1 |
Outcomes | ||||
Clinician-assessed outcome | 8 | 4 | 6 | 3 |
Self-reported outcome | 11 | 4 | 0 | 2 |
Administrative data or automated outcome | 3 | 1 | 0 | 1 |
Sequence generation descriptive
From the 393 trials included in the 43 meta-analyses, 156 trials (39.7%) had appropriate sequence generation and 237 (60.3%) had inappropriate sequence generation (229 were classified as ‘unclear’ and 8 as ‘high’ risk of bias) according to the Cochrane Collaboration guidelines.
When analysing the sequence generation categories, we found that 229 trials (58.27%) did not clearly report the mechanism of how the sequence was generated. One hundred and thirty-two trials (33.6%) generated the sequence through computer software, a table of random numbers or by minimisation method. For a more specific sequence generation methods description, see table 2.
Table 2.
Sequence generation methods | Number of trials (%) |
---|---|
Unclear/not reported | 232 (59.03) |
Computer software | 65 (16.54) |
Random number table | 60 (13.49) |
Drawing of lots | 12 (3.05) |
Shuffling cards or envelopes | 9 (2.29) |
Minimisation | 6 (1.52) |
Other non-random methods | 4 (4.33) |
Coin tossing | 1 (0.25) |
Date of birth | 1 (0.25) |
Day of admission | 1 (0.25) |
Hospital/institution records number | 1 (0.25) |
Throwing a dice | 1 (0.25) |
Total | 393 (100) |
In addition, simple randomisation was reported in 34 trials, block randomisation in 54 trials, stratification in 62 trials and unclear or not reported in 218 trials. The rest of the trials reported other methods of random sequence generation.
Allocation concealment descriptive
From the 43 meta-analysis and 393 trials, 45 trials (11.5%) had appropriate allocation concealment and 348 (88.6%) had inappropriate concealment of allocation according to the Cochrane Collaboration guidelines. Results of analysis of the allocation concealment within each category were as follows: 282 trials (71.8%) had unclear method of allocation concealment, 21 (5.34%) trials used central allocation, 24 (6.11%) trials used envelopes with all 3 safe wards (opaque, sealed and sequentially numbered), 15 (3.82%) trials used envelopes without safeguards and 51 (16.8%) trials used any other non-concealed methods. Further details on allocation concealment methods are displayed in table 3.
Table 3.
Allocation concealment methods | Number of trials (%) |
---|---|
Unclear/not reported | 282 (71.76) |
Central concealment | 21 (5.34) |
Sequentially numbered, opaque and sealed envelopes | 24 (6.11) |
Unsafe envelopes | 15 (3.82) |
Non-safe methods of allocation | 51 (12.98) |
Total | 393 (100) |
Sequence generation and allocation concealment descriptive
Only 8.9% of the trials (n=35) had both appropriate sequence generation and appropriate allocation concealment. A great percentage (51%; n=199) of the trials did not have either appropriate sequence generation or appropriate concealment of allocation.
Sequence generation and treatment effects in PT trials
For the purpose of analysing the effect of sequence generation on treatment effects, 22 meta-analyses including 257 trials and analysing 30 287 patients contributed to this analysis. Figure 2 shows the forest plot of the differences in ES between trials with adequate and inadequate random sequence generation. There was no statistically significant difference between the ES of trials with adequate or inadequate random sequence generation (ES=0.02; 95% CI −0.12 to 0.15). The results of the stratified analyses are displayed in figure 3. None of the meta-analyses characteristics had a statistically significant interaction.
When analysing trials belonging only to category 2 which included trials that had adequate sequence generation (eg, use of a computer software, random number table and minimisation) versus unclear or not reported sequence generation, no statistically significant difference between the ES of these trials was found (ES=0.10; 95% CI −0.04 to 0.23). However, trials with unclear or not reported sequence generation tended to overestimate the treatment effect when compared with trials with adequate sequence generation from category 2. It was not possible to conduct the subgroup analysis between trials of category 2 versus trials from category 4 (using inappropriate methods to perform sequence generation such as date of birth, day of admission, hospital record number) since there were not enough trials for such analysis.
Allocation concealment and treatment effects in PT trials
For the purpose of analysing the effect of allocation concealment on treatment effects, 17 meta-analyses including 198 trials and analysing 27 011 patients contributed to this analysis. Figure 4 shows the forest plot of the differences in ES between trial with adequate and inadequate allocation concealment. Although trials with inappropriate allocation concealment tended to have an overestimate treatment effect when compared with trials with adequate concealment of allocation, the difference was non-statistically significant (ES=0.12; 95% CI −0.06 to 0.30). The results of the stratified analyses are displayed in figure 5. None of the meta-analyses characteristics had a statistically significant interaction. When focusing on trials with appropriate allocation concealment (category 1 and 2) versus trials with clearly inappropriate methods of concealment (category 4), the difference was not statistically significant (ES=0.26; 95% CI −0.04 to 0.55). However, trials with clearly inappropriate methods of concealment (category 4) tended to overestimate the treatment effects. The same was the case when comparing the trials with appropriate allocation concealment (category 1 and 2) versus trials with unclear or unreported concealment of allocation (ES=0.20; 95% CI 0.05 to 0.34). This time, the difference between these trials was statistically significant.
Pooling results with a previous meta-epidemiological study
When pooling our results with those of Nuesch et al,1 who performed a meta-epidemiological study with continuous outcomes as well, we obtained a pooled statistically significant value (ES=0.14; 95% CI 0.02 to 0.26), meaning that trials with inappropriate concealment of allocation had more beneficial effect than trials with appropriate concealment of allocation (figure 6).
Discussion
Our findings showed that adequate sequence generation and appropriate allocation concealment were only accomplished in a low percentage of PT trials (39.7% and 11.5%, respectively). In addition only 8.9% of the trials (n=35) had both appropriate sequence generation and appropriate allocation concealment despite the fact that both of the aforementioned risk of bias domains can be always appropriately performed when conducting an RCT in any field.57 In addition, trials with inappropriate concealment of allocation had an overestimation of treatment effects when compared with trials with adequate concealment of allocation. However, no difference in treatment effects in trials with appropriate or inappropriate sequence generation was found. These results confirm previous results obtained by several meta-epidemiological studies investigating the influence of sequence generation and allocation concealment on several areas of medicine using dichotomous outcomes.7 8 11 12 58 In addition, the pooled estimate obtained from our study and that of Nuesch et al,1 indicated that trials with inappropriate allocation concealment had a more beneficial effect than those with appropriate allocation concealment.
These results have important implications for the research community in general as well as for the discipline of PT. To our knowledge; this study is the first of its kind conducted in PT that examined continuous outcomes. Thus, it provides novel evidence in a very specific area of health. Most of the previous studies were looking at medical areas, such as pregnancy and childbirth, circulatory conditions, infectious disease, surgery and mental health trials. All of these medical areas certainly differ from PT with respect to: type of intervention (being mostly drugs), type of outcomes used and specific area of study.
First, it is not surprising that a large number of trials did not clearly report either randomisation sequence or allocation concealment or both. Our findings showed that approximately 62% and 70% of studies were classified as ‘unclear/not reported’ in their reporting of sequence generation and allocation concealment, respectively. This has also been observed in a meta-epidemiological study investigating meta-analyses before 2005.2 The study found a 64% and 69% of ‘unclear’ reporting of sequence generation and allocation concealment, respectively.2 Thus, despite the efforts made to improve reporting of RCTs such as introducing the CONSORT Statement, there is still inappropriate reporting and implementation of these important characteristics in RCTs. This is in agreement with results reported by Moseley et al,16 who found little or no effect of the CONSORT Statement on the quality of reports of physiotherapy trials. Quality of reporting has always been an issue when trying to understand the association between trial characteristics and treatments effects since most of authors do not properly report their methods. Although inappropriate reporting does not always reflect on weak trial conduct,59 some studies such as Pildal et al,12 found that most trials with unclear allocation concealment had also unclear allocation concealment in their protocol. Therefore, quality of reporting is related to assessment of risk of bias, but the direction of this relationship is still unclear. Future studies should assess how, how much and in which direction trial reporting affects treatment effects.
The magnitude and direction of the effect found in our study is consistent and almost identical with a previously published study performed in osteoarthritic trials investigating allocation concealment. In our study, the average bias associated with lack of allocation concealment corresponds to ¼ or ½ of a typical treatment effect found in PT interventions.60–63 Furthermore, the overestimation of a treatment effect was even higher when only focusing the analysis to trials with clearly inappropriate methods of allocation concealment. As stated by others,2 8 if allocation concealment is not accomplished, it is more likely that the results of the trial are inaccurate because, intentionally or unintentionally, investigators could have interfered with assigning participants to different groups, favouring the effect of the intervention. In addition, allocation concealment could be a proxy measure for other aspects of trial design besides selection bias.2 8
Random sequence generation was found not to be significantly associated with distortion of treatment effects in this study. Thus, it appears that random sequence generation does not have such a strong influence on treatment effects in comparison with concealment of allocation based on the results of the present study. Other meta-epidemiological studies have found similar results. 8 11 64 One of these studies, performed by Balk et al,64 has been questioned because of the high heterogeneity between trials included in meta-analysis. This could introduce a higher noise to be able to detect any effect of the trials characteristics. The other studies reported the same results argue that sequence generation is still an important characteristic that ensures rigour in the trial by allowing a trial to be concealed.8 11 In addition, other authors have suggested that the relationship between certain methodological characteristics and treatment effects varies according to each health area64 65 and more work should be carried out in other health areas to confirm these results.64 65
Our stratified analysis for sequence generation and allocation concealment showed that none of the meta-analysis characteristics presented with significant interaction. This is somehow contrary to other meta-epidemiological studies. For example, Wood et al,3 and Savovic et al,2 found that there was an overestimation of treatment effects that ranged between 22% and 31% for subjective outcomes. We followed exactly the same definition for determining outcomes as ‘objective and subjective’ used by Wood et al,3 to facilitate comparisons and avoid misclassifications. Thus, it seems that for PT area, there is no evidence of bias regarding random sequence generation or allocation concealment when subjective or objective outcome are used. This could be attributed to the fact that both domains can be performed in a trial regardless of the outcome analysed. Thus, whether the outcome is objective or subjective does not affect whether it is easier or not to predict patient's prognosis at the time of recruitment.
Study limitations
We analysed the influence of random sequence generation and allocation concealment on treatment effect estimates. It could be possible that other biases interact with these biases and have an influence in the treatment effects. This should include a multivariate analysis where several biases should be taken into consideration. Future research should look into this. However, such analysis would require meta-analyses with a large number of included studies, which are extremely rare; this makes such an analysis hard to be performed.
We limited our analysis to trials describing a true control group (ie, group receiving no treatment, or a waiting list), or placebo intervention as well as those studies in which the direction of expected treatment effect was evident. In this way, we could anticipate the direction of the bias in analyses. Combining effects of all trials without clear direction of what would be the effect (ie, in case of 2 similar active interventions) would have increased noise in the analyses and heterogeneity limiting our ability to find any significant effect of trials characteristics.66 However, limiting our analyses to these trials, we also reduced the power of our study.
Conclusions
Trials with inappropriate concealment of allocation had an overestimation of treatment effects when compared with trials with adequate concealment of allocation. Our results suggest that when evaluating risk of bias of primary RCTs in PT area, systematic reviewers and clinicians implementing trial findings into their clinical practice should pay attention to these characteristics, especially allocation concealment since it can be associated with an exaggeration of a treatment effect. Systematic reviewers should perform sensitivity analysis including trials with low risk of bias in these domains as primary analysis and/or in combination with less restrictive analyses. In addition, appropriate methods of sequence generation as well as allocation concealment should be implemented. Authors and editors should make sure that allocation concealment and random sequence generation are properly reported in trial reports.
Acknowledgments
In addition, the authors of this study thank the Alberta Research Center for Health Evidence (ARCHE) at the University of Alberta and all research assistants who helped with data collection.
Footnotes
Contributors: SA-O provided concept/idea/research design and writing. SA-O, CH, JF and HS performed the experiments. SA-O, BRdaC, GGC, CH, JF and HS were responsible for interpretation of the results and review of the final manuscript. SA-O, BRdaC analysed the data. All authors critically revised and provided final approval of the version to be published.
Funding: This project is funded by the Canadian Institutes of Health Research (CIHR), Alberta Innovates Health Solutions through a knowledge translation initiative grant, the Knowledge Translation (KT) Canada research Stipend programme, and the Physiotherapy Foundation of Canada (PFC) through a B E Schnurr Memorial Fund Award.
Competing interests: SA-O is supported by the Canadian Institutes of Health Research (CIHR) through a full-time Banting fellowship, the Alberta Innovates Health Solution through an incentive award, the STIHR Training Programme from Knowledge Translation (KT) Canada, the University of Alberta, and by the Faculty of Rehabilitation Medicine, at the University of Alberta, through the ‘Music and Motion Fellowship’. GGC holds a Centennial Professorship at the University of Alberta (2013–2020). JF is supported by the Government of Chile, University of Alberta through a Dissertation fellowship and the University Catholic of Maule. HS is supported through a Clinician Fellowship Award by Alberta Innovates—Health Solutions (AIHS), the Honorary Izaak Walton Killam Memorial Scholarship by the University of Alberta and the Honorary WCHRI Award by the Women and Children's Health Research Institute (WCHRI).
Ethics approval: This Research protocol has been approved by the Ethics Board of the University of Alberta (Pro00038172).
Provenance and peer review: Not commissioned; externally peer reviewed.
Data sharing statement: No additional data are available.
References
- 1.Nuesch E, Reichenbach S, Trelle S et al. . The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. Arthritis Rheum 2009;61:1633–41. 10.1002/art.24894 [DOI] [PubMed] [Google Scholar]
- 2.Savovic J, Jones HE, Altman DG et al. . Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 2012;157:429–38. 10.7326/0003-4819-157-6-201209180-00537 [DOI] [PubMed] [Google Scholar]
- 3.Wood L, Egger M, Gluud LL et al. . Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008;336:601–5. 10.1136/bmj.39465.451748.AD [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Falagas ME, Grigori T, Ioannidou E. A systematic review of trends in the methodological quality of randomized controlled trials in various research fields. J Clin Epidemiol 2009;62:227–31, 231.e229 10.1016/j.jclinepi.2008.07.012 [DOI] [PubMed] [Google Scholar]
- 5.Aickin M. Randomization, balance, and the validity and efficiency of design-adaptive allocation methods. J Stat Plann Inference 2001;94:97–119. 10.1016/S0378-3758(00)00228-7 [DOI] [Google Scholar]
- 6.Bausell RB, Lee WL, Soeken KL et al. . Larger effect sizes were associated with higher quality ratings in complementary and alternative medicine randomized controlled trials. J Clin Epidemiol 2004;57:438–46. 10.1016/j.jclinepi.2003.11.001 [DOI] [PubMed] [Google Scholar]
- 7.Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982–9. 10.7326/0003-4819-135-11-200112040-00010 [DOI] [PubMed] [Google Scholar]
- 8.Schulz KF, Chalmers I, Hayes RJ et al. . Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408–12. 10.1001/jama.1995.03520290060030 [DOI] [PubMed] [Google Scholar]
- 9.Siersma V, Als-Nielsen B, Chen W et al. . Multivariable modelling for meta-epidemiological assessment of the association between trial quality and treatment effects estimated in randomized clinical trials. Stat Med 2007;26:2745–58. 10.1002/sim.2752 [DOI] [PubMed] [Google Scholar]
- 10.Als-Nielsen B, Gluud L, Gluud C. Methodological quality and treatment effects in randomised trials: a review of six empirical studies. Paper presented at: 12th Cochrane Colloquium; 2–6 October 2004, Ottawa, Ontario, Canada. [Google Scholar]
- 11.Moher D, Pham B, Jones A et al. . Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? [see comment]. Lancet 1998;352:609–13. 10.1016/S0140-6736(98)01085-X [DOI] [PubMed] [Google Scholar]
- 12.Pildal J, Hrobjartsson A, Jorgensen KJ et al. . Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2007;36:847–57. 10.1093/ije/dym087 [DOI] [PubMed] [Google Scholar]
- 13.Hewitt CE, Kumaravel B, Dumville JC et al. . Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol 2010;63:1264–70. 10.1016/j.jclinepi.2010.01.010 [DOI] [PubMed] [Google Scholar]
- 14.Kjaergard LL, Als-Nielsen B. Association between competing interests and authors’ conclusions: Epidemiological study of randomised clinical trials published in the BMJ. BMJ 2002;325:249–52. 10.1136/bmj.325.7358.249 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Trowman R, Dumville JC, Torgerson DJ et al. . The impact of trial baseline imbalances should be considered in systematic reviews: a methodological case study. J Clin Epidemiol 2007;60: 1229–33. 10.1016/j.jclinepi.2007.03.014 [DOI] [PubMed] [Google Scholar]
- 16.Moseley AM, Herbert RD, Maher CG et al. . Reported quality of randomized controlled trials of physiotherapy interventions has improved over time. J Clin Epidemiol 2011;64:594–601. 10.1016/j.jclinepi.2010.08.009 [DOI] [PubMed] [Google Scholar]
- 17.Armijo-Olivo S, Cummings G, Fuentes CJ et al. . Identifying items to assess methodological quality in physical therapy trials: a factor analysis. Phys Ther 2014;94:1272–84. 10.2522/ptj.20130464 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.da Costa B, Resta N, Beckett B et al. . Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol. Sys Rev 2014;3:1–6. 10.1186/2046-4053-3-144 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.World Confederation for Physical Therapy. Position statement: standards of physical therapy practice. World Confederation for Physical Therapy, 2011. [Google Scholar]
- 20.Higgins JP, Altman DG, Goetzsche PC et al. , Cochrane Bias Methods Group; Cochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928 10.1136/bmj.d5928 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Sterne JA, Juni P, Schulz KF et al. . Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med 2002;21:1513–24. 10.1002/sim.1184 [DOI] [PubMed] [Google Scholar]
- 22.Cohen J. Chapter 1: the concepts of power analysis. In: Cohen J, ed. Statistical power analysis for the behavioral sciences. 2nd edn Hillsdale, NJ: Academic Press, INC, 1988:1–17. [Google Scholar]
- 23.Reichenbach S, Sterchi R, Scherer M et al. . Meta-analysis: chondroitin for osteoarthritis of the knee or hip. Ann Intern Med 2007;146:580–90. 10.7326/0003-4819-146-8-200704170-00009 [DOI] [PubMed] [Google Scholar]
- 24.Nuesch E, Trelle S, Reichenbach S et al. . The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ 2009;339:679–83. 10.1136/bmj.b3244 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986;7:177–88. 10.1016/0197-2456(86)90046-2 [DOI] [PubMed] [Google Scholar]
- 26.Spittle AJ, Orton J, Doyle LW et al. . Early developmental intervention programs post hospital discharge to prevent motor and cognitive impairments in preterm infants. Cochrane Database Syst Rev 2007;(2):CD005495 Chichester: John Wiley & Sons, Ltd. [DOI] [PubMed] [Google Scholar]
- 27.Fransen M, McConnell S, Hernandez-Molina G et al. . Exercise for osteoarthritis of the hip. Cochrane Database Syst Rev 2009;(3):CD007912 Chichester: John Wiley & Sons, Ltd 10.1002/14651858.CD007912 [DOI] [PubMed] [Google Scholar]
- 28.Handoll HH, Cameron ID, Mak JC et al. . Multidisciplinary rehabilitation for older people with hip fractures. Cochrane Database Syst Rev 2009;(4):CD007125. Chichester: John Wiley & Sons, Ltd, Chichester 10.1002/14651858.CD007125.pub2 [DOI] [PubMed] [Google Scholar]
- 29.Harvey LA, Brosseau L, Herbert RD. Continuous passive motion following total knee arthroplasty in people with arthritis. Cochrane Database Syst Rev 2010;(3):CD004260 Chichester: John Wiley & Sons, Ltd 10.1002/14651858.CD004260.pub2 [DOI] [PubMed] [Google Scholar]
- 30.Katalinic OM, Harvey LA, Herbert RD et al. . Stretch for the treatment and prevention of contractures. Cochrane Database Syst Rev 2010;(9):CD007455 Chichester: John Wiley & Sons, Ltd, Chichester 10.1002/14651858.CD007455.pub2 [DOI] [PubMed] [Google Scholar]
- 31.Manheimer E, Cheng K, Linde K et al. . Acupuncture for peripheral joint osteoarthritis. Cochrane Database Syst Rev 2010;(1):CD001977 John Wiley& Sons, Ltd, Chichester, UK 10.1002/14651858.CD001977.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Ostelo RW, Costa LO, Maher CG et al. . Rehabilitation after lumbar disc surgery. Cochrane Database Syst Rev 2008;(4):CD003007 John Wiley & Sons, Ltd. Chichester, UK 10.1002/14651858.CD003007.pub2 [DOI] [PubMed] [Google Scholar]
- 33.Rutjes AW, Nüesch E, Sterchi R et al. . Therapeutic ultrasound for osteoarthritis of the knee or hip. Cochrane Database Syst Rev 2010;(1):CD003132 John Wiley & Sons, Ltd, Chichester 10.1002/14651858.CD003132.pub2 [DOI] [PubMed] [Google Scholar]
- 34.Rutjes AW, Nüesch E, Sterchi R et al. . Transcutaneous electrostimulation for osteoarthritis of the knee. Cochrane Database Syst Rev 2009;(4):CD002823 John Wiley & Sons, Ltd, Chichester 10.1002/14651858.CD002823.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Schaafsma F, Schonstein E, Whelan KM et al. . Physical conditioning programs for improving work outcomes in workers with back pain. Cochrane Database Syst Rev 2010;(1):CD001822 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD007131.pub2 [DOI] [PubMed] [Google Scholar]
- 36.Davies P, Taylor F, Beswick A et al. . Promoting patient uptake and adherence in cardiac rehabilitation. Cochrane Database Syst Rev 2010;(7):CD007131 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Effing T, Monninkhof EM, van der Valk PD et al. . Self-management education for patients with chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2007;(4):CD002990 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PubMed] [Google Scholar]
- 38.Puhan M, Scharplatz M, Troosters T et al. . Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2009;(1):CD005305 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD005305.pub2 [DOI] [PubMed] [Google Scholar]
- 39.Taylor RS, Dalal H, Jolly K et al. . Home-based versus centre-based cardiac rehabilitation. Cochrane Database Syst Rev 2010;(1):CD007130 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD007130.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Main E, Prasad A, Schans C. Conventional chest physiotherapy compared to other airway clearance techniques for cystic fibrosis. Cochrane Database Syst Rev 2005;(1):CD002011 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.O'Brien K, Nixon S, Tynan AM et al. . Aerobic exercise interventions for adults living with HIV/AIDS. Cochrane Database Syst Rev 2010;(8):CD001796 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Jolliffe JA, Rees K, Taylor RS et al. . Exercise-based rehabilitation for coronary heart disease. Cochrane Database Syst Rev 2001;(1):CD001800 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PubMed] [Google Scholar]
- 43.Shaw K, Gennat H, O'Rourke P et al. . Exercise for overweight or obesity. Cochrane Database Syst Rev 2006;(4):CD003817 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Lacasse Y, Goldstein R, Lasserson TJ et al. . Pulmonary rehabilitation for chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2006;(4):CD003793 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PubMed] [Google Scholar]
- 45.Sirtori V, Corbetta D, Moja L et al. . Constraint-induced movement therapy for upper extremities in stroke patients. Cochrane Database Syst Rev 2009;(4):CD004433 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD004433.pub2 [DOI] [PubMed] [Google Scholar]
- 46.States RA, Pappas E, Salem Y. Overground physical therapy gait training for chronic stroke patients with mobility deficits. Cochrane Database Syst Rev 2009;(3):CD006075 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD006075.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Mead GE, Morley W, Campbell P et al. . Exercise for depression. Cochrane Database Syst Rev 2009;(3):CD004366 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD004366.pub4 [DOI] [PubMed] [Google Scholar]
- 48.Pollock A, Baer G, Pomeroy V et al. . Physiotherapy treatment approaches for the recovery of postural control and lower limb function following stroke. Cochrane Database Syst Rev 2007;(1):CD001920 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PubMed] [Google Scholar]
- 49.Mehrholz J, Friis R, Kugler J et al. . Treadmill training for patients with Parkinson's disease. Cochrane Database Syst Rev 2010;(1):CD007830 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD007830.pub2 [DOI] [PubMed] [Google Scholar]
- 50.Woodford H, Price C. EMG biofeedback for the recovery of motor function after stroke. Cochrane Database Syst Rev 2007;(2):CD004585 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Watson L, Ellis B, Leng GC. Exercise for intermittent claudication. Cochrane Database Syst Rev 2008;(4):CD000990 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD000990.pub2 [DOI] [PubMed] [Google Scholar]
- 52.Kramer MS, McDonald SW. Aerobic exercise for women during pregnancy. Cochrane Database Syst Rev 2006;(3):CD000180 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Orozco LJ, Buchleitner AM, Gimenez-Perez G et al. . Exercise or exercise and diet for preventing type 2 diabetes mellitus. Cochrane Database Syst Rev 2008;(3):CD003054 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD003054.pub3 [DOI] [PubMed] [Google Scholar]
- 54.Bendermacher BL, Willigendael EM, Teijink JA et al. . Supervised exercise therapy versus non-supervised exercise therapy for intermittent claudication. Cochrane Database Syst Rev 2006;(2):CD005263 John Wiley & Sons, Ltd, Chichester, UK. [DOI] [PubMed] [Google Scholar]
- 55.Forster A, Lambley R, Hardy J et al. . Rehabilitation for older people in long-term care. Cochrane Database Syst Rev 2009;(1):CD004294 John.Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD004294.pub2 [DOI] [PubMed] [Google Scholar]
- 56.Cramp F, Daniel J. Exercise for the management of cancer-related fatigue in adults. Cochrane Database Syst Rev 2008;(2):CD006145 John Wiley & Sons, Ltd, Chichester, UK 10.1002/14651858.CD006145.pub2 [DOI] [PubMed] [Google Scholar]
- 57.Jadad AR. Bias in randomized controlled trials. In: Jadad AR, Enkin M, eds. Randomized controlled trials: questions, answers, and musings. 2nd edn Malden, MA: Blackwell Pub, 2007:29–47. [Google Scholar]
- 58.Egger M, Juni P, Bartlett C et al. . How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 2003;7:1–76. [PubMed] [Google Scholar]
- 59.Soares HP, Daniels S, Kumar A, et al., Radiation Therapy Oncology Group. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004;328: 22–4. 10.1136/bmj.328.7430.22 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Abdul Latif L, Daud Amadera JE, Pimentel D et al. . Sample size calculation in physical medicine and rehabilitation: a systematic review of reporting, characteristics, and results in randomized controlled trials. Arch Phys Med Rehabil 2011;92:306–15. 10.1016/j.apmr.2010.10.003 [DOI] [PubMed] [Google Scholar]
- 61.Dorstyn D, Mathias J, Denson L. Applications of telecounselling in spinal cord injury rehabilitation: a systematic review with effect sizes. Clin Rehabil 2013;27:1072–83. 10.1177/0269215513488001 [DOI] [PubMed] [Google Scholar]
- 62.Barker AL, Talevski J, Morello RT et al. . Effectiveness of aquatic exercise for musculoskeletal conditions: a meta-analysis. Arch Phys Med Rehabil 2014;95:1776–86. 10.1016/j.apmr.2014.04.005 [DOI] [PubMed] [Google Scholar]
- 63.Pollock A, Baer G, Langhorne P et al. . Physiotherapy treatment approaches for the recovery of postural control and lower limb function following stroke: a systematic review. Clin Rehabil 2007;21:395–410. 10.1177/0269215507073438 [DOI] [PubMed] [Google Scholar]
- 64.Balk EM, Bonis PA, Moskowitz H et al. . Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 2002;287:2973–82. 10.1001/jama.287.22.2973 [DOI] [PubMed] [Google Scholar]
- 65.Verhagen AP, de Vet HC, Willemsen S et al. . A meta-regression analysis shows no impact of design characteristics on outcome in trials on tension-type headaches. J Clin Epidemiol 2008;61:813–18. 10.1016/j.jclinepi.2007.10.006 [DOI] [PubMed] [Google Scholar]
- 66.Hempel S. Detection of associations between trial quality and effect sizes 2012; http://www.ncbi.nlm.nih.gov/books/NBK84249/