Background
The recent controversy concerning the balance of safety and efficacy for the oral anti-diabetic drug rosiglitazone has highlighted fundamental inadequacies in data collection and trial design and the need for an overhaul of trial standards.1 In this article, we consider the question: are trials in physiotherapy up to scratch? The article is based on the premise that there is a need for good quality meta-analyses in all disciplines, and at present there is limited scope for rigorous and well-reported trial inclusion. While pharmaceutical regulators are likely to drive change in drug trials the article considers the issues for other disciplines such as physiotherapy that do not have the benefit of external scrutiny, and therefore need to drive the change from within. Such change is necessary to provide robust evidence for commissioning and healthy policy.
Evidence-based physiotherapy?
As the number of physiotherapy trials and systematic reviews increase, we could hope that we are developing a robust evidence base to inform patient care. Data from the Physiotherapy Evidence Database (PEDro), however, suggest that the quality of many physiotherapy trials is poor. Over 40% trials published 2006–2010 were evaluated as poor, and this percentage does not appear to have improved over the last decade (Table 1). Unfortunately, PEDro's use of summary scoring to evaluate aspects of trial quality is itself problematic2 as a ‘good’ score can be attained by a trial with, for example, no a priori specification of a primary outcome. This limits the use of the database and leaves us uncertain on the standard of physiotherapy trials.
Table 1.
Inclusion of systematic reviews and trials in the PEDro database (data extraction 12 November 2010)
| Year of publication | Systematic reviews (n) | Trials (n) | Trials (n, %) with scores on the PEDro scale indicating number of acceptable items evaluated by two blind reviewers | ||
|---|---|---|---|---|---|
| 8–10 Good | 5–7 Adequate | ≤4 Poor | |||
| 2006–2010 | 1398 | 4328 | 336 (7.7) | 2248 (51.9) | 1744 (40.4) |
| 2001–2005 | 782 | 3903 | 237 (6.1) | 2043 (52.2) | 1623 (41.7) |
| 1996–2000 | 325 | 2326 | 82 (3.5) | 1099 (47.2) | 1145 (49.3) |
| 1991–1995 | 98 | 1497 | 37 (2.5) | 636 (42.5) | 824 (55.0) |
| 1986–1990 | 30 | 1066 | 10 (1.0) | 394 (37.0) | 662 (62.0) |
| 1981–1985 | 5 | 589 | 6 (1.0) | 192 (32.6) | 391 (66.4) |
| 1976–1980 | 0 | 274 | 1 (0.4) | 72 (26.3) | 201 (73.3) |
| 1971–1975 | 0 | 121 | 2 (1.7) | 27 (17.9) | 92 (80.4) |
With an eye to recent experience in the drug regulatory setting,1,3 it is prudent to evaluate and, if necessary, improve the standards of clinical trials across disciplines. Pharmaceutical regulators are likely to drive such change in drug trials, but what about in other settings such as physiotherapy that do not have the benefit of external scrutiny? In evaluating how the quality of trials may affect the findings of systematic reviews and thus impact upon clinical interventions and patient outcomes, the internal validity of trials is an important consideration. We illustrate the challenges for physiotherapy trials through our recent systematic review4 evaluating the effectiveness of outpatient physiotherapy intervention in patients post first lumbar discectomy.
Discussion
There are multiple checklists available to evaluate the internal validity of trials (avoiding risk of systematic bias), each including the following important considerations: a priori specification of primary outcome, randomization and concealment of allocation, blinding, intention-to-treat, and avoidance of loss to follow-up.2,5 In our review, we used the Cochrane risk of bias assessment tool,5 and 16 trials (1336 participants) from 11 countries were included. Eight out of 16 (50.0%) trials evaluated as high risk of bias, seven were rated as unclear, and one as low risk. Reasons for trial exclusion from meta-analyses are risks that can be addressed through careful trial planning and management (Table 2).
Table 2.
Analysis of trials included in our systematic review4
| Component of internal validity | Evaluation of trial for risk of bias across n = 16 included trials |
|---|---|
| A priori specification of primary outcome | 1 trial (6.2%) protocol published that included a priori specification of primary outcome |
| 4 trials (25.0%) specified a primary outcome | |
| Randomization and concealment allocation | 11 trials (68.7%) reported appropriate randomization |
| 5 trials (31.2%) provided insufficient information re sequence generation | |
| 6 trials (37.5%) reported adequate concealment of allocation | |
| 10 trials (62.5%) provided insufficient information re concealment allocation | |
| Blinding | 5 trials (31.2%) did not report blinding of assessor(s) |
| Blinding of participants and treating physiotherapists was not possible in all trials | |
| Intention-to-treat (ITT) | 6 trials (37.5%) stated using ITT, although clarity of ITT was confusing in 1 trial |
| Of the 6 trials employing ITT, only 3 (18.7%) reported adherence to group allocation | |
| Of the 10 trials not using ITT, 9 trials (56.2%) provided no data regarding participants receiving allocated intervention or adherence to group allocation | |
| Avoidance of loss to follow-up | 4 trials (25.0%) reported high losses to follow up (3 trials reported >20% losses) |
| There was no strategy to address losses in the 3 trials with more than 20% losses | |
| 2 further trials (12.5%) excluded data owing to increased pain / poor compliance |
A priori specification of primary outcome measure
A priori specification of the primary outcome in a protocol is a straightforward application of alpha spending; avoiding unacceptable redefinition post unblinding that would introduce reporting bias and multiplicity concerns. The primary outcome is the outcome ‘capable of providing the most clinically relevant and convincing evidence directly related to the primary objective of the trial’ (p5).6 The ICH guidelines identify measurement properties, and any existing standards or normal practice as key considerations in selecting the primary outcome, to ensure that sufficient evidence exists for the potential of the measure to evaluate a clinical relevant and important benefit of treatment in the population of interest. Secondary outcome measures can be supportive of the primary objective of the trial or used to address secondary objectives.6 Pre-specification of the role of secondary outcome measures in the interpretation of results is also important to avoid multiple statistical testing or cherry-picking of outcome measures.7
In our review, only one trial protocol had been published and only 4/16 (25.0%) trials specified a primary outcome. It was not possible to evaluate whether outcomes were fully reported, contributing to a high number of unclear evaluations of risk.8 Most trials reported some assessment of disability, pain, physical functioning, overall impairment and return to work, but one trial had no outcome that could be included in meta-analyses (no outcome addressing disability, function or health). The variation in outcomes limited the possible comparisons, and leg pain was not described in a consistent manner to facilitate meta-analysis. This is particularly remarkable since leg pain is the main complaint for this population and the principal indication for lumbar discectomy. This identifies a lack of consistency of outcomes across trials in the same setting and population.
Outcome measures are not imposed upon physiotherapy trials, unlike the situation for investigational drug trials where it is our experience that regulators play an active role in ensuring major outcomes are included and defined consistently. It may be tempting to use surrogate measures, such as postural sway, as a substitute for important clinical endpoints, but the evidence that surrogates are a reliable predictor of clinical benefit may not be persuasive. Composite outcomes can provide increased efficiency in clinical trials,9 although they are not without their challenges, and their appropriate use in physiotherapy trials could be explored. Physiotherapists therefore need to consider means of achieving consistency of robust clinically important outcome measures themselves. Some work is addressing this issue, including definition of standardized outcome measures for low back pain,10 OMERACT's international initiative to improve outcome measurement in rheumatology trials by endorsing specific outcome measures across disciplines,11 and the MRC COMET initiative to define minimum core sets of outcomes across a range of clinical areas;12 however, further progress is required. A recent consensus technique to establish international recommendations for musculoskeletal physiotherapy research provides support for this point; identifying the development of core sets of outcomes for new trials as a priority.13
Randomization and concealment of allocation
Randomization utilizes the play of chance to allocate participants to experimental groups. The allocation should be a concealed process to minimize bias.5 In our review, 11/16 (69.7%) trials presented a low risk of bias for randomization, but only 6/16 (37.5%) trials presented a low risk of bias for concealment of allocation and in 10 (62.5%) trials allocation concealment was unclear. At a minimum this indicates poor reporting14 but may also indicate a risk of selection bias, which undermines trial results.
Blinding
Blinding is designed to reduce the risk that knowledge of the received intervention can affect outcome;5 therefore avoiding performance, attribution, attrition and detection bias. Blinding of the treating physiotherapist has been acknowledged as generally impossible in physiotherapy trials except when using interventions such as laser when codes can be used to programme the intervention. Blinding of the patient is again not always possible, for example in situations when active physiotherapy is compared to advice only. Blinded outcome assessment however, is possible, and trials should allow costs for independent assessors. In our review 5/16 (31.2%) trials did not mention blinding.
Intention-to-treat (ITT)
The principle of ITT is that participants should be analysed in the groups to which they were randomized,5 irrespective of what, if any, treatment they ultimately received. ITT preserves randomization and thus avoids bias15. This principle should be maintained in situations where participants do not receive or adhere to their allocated intervention to ensure that effects are estimated on the allocation of the intervention (for example by following up those who withdraw from treatment). This can be a particular challenge in physiotherapy trials as patients not allocated to a physiotherapy intervention might seek physiotherapy elsewhere. For example, in one trial in our review, 3/47 (6.4%) patients in the comparison group crossed over to receive the physiotherapy intervention, and a further 19 (40.4%) sought physiotherapy elsewhere. The only methodologically robust solution to the problem of patients being treated differently from their randomized allocation is to minimize such cross-over through design and trial conduct. Strategies to minimize cross-over include appropriate trial administration, appropriate use of inclusion criteria, appropriate procedures for obtaining informed consent, and randomization shortly before commencement of treatment interventions. The only methodologically robust analysis to conduct is an ITT analysis that will often dilute the effects of treatment in trials in which a number of subjects do not receive their randomized treatment. ITT analysis is therefore important within physiotherapy research and only six trials (37.5%) stated using ITT in our review, and of the six trials employing ITT, only three (18.7%) reported adherence to group allocation.
Avoidance of loss to follow-up
Loss to follow-up provides potential for bias as it undermines trial randomization, which ensures that participants differ only by the play of chance and the treatment allocation. The outcome status of participants lost to follow-up is unknown, might be linked to the intervention of interest, and can change the magnitude of the effect estimate.5 It has been proposed that a loss to follow-up of >20% poses a serious threat to the validity of a trial,16 and we would suggest that lower rates are also a major cause for concern. Indeed, the magnitude of loss to follow-up that represents a concern depends upon the rate of events in a trial and the size of treatment effect.
Loss to follow-up contributed to a high risk of bias in 4/16 (25.0%) trials in our review and was >20% in three trials that were subsequently excluded from the meta-analysis. Incomplete outcome data also contributed to high risk of bias in two further trials, where participants' data were excluded owing to increased pain/lack of compliance with the intervention. Loss to follow-up is an issue reflected in the wider physiotherapy literature. For example the UK BEAM trial17 evaluating physical treatments for low back pain experienced losses of 23% and 26% at 3 and 12 months, respectively. In physiotherapy trials, loss to follow-up has been regarded with some inevitability, particularly in trials with long-term follow-up. There is, however, evidence of physiotherapy trials avoiding losses, for example in a study of physiotherapy and occupational therapy in care homes, losses were 1% and 4% in groups at 6-month follow-up;18 although follow-up might be more straightforward in a care home context compared to the outpatient clinics in our review. Innovative prevention and retention strategies should be incorporated into trial design and conduct to minimize losses.
Poor reporting
The multiple scoring of ‘unclear’ in our assessment of risk of bias for included trials identifies problematic reporting of trials in physiotherapy. Publication of the Consolidated Standards of Reporting Trials (CONSORT) statement in 1996 (revised 2001 and 2010)14 has impacted upon trial reporting in PubMed; with improvements in primary outcome reporting, sample size calculation, method for random sequence generation and allocation concealment; but reporting remaining below an acceptable level.19 Our findings suggest that reporting remains an issue for physiotherapy.
The way forwards for physiotherapy?
This analysis illustrates that while some physiotherapy trials are well-planned and executed, many are not meeting well-defined criteria to ensure confidence in results. There is a need for good quality meta-analyses in physiotherapy. However, there is limited scope at present for rigorous and well-reported trial inclusion. Physiotherapy trials need to avoid risk of bias and improve their coherence and efficiency. Planning for quality is important, particularly for issues that present known problems for physiotherapy trials, including loss to follow-up. Areas for specific development include the publication of trial protocols to inform assessment of risk of bias, and the need to develop further interdisciplinary teams for physiotherapy trials involving trials units, statisticians, quality of life experts, and health economists working in partnership with clinicians. Including large numbers of participants in poorly designed trials is unethical. This supports the need to take urgent action.
Summary
Although physiotherapy possesses examples of researchers and trials that are rigorous, broadly as a profession physiotherapy needs to address urgently the methodological quality of trials. This in turn will enable greater trust in trial and systematic review findings. As a minimum, all physiotherapy trials should be based on a trial protocol that explicitly addresses components of possible systematic bias (a priori specification of a primary outcome, randomization and concealment of allocation, blinding, intention-to-treat analysis, and avoidance of loss to follow-up) and the appropriate selection of outcome measures. Established physiotherapy researchers can facilitate this by providing consensus for minimum core sets of outcome measures for specific populations. The issues identified in this article have a wider reach than physiotherapy researchers; informing all disciplines that are without external regulation; as well as funding bodies, policymakers and commissioners.
DECLARATIONS
Competing interests
None declared
Funding
None
Ethical approval
Not applicable
Guarantor
AR
Contributorship
AR, MC and NF were responsible for the conception of the article; all authors contributed to the systematic review on which this article is based, and have been involved in developing the content of the article; AR wrote the first draft in discussion with CW; AR worked with MC and NF re-working content into subsequent drafts; all authors gave final approval of the version to be published
Acknowledgements
None
References
- 1.Freemantle N Commentary: What can we learn from the continuing regulatory focus on the thiazolidinediones? BMJ 2010;341:c4812 [DOI] [PubMed] [Google Scholar]
- 2.Juni P, Witschi A, Bloch R, Egger M The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282:1054–60 [DOI] [PubMed] [Google Scholar]
- 3.Moynihan R Rosiglitazone, marketing, and medical science. BMJ 2010;340:785–9 [DOI] [PubMed] [Google Scholar]
- 4.Rushton A, Wright C, Goodwin P, Calvert M, Freemantle N Physiotherapy rehabilitation post first lumbar discectomy: a systematic review and meta-analysis of Randomised Controlled Trials. Spine 2011;36:E961–72 [DOI] [PubMed] [Google Scholar]
- 5.Higgins JPT, Altman DG, Sterne JAC Assessing risk of bias in included studies : Higgins JPT, Green S, Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. Oxford: The Cochrane Collaboration, 2011. See www.cochrane-handbook.org [Google Scholar]
- 6.ICH Topic E9 Statistical Principles for clinical trials. Step 5. Note for guidance on statistical principles for clinical trials. CPMP/ICH/363/96 London: EMEA, 2006 [Google Scholar]
- 7.Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman D Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291:2457–65 [DOI] [PubMed] [Google Scholar]
- 8.Freemantle N Interpreting the results of secondary end points and subgroup analyses in clinical trials: should we lock the crazy aunt in the attic? BMJ 2001;322:989–91 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Freemantle N, Calvert M, Wood J, Eastaugh J, Griffin C Composite outcomes in randomized trials: greater precision but with greater uncertainty? JAMA 2003;289:2554 [DOI] [PubMed] [Google Scholar]
- 10.Deyo RA, Battie M, Beurskens AJ et al. Outcome measures for low back pain research. A proposal for standardized use. Spine 1998;23:2003–13 [DOI] [PubMed] [Google Scholar]
- 11.Tugwell P, Boers M, Brooks P, Simon L, Strand V, Idzerda L OMERACT: An international initiative to improve outcome measurement in rheumatology trials. Trials 2007;8:38 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.MRC Network of Hubs for Trial Methodology and Research The COMET (Core Outcome Measures in Effectiveness Trials) Initiative. See http://www.methodologyhubs.mrc.ac.uk/news__events/comet_initiative.aspx (last checked 9 February 2011)
- 13.Foster NE, Dziedzic KS, van der Windt DAWM, Fritz JM, Hay EM Research priorities for non-pharmacological therapies for common musculoskeletal problems: nationally and internationally agreed recommendations. BMC Musculoskeletal Disorders 2009;10:3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Schulz KF, Altman DG, Moher D CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c332 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Hollis S, Campbell F What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ 1999;319:670–4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Oxford CEBM. See http://www.cebm.net/levels_of_evidence.asp. (last checked 14 January 2011)
- 17.UK BEAM trial team United Kingdom back pain exercise and manipulation (UK BEAM) randomised trial: effectiveness of physical treatments for back pain in primary care. BMJ 2004;329:1377 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Sackley CM, van den Berg M, Lett K et al. Effects of a physiotherapy and occupational therapy intervention on mobility and activity in care home residents: a cluster randomised controlled trial. BMJ 2009;339:b3123 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hopewell S, Dutton S, Yu l, Chan A, Altman D The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 2010;340:c723 [DOI] [PMC free article] [PubMed] [Google Scholar]