Influence of allocation concealment and intention-to-treat analysis on treatment effects of physical therapy interventions in low back pain randomised controlled trials: a protocol of a meta-epidemiological study

Abstract

Introduction

Meta-epidemiological studies examining the influence of methodological characteristics, such as allocation concealment and intention-to-treat analysis have been performed in a large number of healthcare areas. However, there are no studies investigating these characteristics in physical therapy interventions for patients with low back pain. The aim of this study is to investigate the influence of allocation concealment and the use of intention-to-treat analysis on estimates of treatment effects of physical therapy interventions in low back pain clinical trials.

Methods and analysis

Searches on PubMed, Embase, Cochrane Database of Systematic Reviews, Physiotherapy Evidence Database (PEDro) and CINAHL databases will be performed. We will search for systematic reviews that include a meta-analysis of randomised controlled trials that compared physical therapy interventions in patients with low back pain with placebo or no intervention, and have pain intensity or disability as the primary outcomes. Information about selection (allocation concealment) and attrition bias (intention-to-treat analysis) will be extracted from the PEDro database for each included trial. Information about bibliographic data, study characteristics, participants’ characteristics and study results will be extracted. A random-effects model will be used to provide separate estimates of treatment effects for trials with and without allocation concealment and with and without intention-to-treat analysis (eg, four estimates). A meta-regression will be performed to measure the association between methodological features and treatment effects from each trial. The dependent variable will be the treatment effect (the mean between-group differences) for the primary outcomes (pain or disability), while the independent variables will be the methodological features of interest (allocation concealment and intention-to-treat analysis). Other covariates will include sample size and sequence generation.

Ethics and dissemination

No ethical approval will be required for this study. The study findings will be published in a peer-reviewed journal and presented at international conferences.

Registration number

International Prospective Register of Systematic Reviews (CRD42016052347).

Strengths and limitations of this study.

• This protocol was specified following the Preferred Reporting Items for Systematic review and Meta-Analyses Protocols guidelines.

• This study will be the first to evaluate the association between methodological characteristics and estimates of the treatment effects of physical therapy interventions in low back pain trials.

• The results from this meta-epidemiological study are likely to bring new insights to the physical therapy scientific community by informing issues that need to be considered in randomised trials.

• The findings from this meta-epidemiological study may have limited generalisability to other clinical conditions, since it is restricted to physical therapy treatment of low back pain.

Introduction

Systematic reviews and meta-analyses of randomised controlled trials are considered the ‘gold standard’ to evaluate the efficacy and effectiveness of healthcare interventions.^{1 2} While systematic reviews and meta-analyses provide the best available evidence to support clinical decision-making and to promote changes in health policies, they are not free of bias.^{3 4} The presence of potential biases in clinical trials, due to inadequate allocation concealment (selection bias) or lack of intention-to-treat (ITT) analysis (attrition bias) may lead to inflated estimates of treatment effect.^5–8 These types of bias can directly influence the results from meta-analyses, leading to inaccurate conclusions, misleading clinicians and researchers.⁹

Meta-epidemiological studies are designed to understand the impact of study level characteristics (eg, methodological quality, study design) in randomised controlled trials (RCTs), investigating the association between these specific study characteristics and the intervention effect estimates from collections of meta-analyses.^{10 11} Previous meta-epidemiological studies have evaluated the influence of methodological characteristics, such as allocation concealment and ITT analysis on the treatment effects of clinical interventions in different healthcare areas.^{6 12–14} The result of a combined analysis of meta-epidemiological studies demonstrated that trials with inadequate or unclear allocation concealment overestimated the treatment effects up to 7%.¹⁵ This means that the conclusions of meta-analyses are likely to change if restricted to trials with adequate allocation concealment. It has been reported that 69% of meta-analyses were no longer statistically significant when trials with unclear or inadequate allocation concealment were excluded.¹⁶

Deviation from ITT analysis is common in systematic reviews and RCTs. A study of 222 systematic reviews reported that 36% included at least one trial that deviated from ITT.¹⁷ However, the influence of performing ITT analysis on the effect estimates of RCTs is still unclear.^{13 14 18} The direction and magnitude of the influence of performing ITT analysis may vary between studies according to different methods and definitions used. For example, Dossing et al ¹⁹ classified trials as ITT analysis or ‘modified ITT’ analysis. They classified trials as using an ITT analysis if all randomised patients were included in analysis, including both patients with clinical outcome data and patients with imputed outcome data; and ‘modified ITT’ analysis if some individuals were excluded from the analyses despite the authors of the trial referring to it as being ITT. No significant difference in the treatment effect was found between trials that performed ITT analysis compared with ‘modified ITT’ analysis. In contrast, Abraha et al,²⁰ using a different definition for deviation from ITT analysis without taking into account the occurrence of postrandomisation exclusions, found that the treatment effect of trials that performed ‘modified ITT’ analysis was inflated by 17% compared with trials that performed ITT analysis.

Most meta-epidemiological studies derived from reports of RCTs are from the field of medicine. These studies are likely to be different from physical therapy trials, especially regarding the type of intervention and outcomes assessed.²¹ To date, there is only one meta-epidemiological study published that evaluated the relationship between methodological characteristics and estimates of treatment effect of physical therapy interventions.²¹ This study included different areas of physical therapy and therefore, selecting a wide range of different clinical conditions, leading to a large level of heterogeneity that may hamper the association between the characteristics assessed and treatment effects. Therefore, we chose to focus on low back pain trials. Besides being the most prevalent, costly and disabling musculoskeletal condition, low back pain is the musculoskeletal condition with largest number of clinical trials in the physical therapy literature.^22–25

The lack of meta-epidemiological studies that have evaluated the association between methodological characteristics and the treatment effects of interventions in low back pain trials motivated us to conduct this study.

Objectives

The objectives of this study are: (1) to establish if adequate allocation concealment and the use of ITT analysis influence the estimates of treatment effect of physical therapy interventions in low back pain RCTs and (2) to evaluate if allocation concealment and ITT analysis are evaluated and reported adequately in clinical trials.

Methods

Study design

This is a protocol of a meta-epidemiological study.

Protocol and registration

The protocol was prospectively registered at the International Prospective Register of Systematic Reviews, registration number: CRD42016052347. We followed the Preferred Reporting Items for Systematic review and Meta-Analyses Protocols²⁶ guidelines and the checklist is available in online supplementary appendix 1.

Identification and selection of studies

All RCTs included in systematic reviews with meta-analysis evaluating physical therapy treatment in adults with non-specific low back pain that included pain or disability (as continuous variables) as the main outcomes will be included in this study. Non-specific low back pain is defined as low back pain not attributed to a specific pathology, such as nerve root compromise or serious spinal pathology.²⁷ The physical therapy intervention will be compared with placebo or no intervention. We will consider all possible meta-analyses from the same systematic review, since a systematic review may contain more than one meta-analysis (eg, different outcomes). Meta-analyses with only one trial will be excluded. Any systematic reviews with meta-analysis with mixed populations will not be considered. Overview of reviews will also not be considered in our study. In the case of Cochrane reviews, if there are multiple versions of the same review (eg, updates), we will consider only the most recent one.

Potentially eligible systematic reviews to be included will be retrieved through an electronic search in the following databases from their inception up to February 2017: PubMed (https://www.ncbi.nlm.nih.gov/pubmed), Embase via OvidSP, Cochrane Database of Systematic Reviews (http://www.cochranelibrary.com/), PEDro (http://pedro.org.au/) and CINAHL (https://health.ebsco.com/products/the-cinahl-database). There will be no restriction to language of studies. The search strategy will combine validated filters related to ‘systematic reviews and meta-analyses’, ‘physical therapy interventions’ and ‘low back pain’ (see online supplementary appendix 2 for more details). Two review authors will independently assess and select potential studies to be included based on titles and abstracts. Any discordance will be resolved by consensus, and if necessary, a third assessor will arbitrate the final decision. Full texts of selected systematic reviews will be collected and evaluated in the same manner. After the process of identification and selection, we will screen the systematic reviews for meta-analyses that fulfil our inclusion criteria. Once we have selected the systematic reviews with the meta-analyses to be included in our study, we will look for the full texts of the RCTs included on these meta-analyses in order to extract their original data. Two authors from our study are trained raters of the Physiotherapy Evidence Database (PEDro) database, so it will be possible to obtain the full texts that are indexed on this database. It is important to state that about 92% of physical therapy RCTs are indexed on the PEDro database.²⁸ For those RCTs not indexed on PEDro, we will make all efforts to get these full texts (searching other databases, contact authors).

Risk of bias assessment (selection and attrition bias)

We will extract the information about selection (allocation concealment) and attrition bias (ITT analysis) from PEDro for all trials included in the systematic reviews.²⁹ We decided to use the PEDro scale due to the following reasons: 1) the PEDro scale has high reliability for individual ratings and consensus ratings and can be used as a continuous scale for measuring the methodological quality of trials^{29 30}; 2) the PEDro scale is strongly correlated (r=0.83; 95% CI 0.76 to 0.88) with the Cochrane Risk of Bias tool³¹ and 3) feasibility: as two authors from this study are raters from the PEDro database, we can easily download the PEDro scores for the included RCTs in our study.

The definitions used by the PEDro database for these domains are:

‘Concealed allocation means that the person who determined if a subject was eligible for inclusion in the trial was unaware, when this decision was made, of which group the subject would be allocated to. A point is awarded for this criteria, even if it is not stated that allocation was concealed, when the report states that allocation was by sealed opaque envelopes or that allocation involved contacting the holder of the allocation schedule who was off-site’;

‘An intention to treat analysis means that, where subjects did not receive treatment (or the control condition) as allocated, and where measures of outcomes were available, the analysis was performed as if subjects received the treatment (or control condition) they were allocated to. This criterion is satisfied, even if there is no mention of analysis by intention to treat, if the report explicitly states that all subjects received treatment or control conditions as allocated’.²⁹

Each domain will be rated as ‘yes’, when the criterion is clearly satisfied, or ‘no’ when the criterion is not satisfied or the information is unclear in the text. For ITT classification, trials will be rated as ‘yes’ for ITT analysis, if they use the term ‘intention to treat’, and it is clear that all subjects received treatment or control conditions as allocated, or that subjects were analysed according to their initial group allocation. When there are postrandomisation exclusions, a trial will be rated as ‘no’ if the exclusion is on the basis of not receiving allocated treatment. There are some trials that exclude patients after randomisation if authors subsequentially realise that the participant was not eligible for the trial.^{32 33} In this specific case, the trials will be rated as ‘yes’. Trials will be rated as ‘no’, if authors did not mention any ITT approach or reported the use of a modified ITT approach. However, if it is clear that there were no postrandomisation exclusions and all subjects were analysed according to their initial group allocation, it will be rated as ‘yes’.

When PEDro score is not available at the PEDro database (the article may not be indexed in PEDro database; or the article may be in process to be rated), two assessors, not involved in the study, will independently assess it, using the PEDro rating protocol. Any disagreements will be resolved by discussion or arbitration by a third assessor when consensus cannot be reached. The assessors will be trained raters of the PEDro scale that work in our department and will be blinded to the scope of the manuscript.

Data extraction and synthesis

Two review authors will independently extract data from all trials included in the selected systematic reviews using a standardised extraction form. Any disagreements will also be resolved by discussion or arbitration by a third reviewer when consensus cannot be reached.

We will extract bibliographic data (authors, title and year of publication), study characteristics (sample size and interventions used), characteristics of the participants (gender, age, duration and severity of the condition), and outcomes results (mean and SD). We will only extract outcomes results for the short-term follow-up (closest to 4 weeks). Email requests will be sent to trial authors for additional information data, when necessary.

Data analysis and synthesis

In order to determine whether allocation concealment and ITT analysis affect estimates of treatment effect, a two-level analysis will be conducted. Individual trial data will be retrieved from the meta-analyses included in our study. Initially, the treatment effects of each trial will be calculated using mean differences (MD) with 95% CIs for between-group differences at short-term follow-up, or for between-group differences in change scores.^{34 35} Data will be converted to a common 0 to 100-point scale if trials had evaluated the same outcome on different scales. Data will be presented separately for each outcome of interest (pain and disability).

After establishing the treatment effects for the trials included, we will calculate four pooled treatment effect estimates using a random-effects model: (i) trials with allocation concealment, (ii) trials without allocation concealment, (iii) trials with ITT analysis and (iv) trials without ITT analysis. This will be done separately for pain and disability outcomes (eg, a total of eight estimates).

We will assess between-trial heterogeneity using I² test. This test demonstrates whether the percentage of total variation across studies is explained by heterogeneity rather than chance. An I²>75% will be considered as ‘high heterogeneity’, an I² of 50%–75% will be considered as ‘moderate heterogeneity’ and an I²<25% will be considered as ‘low heterogeneity’.³⁶ Review Manager V.5 software will be used for all meta-analyses and heterogeneity assessment.

The second stage of the analysis will be a meta-regression to evaluate the association between methodological characteristics of included trials and the estimates of treatment effect.¹¹ The dependent variable will be the treatment effects (mean between-group difference) for the main outcomes (pain or disability), while the independent variables will be the methodological characteristics of interest (allocation concealment and ITT analysis). Additionally, we will add two independent covariates in meta-regression analysis: sample size and sequence generation. We decided to investigate the effect of sequence generation and sample size since these variables have been associated with treatment effect estimates.³⁷ We will extract the information about sequence generation from the PEDro database and is defined as adequate if ‘subjects were randomly allocated to groups’ (eg, computer-generated random numbers; coin-tossing and dice-rolling).²⁹ It will be rated as ‘yes’, when the criterion is clearly satisfied, or ‘no’ when the criterion is not satisfied or the information is unclear in the text. The sample size will be considered as a continuous quantitative variable in the meta-regression model. The covariates will be used according to the number of studies included (eg, one covariate for every 10 trials). Random-effects meta-regression will be conducted using the ‘metareg’ command in STATA V.10 and weighted using effect size standard errors.¹¹

Discussion

To our knowledge, this will be the first study aimed to investigate if the magnitude of the treatment effects of physical therapy interventions in low back pain RCTs is influenced by methodological characteristics (eg, allocation concealment and ITT analysis). Meta-analyses from RCTs are responsible for the most reliable evidence on the treatment of patients with low back pain, and their results are of interest to a range of stakeholders.

However, biased results from RCTs can lead to inadequate clinical decision-making and consequently affect patient outcomes. Clinicians may select interventions for their patients based on trials results that are inflated, and so these are not a very reliable guide to treatment selection. For example, a systematic review about the effectiveness of low level laser therapy for chronic low back pain found a significantly greater reduction in pain in response to laser therapy compared with placebo.³⁸ However, this finding was based on a meta-analysis with three clinical trials that did not conceal allocation or use an ITT analysis, so this result could be an overestimate of the true effect of laser. If this premise of overestimated effect is true, clinicians might in good faith select interventions that are unlikely to help their patients.

Therefore, we believe that the findings of this meta-epidemiological study will provide important contributions to clinicians, researchers and policy-makers in the low back pain field. This study will improve the available evidence for physical therapists, through the disclosure of reliable and unbiased results on clinical decision-making. The present meta-epidemiological study has several strengths. Since this approach is similar to a systematic review, we will prespecify methods to identify and select studies to be included through predefined inclusion criteria and sensitive search strategy. Data extraction and analysis will be also performed by rigorous methods to avoid potential bias in this study.

The limitation of the present study is that it is restricted to meta-analyses of physical therapy interventions in non-specific low back pain. Thus, our results may have limited generalisability to other interventions as well as to other clinical conditions.

Dissemination

We intend to disseminate our results through presentations at national and international conferences (eg, World Confederation for Physical Therapy and International Back and Neck Pain Forum) as well as publishing our study in a high-impact international scientific journal.