The influence of bias in randomized controlled trials on rehabilitation intervention effect estimates: what we have learned from meta-epidemiological studies

Abstract

This study aimed to synthesize evidence from studies that addressed the influence of bias domains in randomized controlled trials on rehabilitation intervention effect estimates and discuss how these findings can maximize the trustworthiness of an RCT in rehabilitation. We screened studies about the influence of bias on rehabilitation intervention effect estimates published until June 2023. The characteristics and results of the included studies were categorized based on methodological characteristics and summarized narratively. We included seven studies with data on 227,806 RCT participants. Our findings showed that rehabilitation intervention effect estimates are likely exaggerated in trials with inadequate/unclear sequence generation and allocation concealment when using continuous outcomes. The influence of blinding was inconsistent and different from the rest of medical science, as meta-epidemiological studies showed overestimation, underestimation, or neutral associations for different types of blinding on rehabilitation treatment effect estimates. Still, it showed a more consistent pattern when looking at patient-reported outcomes. The impact of attrition bias and intention to treat has been analyzed only in two studies with inconsistent results. The risk of reporting bias seems to be associated with overestimation of treatment effects. Bias domains can influence rehabilitation treatment effects in different directions. The evidence is mixed and inconclusive due to the poor methodological quality of RCTs and the limited number and quality of studies looking at the influence of bias and treatment effects in rehabilitation. Further studies about the influence of bias in RCTs on rehabilitation intervention effect estimates are needed.

Randomized controlled trials (RCTs) are prospective studies that measure the efficacy, effectiveness and safety of interventions. Although no study design is likely to prove absolute causality, randomization reduces bias and facilitates the examination of cause-effect relationships between an intervention and an outcome. This is because randomization and allocation concealment help balance participant characteristics (both known and unknown) between the groups allowing unbiased conclusions about the effect of an intervention under study. This is very unlikely with any other study design.¹

The RCT design requires that researchers carefully select the population, the interventions to be compared, and the outcomes of interest.² Once these are defined, the sample size needs to be calculated to estimate the appropriate number of participants to detect, with reasonable probability, an expected and/or clinically relevant difference between groups for a given study. To achieve comparability between groups, the participant characteristics must be homogeneous, cointerventions should be avoided or similar between groups, participants, intervention providers, and outcome assessors should be ideally blinded to the intervention allocations if possible, and dropout from the study should be minimal.³ If the intervention is the only difference between the trial groups, any differences in the outcomes could reliably be attributed to the intervention. These ideal experimental conditions allow high internal validity of RCTs. Still, the generalizability of their results to patients outside the studied population (external validity) is inherently limited, and clinicians must employ their judgment when applying results from RCTs to their clinical practice, which is potentially problematic.^{4, 5} During the last two decades, we have seen significant expansion in the use of RCT’s model to evaluate interventions in real-world circumstances, referred to as “effectiveness” research rather than traditional RCT “efficacy” research.^6-8 In this particular “effectiveness” context, the number of RCTs published in the rehabilitation field is increasing more than in other fields.⁷ In 2022, rehabilitation has been defined by Cochrane Rehabilitation as “a multimodal, person-centered, collaborative process, including interventions targeting a person’s capacity (by addressing body structures, functions, and activities/participation) and/or contextual factors related to performance, with the goal of optimizing the functioning of persons with health conditions currently experiencing disability or likely to experience disability, or persons with disability in interaction with the environment”⁹. This new definition proposes that rehabilitation interventions are “complex interventions” because they are a process (changing in time), multimodal (more than one intervention), collaborative (interaction of behaviors) and person-centered (aims and means can accordingly change). It is suggested that “the greater the difficulty in defining precisely what exactly are the `active ingredients’ of an intervention and how they relate to each other, the greater the likelihood that you are dealing with a complex intervention”¹⁰. Therefore, there are challenges when applying methodological standards of high-quality clinical research to rehabilitation interventions.¹¹

The “effectiveness RCT”, also considered as pragmatic trials⁸ can be a trade-off between rehabilitation characteristics and the ideal RCT model because they are typically more inclusive, reflecting the heterogeneity of patients in clinical practice, allowing for flexible delivery of the interventions, measuring functional outcomes, and often lack of blinding of participants and those providing treatment.⁵ However, by maintaining randomization while relaxing the other experimental conditions (e.g., strict fidelity to rehabilitation interventions, measurement of functional outcomes, and blinding), it is argued that the trial results could be rigorous enough, generalizable to real-world settings and probably applicable in the rehabilitation context.⁶

In the literature, many studies have developed specific frameworks for developing and evaluating complex interventions and advocate for methodological standards that should be respected when planning an effectiveness study using complex interventions.^{12, 13} In addition, many manuscripts have been published on improving the quality of the evidence in rehabilitation.^{6, 7, 14, 15} These studies have highlighted the need to identify whether biases resulting from methodological issues related to rehabilitation characteristics, such as lack or difficulty of blinding, lack or difficulty of intervention standardization or fidelity and adherence to the intervention, can lead to an overestimation or underestimation of the true rehabilitation intervention effect estimates. This can be achieved using empirical evidence, by using meta-epidemiological studies, which collate numerous meta-analyses and, within each of them, contrast the estimates of intervention effects of trials with and without a methodological characteristic of interest.¹⁶ Indeed, effect size estimates may be related to the conduct (i.e., methodological variables) of the studies, beyond the study design per se (RCTs vs. observational studies).¹⁷ Therefore, it is important to determine what important methodological features of studies conducted in the rehabilitation field could potentially bias treatment effects results.

In the last few years, several meta-epidemiological studies about the potential influence of bias on treatment effects have been conducted in the area of rehabilitation; however, to our knowledge, no one has attempted to synthesize this evidence to date. To respond to this need, we decided to conduct a meta-research study synthesizing this information. This manuscript is part of a series of papers developed by the Cochrane Rehabilitation field and discussed extensively during the 5^th Cochrane Rehabilitation Methodological Meeting: “The Rehabilitation evidence ecosystem: useful study designs”, that took place in Milan on 7^th and 8^th September 2023. This series of papers has attempted to answer the following questions: 1) Which study design can and should be used for which kind of research question to provide useful evidence for rehabilitation? 2) How do we maximize the trustworthiness of the results of these study designs to produce useful evidence on rehabilitation? 3) How can we conduct studies of different designs to make them useful for rehabilitation end-users (clinicians, policymakers and patients)? 4) When and how can these designs contribute to building the evidence in rehabilitation if RCTs are not yet available?

To provide data to address question 2, we summarized all relevant studies that addressed the influence of bias domains in RCTs, examining rehabilitation intervention effect estimates and discussing how these findings can help to maximize the trustworthiness of RCTs in the field of rehabilitation.

Methods

We systematically identified and summarized studies which investigated the association between reported bias domains (selection bias, performance bias, detection bias, attrition bias, reporting bias and other bias) and rehabilitation intervention effect estimates in RCTs included in Agency for Healthcare Research and Quality – Evidence-based Practice Center (AHRQ-EPC) review, published in 2014,¹⁸ and in Page’s systematic review, published in 2016.¹⁹ We focused on bias domains, as suggested by Cochrane³ because it is argued that these are the main methodological characteristics that can change the direction of effect regardless of the study design used to evaluate the effectiveness of the intervention.

Evidence collection and eligibility criteria

We screened eligible studies included in the AHRQ-EPC review (latest search September 2012)¹⁸ and in Page’s systematic review (latest search May 2015).¹⁹ To identify more recent meta-epidemiological studies in rehabilitation, we performed a narrow quick search in PubMed from June 2015 to June 2023 using the keywords “meta-epidemiological studies” AND “rehabilitation” without the development of a specific search strategy. No language limitations were applied in the database searches nor as inclusion criteria. Two authors (CA and SM) independently reviewed the reference lists of all the included meta-epidemiological studies to identify additional meta-epidemiological studies in rehabilitation that the electronic search could have missed.

The eligibility criteria were: 1) type of studies: meta-epidemiological studies or studies that evaluate the effect of methodological characteristics on effect estimate²⁰ of rehabilitation interventions, and 2) types of methodological characteristics: studies had to evaluate biases using the conceptual framework of the Cochrane risk of bias tool for RCTs as described above.³

Two independent reviewers (EP and CA) screened the studies using the predefined eligibility criteria. The studies were analyzed using the available information and a senior author (SM) resolved disagreements between reviewers through discussion.

Data extraction

We extracted the following data: study characteristics (author, year, design, setting and type of rehabilitation intervention), number of RCTs and participants included in the meta-epidemiological synthesis, and type of bias domains (i.e., selection bias, performance bias, detection bias, attrition bias and reporting bias). We also extracted all estimates of the relationship between different types of methodological characteristics and treatment effects. One reviewer extracted the information (CA), and a second reviewer independently checked all data extracted (SM).

Data analysis and synthesis

The information obtained from the included studies was organized by type of methodological characteristics and summarized using a narrative approach, including the effect size estimation (ES) and 95% confidence interval (CI). Evidence tables and figures were used to present data where appropriate.

Results

We retrieved 53 full-text articles from the two previous systematic reviews^{18, 19} and 13 full-text articles from the electronic search. We found only studies published in English. Seven meta-epidemiological studies met the inclusion criteria.^21-27 See Figure 1 for the screening process.

Figure 1.

—Flowchart of the studies.

Characteristics of the included studies

We included seven eligible meta-epidemiological studies (included data on 227,806 participants)^21-27 in a rehabilitation context. Armijo-Olivo and De Almeida^{23, 24} addressed the influence of sequence generation and allocation concealment on treatment effect, Armijo-Olivo and Liu^{21, 27} on blinding of participants, clinicians and assessors, Armijo-Olivo²² on attrition bias, Hayden ²⁶ on reporting bias and Fuentes²⁵ on sponsorship bias. The rehabilitation interventions considered were physical therapy (PT). Still, in four studies,^{21-23, 25} there was no definition of which interventions they focused on, and De Almeida²⁴ included the following interventions as physical therapy: exercise therapy, electrophysical agents, manual therapy, acupuncture, behavioral therapy, advice or education, multidisciplinary treatment, taping, and traction. Liu²⁷ included only progressive resistance strength intervention, and Hayden²⁶ only exercise therapy.

All studies used a meta-analytic approach²⁰ to analyze the influence of certain methodological characteristics (rated as inadequate/adequate/unclear) on treatment effect estimates. The average bias associated with methodological characteristics was reported in all meta-epidemiological studies. Full details are reported in Table I.^21-27

Table I. —Characteristics of included studies.^21-27.

Author year	Design	Setting	Rehabilitation intervention	Outcome	Sample size	Number of included RCTs	Methodological characterstics
Armijo-Olivo 23	Meta-epidemiological study	Rehabilitation	Physical Therapy	NA	44622	393	Sequence generation and allocation concealment
Armijo-Olivo 21	Meta-epidemiological study	Rehabilitation	Physical therapy	NA	44622	393	Blinding
Armijo-Olivo 22	Meta-epidemiological study	Rehabilitation	Physical therapy	NA	44622	393	Bias related to attrition, missing data and the use of ITT
De Almeida 24	Meta-epidemiological study	Rehabilitation	Exercise therapy, electrophysical agents, manual therapy, acupuncture, multidisciplinary treatment, taping, traction, educa- tion, behavioral therapy	Low back pain	20555	128	Allocation Concealment and ITT analysis
Fuentes 25	Meta-epidemiological study	Rehabilitation	Physical therapy	NA	44622	393	Sponsorhip bias
Hayden 26	Cross-sectional meta-epidemiological study	Rehabilitation	Exercise therapy for chronic low back pain	Low back pain	25704	279	Publication integrity, quality of conduct and reporting
Liu 27	Meta-epidemiological study	Rehabilitation	Progressive resistance strength training	Lower limb muscle strength	3059	73	Blinding

NA: not applicable; RCTs: randomized control trials; ITT: intention to treat.

Selection bias – random sequence generation and allocation concealment

Armijo-Olivo and De Almeida^{23, 24} evaluated the association between random sequence generation and allocation concealment on physical therapy effect estimates.

Armijo-Olivo²³ included 43 meta-analyses, 393 RCTs with 44,622 participants. It found no difference between adequate and inadequate random sequence generation (ES difference: 0.02; 95% CI -0.12 to 0.15), while trials with inadequate allocation concealment tended to have an overestimated effect estimate when compared with those with adequate allocation concealment, but the mean difference was non-statistically significant (ES difference: 0.12; 95% CI -0.06 to 0.30).

De Almeida²⁴ included 128 RCTs with 20,555 participants, who were treated using exercise therapy, electrophysical agents, manual therapy, acupuncture, behavioral therapy, advice or education, multidisciplinary treatment, taping, and traction. It showed that, although trials that had inadequate allocation concealment had larger treatment effect estimates than those trials with adequate allocation concealment, the difference between the estimates was not statistically significant for pain (ES difference: 0.31 95% CI -0.04 to 0.66) and disability (ES difference: 0.19; 95% CI -0.16 to 0.54) outcomes.

Performance bias and detection bias – blinding

Armijo-Olivo and Liu^{21, 27} evaluated the association between blinding of participants, assessors, and/or health providers (overall) on treatment effect estimates when analyzing physical therapy interventions or progressive resistance strength using continuous outcomes. Armijo-Olivo²¹ included 393 RCTs with 44,622 participants. It suggested that inappropriate overall blinding (whether the appropriate component of blinding [participants, assessors, therapist or irrelevant] based on the main outcome of interest for the trial was used) underestimated treatment effect estimates, but the mean difference between trials with inappropriate vs. appropriate overall blinding was not statistically significant (ES -0.08; 95% CI -0.28 to 0.12). In addition, although trials with inappropriate blinding of assessors and participants tended to underestimate treatment effects when compared with trials with appropriate blinding of assessors and participants, the difference was not statistically significant (ES: -0.07; 95% CI -0.22, 0.08; ES -0.12; 95% CI -0.30, 0.06, respectively).

Liu,²⁷ which includes 73 RCTs with 3,059 participants within a meta-analysis, showed that trials using blinded assessors tended to report smaller effect sizes than those using unblinded assessors on muscle strength outcomes, with a difference of -0.80 (95% CI -1.35 to -0.25). The reported effects were exaggerated in trials that used unblinded assessors.

Attrition bias and intention-to-treat analysis

Two studies^{21, 23} evaluated the association between attrition bias and/or intention-to-treat (ITT) analysis. Armijo-Olivo²² including 43 meta-analysis, 393 RCTs with 44,622 participants, evaluated the association between attrition bias and treatment effect estimates when evaluating physical therapy interventions using continuous outcomes. This study²² showed that trials which did not use the ITT principle, or which were assessed as having inappropriate control of incomplete outcome data, tended to underestimate the effect estimates when compared with trials with adequate use of ITT (ES difference: -0.13; 95% CI -0.26 to 0.01) and control of incomplete outcome data (ES difference: -0.18; 95% CI -0.29 to -0.08).

Almeida,²⁴ stated that, overall, ITT had no significant influence on treatment effects in trials for low back pain. However, the findings demonstrated that trials using ITT had a significantly lower effect estimate than those without ITT. The difference between trials with and without ITT on pain outcomes was ES: 0.49; 95%CI 0.12 to 0.87. In contrast, for disability outcomes, the difference between trials with and without ITT was not statistically significant (ES: 0.16; 95%CI -0.21 to 0.53).

Reporting bias

Hayden,²⁶ including 279 RCTs with 25,704 participants within a Cochrane review, evaluating the association between reporting bias and exercise therapy estimation’s effect on pain and functional outcomes. This study reported that RCTs that did not report trial registration or had no published protocol available reported larger mean differences in pain outcomes (MD: -5.0; 95% CI -8.8 to -1.3 unadjusted model; MD: -2.3; 95% CI -6.7 to 2.1 adjusted model). This was true also for studies that did not report core outcome measures (MD, -8.1; 95% CI -11.9 to -4.4 unadjusted model; MD: -7.3; 95% CI -11.3 to -3.3 for the adjusted model), that did not include follow-up outcome measurement of at least 12 weeks (MD: -3.9; 95% CI -7.5 to -0.3 unadjusted model; MD: -2.9, 95% CI -7.1 to 1.4 for the adjusted model), or that did not report complete participant flow information (i.e., CONSORT flow chart; MD: -3.9; 95% CI -7.5 to -0.3 unadjusted model; MD: -1.8; 95% CI: -5.8 to 2.2 for the adjusted model). Only missing core outcome measures remained significantly associated with increased mean effect estimates on pain outcome after adjustment for all the rest of variables (12 in total) (MD: -7.3; 95% CI -11.3 to -3.3 for the adjusted model). Inadequate sample size was found to be statistically significantly associated with increased functional limitations outcomes reported in the included trials (MD: -7.1; 95% CI -12.2 to -1.9) in unadjusted and adjusted analyses (MD: -6.4; 95% CI -12.0 to -0.8).

Other bias - sponsorship bias

Fuentes,²⁵ including 43 meta-analyses, 377 RCTs and 43,651 participants, evaluated the association between sponsorship bias and treatment effect estimates for physical therapy intervention using continuous outcomes. This study reported that funding was not declared in many trials (N.=85, 22%). When it was declared, the influence of the trial sponsor was assessed as being appropriate (i.e., statement of no sponsor involvement in the trial) in 246 trials (63%), and considered inappropriate/unclear (i.e., statement of sponsor involvement in the trial/not enough information) in the remaining 147 (37%) trials. The authors concluded that trials with inappropriate/unclear influence of funders tended to have, on average, a larger effect size than those with the appropriate influence of funding (ES difference: 0.15; 95% CI −0.03 to 0.33). Although the difference was not statistically significant, trials with sponsor involvement tended to overestimate the effect estimates. See Table II for full details.

Table II. —Data synthesis.

Bias	N. studies	Rehabilitation intervention	Outcomes	Effect estimates	Findings
Selection bias (random sequence generation and allocation concealment)	2	Physical Therapy and exercise therapy	Pain and disability outcomes (only one study)	ES difference: 0.12; 95% CI -0.06 to 0.30 Pain: ES difference: 0.31 95% CI -0.04 to 0.66 Disability: ES difference: 0.19; 95%CI -0.16 to 0.54	Inadequate allocation concealment tended to have an overestimated effect estimate when compared with those with adequate allocation concealment
Performance bias and detection bias (blinding)	2	Physical therapy and progressive resistance strength	Continuous outcomes and muscle strength outcomes	ES: -0.08; 95% CI -0.28 to 0.12	1) Inappropriate overall blinding underestimated treatment effect estimates; inappropriate blinding of assessors and participants tended to underestimate treatment effects 2) Blinded assessors underestimated the treatment effect estimates
Attrition bias and intention-to-treat analysis	1	Physical intervention	Pain and disability (continuous data)	ES difference (attrition): -0.13; 95% CI -0.26 to 0.01; ES (ITT): 0.49; 95% CI 0.12 to 0.87	Inappropriate control of incomplete outcome data, tended to underestimate the effect estimates. ITT tended to underestimate estimate than those trials without ITT
Reporting bias	1	Exercise therapy	Pain and functional outcomes	MD: −5.0; 95% CI −8.8 to −1.3 unadjusted model; MD: −2.3; 95% CI −6.7 to 2.1 adjusted model	No trial registration or no published protocol available tended to overestimate the treatment effect
Other bias (sponsorship)	1	Physical therapy	Continuous outcomes	ES: 0.15; 95% CI −0.03 to 0.33	Inappropriate/unclear influence of funders tended to overestimate the treatment effect estimates

ES: effect size estimation; CI: confidence interval; MD: mean difference; ITT: intention-to-treat.

Discussion

We found only seven studies that investigated the influence of bias in treatment estimates in the rehabilitation field. This limited evidence is contrasted with the higher number of meta-epidemiological studies conducted in other medical fields identified by Page’s systematic review and AHRQ-EPC review (N.=49). Our findings align with the findings of Page’s systematic review,¹⁹ agreeing that intervention effect estimates are most likely exaggerated in trials with inadequate/unclear sequence generation and allocation concealment (selection bias) when using continuous outcomes. The influence of blinding (performance and detection bias) on the results of trials in rehabilitation seems to be inconsistent. It differs from other medical research, which shows a more consistent pattern when looking at patient-reported outcome measures (PROMs). In other medical fields, it seems that failure to blind patients probably overestimates treatment effects for PROMs. At the same time, there is high uncertainty about its influence on other outcomes. Also, blinding of outcome assessors has been found to influence treatment effect estimates in medical trials, and thus, blinding of outcome assessors has been found important for subjective outcomes. We found that studies looking at rehabilitation studies and blinding (N.=2) showed overestimation, underestimation, or neutral associations for different types of blinding (i.e., participants, assessors, therapists, statisticians) on rehabilitation treatment effect estimates. Consequently, no clear conclusions can be drawn on the influence of blinding on trial estimates based on this literature.²⁸ This means that blinding, as currently evaluated, probably could be less important than often believed²⁹ or that studies may have used other ways to decrease performance or detection biases (but not necessarily have used blinding of allocation). For example, a way of decreasing these biases could be by providing limited information about the hypothesis of the study to participants and assessors, using expertise-based RCTs to avoid biases at the therapist level, or providing neutral expectations for treatments offered, among others.³⁰ These alternative ways of conducting RCTs in the rehabilitation field arguably highlight that – despite the challenges associated with blinding within RCTs in rehabilitation – there are potential adaptations which may help overcome these biases.

The influence of attrition bias on treatment effects on continuous outcomes is also unclear. This could be related to the variation of the definition of attrition and associated biases across meta-epidemiological studies. Also, the definition of the concept of ITT has been misused and misinterpreted by several studies^{31, 32} (e.g., trials reported the phrase “intention to treat” with no apparent deviation in the description or trials that correctly described the ITT principle).³³ In our study, from the two included studies that looked at the concept of ITT, one found an overestimation for trials with inappropriate ITT, and the other found an underestimation regardless of the type of outcome. This difference could depend on the definition of attrition bias used,³⁴ as well as the outcomes investigated since, according to some,³⁵ the influence of biases on treatment effect estimates depends on the outcome investigated.

The results of this study suggest also that the lack of core outcome reporting (reporting bias) seems to be associated with higher treatment effect sizes reported for pain outcomes and probably in functional limitations outcomes. The influence of other characteristics on treatment effect estimates in rehabilitation is uncertain because, as far as our knowledge, they have not been yet examined.

Given the scarce number of studies looking at the influence of bias and treatment effects in the rehabilitation field and their heterogeneity regarding the definition of the biases and types of rehabilitation interventions and outcomes, we cannot draw any definitive conclusion regarding the influence of bias on rehabilitation treatment effects. In addition, these included studies have been exploratory since they did not perform sample size calculations³⁶ and were underpowered due to the small magnitude of the differences between trials with and without the bias domain, small sample sizes, and the high heterogeneity of the datasets. It has been recommended^{18, 37} that larger sample sizes (≥600 trials and/or 50 meta-analyses or even more³⁸) are required to have adequate power to investigate these associations between bias and treatment effects. None of the included studies accomplished this standard. Also, it has been suggested²⁸ to assemble a homogenous set of meta-analyses and trials in a specific area of research to decrease the heterogeneity of the datasets and increase the likelihood of finding a difference when it really exists.

It is also important to highlight that most existing meta-epidemiological studies have analysed the isolated effect of biases on treatment effect estimates. However, it has been highlighted that the influence of certain biases on treatment effect estimates may vary across different areas of research and be influenced by interaction with other biases.¹⁸ Therefore, future meta-epidemiological studies should look at this interaction, to address this important gap in the literature.

Empirical evidence from meta-epidemiological studies is of value to guiding risk of bias assessment when conducting systematic reviews to guide healthcare decision-making. Although these studies have been recognized as a good method to investigate biases, they are not free from limitations, as highlighted by Moustgaard et al.,¹⁶ and Page,³⁹ who have emphasized several steps when interpreting the results of these studies; especially confounding. Thus, results from these studies should be interpreted with caution. Future meta-epidemiological studies should follow this guidance to provide accurate estimates of the influence of biases in treatment effect estimates in health research and the rehabilitation literature. Further, the conclusions of these included studies report that often, the description of the methodological characteristics of RCTs was very poor in primary studies, relevant information was commonly missing, and most of the RCTs included in meta-epidemiological studies were underpowered.

Once again, we highlight the urgent need to improve the quality of the reporting and conduct of RCTs by following the CONSORT statement, because if we do not have enough information on how an RCT has been conducted, we cannot know whether the study is biased and whether the bias can influence the treatment effect estimates and consequently the trustworthiness and value of the evidence provided by them.⁴⁰ This might be supported by investing time, energy, and funding in training future generations on methodological principles and epidemiology. It has been shown that researchers who are inexperienced and untrained in the steps and procedures of the scientific method may not know how to develop a research question and how to design a well-conducted RCT in an efficient manner. Training future rehabilitation scientists would eventually improve the quality of conduct of trials in our field.⁴¹ We also need to produce more high-quality RCTs to have more reliable data and consistent findings that allow us to study the relationship between bias and effect estimates and to be more confident regarding the effectiveness of the rehabilitation interventions.

Limitations of the study

This study had several limitations. Firstly, a limited search strategy in PubMed was used for searching meta-epidemiological studies published between 2015 and 2023. However, it should be acknowledged that PubMed has been recognized as one of the most complete and updated databases in the health literature. This was a meta-research study with a narrow scope. Therefore, some studies could have been missed. However, the aim was to bring together and discuss how the findings from meta-epidemiological studies can support the trustworthiness of RCTs in rehabilitation. Our study included only published reports, meaning that there is a risk that we have failed to incorporate unpublished studies. We did not consider whether the studies adhered to the actual definition of meta-epidemiological studies as described by Moustgaard et al^{16, 42} As highlighted by the literature, the definition of meta-epidemiological studies is unclear, and researchers used it inconsistently. Secondly, meta-epidemiological studies did not assess the risk of bias; therefore, they included RCTs of any methodological quality, indirectly creating bias across the included meta-epidemiological studies.^{43, 44} For this reason, we only considered the bias domains used in the Cochrane Risk of Bias tool, excluding other significant biases such as publication⁴⁵ or bias in measuring the outcome.⁴³ Thirdly, most interventions included in meta-epidemiological studies were physical therapy and exercise without any information regarding controls or the outcome measurements. This might lead to limited generalizability of the findings to other rehabilitation interventions, participants, and outcomes.

Conclusions

Our findings show that bias domains can influence the treatment effect in different directions. The evidence is mixed and inconclusive due to the poor methodological quality of RCTs as well as the limited number and quality of studies looking at the influence of bias and treatment effects in rehabilitation. We can only confirm the need to improve the reporting and the conduction of randomized studies. Further studies about the influence of bias in RCTs on rehabilitation intervention effect estimates are needed.