Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study

Abstract

Objective To compare estimates of intervention effects between single centre and multicentre randomised controlled trials with continuous outcomes.

Design Meta-epidemiological study.

Data sources 26 meta-analyses totalling 292 randomised controlled trials (177 single centre, 115 multicentre) with continuous outcomes published between January 2007 and January 2010 in the Cochrane database of systematic reviews.

Data extraction Data were extracted on characteristics of trials, single or multicentre status, risk of bias using the risk of bias tool of the Cochrane Collaboration, and results.

Data synthesis The intervention effects were estimated with standardised mean differences. For each meta-analysis, random effects meta-regression was used to estimate the difference in standardised mean differences between single centre and multicentre trials. Differences in standardised mean differences were then pooled across meta-analyses by a random-effects meta-analysis model. A combined difference in standardised mean differences of less than 0 indicated that single centre trials showed larger treatment effects, on average, than did multicentre trials. Because single centre trials may be more prone to publication bias and may have lower methodological quality than multicentre trials, sensitivity analyses were done with adjustment for sample size and domains of the risk of bias tool.

Results Single centre trials showed larger intervention effects than did multicentre trials (combined difference in standardised mean differences −0.09, 95% confidence interval −0.17 to −0.01, P=0.04), with low heterogeneity across individual meta-analyses (I²=0%, between meta-analyses variance τ²=0.00). Adjustment for sample size slightly attenuated the difference (−0.08, −0.17 to 0.01). Adjustment for risk of bias yielded similar estimates with wider confidence intervals, some of them crossing 0 (−0.09, −0.17 to 0.00 for overall risk of bias).

Conclusions On average, single centre clinical trials with continuous outcomes showed slightly larger intervention effects than did multicentre trials. Further research is needed to investigate potential causes of these differences.

Introduction

Empirical evidence shows that lack of allocation concealment, lack of blinding, or excluding patients from the analysis may lead to an underestimation or overestimation of the true intervention effect.^{1 2 3 4 5 6 7 8} Such empirical evidence is often based on meta-epidemiological studies, which use a collection of meta-analyses to explore the influence of specific methodological characteristics of a trial, such as flaws in trial design or conduct, on the estimates of treatment effect.⁹

Another trial characteristic that could be associated with intervention effect estimates is the status of the study centres, whether the trial is carried out in a single centre or in several centres (multicentre). One study¹⁰ warned against the validity of single centre trials in critical care medicine because the positive results of many single centre trials were frequently contradicted when tested in multicentre settings. More recently, a meta-epidemiological study of binary outcomes showed that intervention effects were on average larger in single centre randomised trials than in multicentre trials.¹¹

According to a recent methodological study of a representative sample of randomised controlled trials, binary outcomes represented only 28% of all outcomes reported and continuous outcomes 69%.¹² Results for binary outcomes may not be extrapolated to trials assessing continuous outcomes because such trials usually differ in medical condition, risk of bias, sample size, and statistical analysis. We carried out a meta-epidemiological study to compare intervention effect estimates from single centre and multicentre trials that evaluated a continuous end point.

Methods

We searched for eligible meta-analyses published between 1 January 2007 and 1 January 2010 in the Cochrane database of systematic reviews using a previous search strategy¹³: “standardised mean difference” OR “standardized mean difference” OR “SMD” OR “effect size” OR “mean difference” in the title, abstract, or keywords. We focused on Cochrane meta-analyses because studies have shown that these have high methodological quality, are well reported, and are associated with fewer conflicts of interest than are non-Cochrane meta-analyses.^{14 15 16}

Study selection

Prespecified eligibility criteria were systematic reviews including at least one meta-analysis of a continuous primary outcome, as defined by the authors of the Cochrane review, and assessing the effect of therapeutic interventions for at least five randomised controlled trials. If the systematic review included several meta-analyses with multiple continuous outcomes and did not specify a primary outcome, we selected the meta-analysis with the highest number of trials. We excluded meta-analyses of non-randomised controlled trials as well as those of individual patient data, those including only unpublished data, and updates of meta-analyses. Meta-analyses in which all trials were single centre or multicentre could not contribute to the analysis and were also excluded. Because final value and change scores should not be combined together as standardised mean difference, we excluded meta-analyses that originally contained a mixture of change from baseline and final value scores as mean differences.

One reviewer selected potentially relevant meta-analyses after screening the title and abstract, and full text if needed, according to the prespecified eligibility criteria. A second reviewer checked all included and excluded meta-analyses. Disagreements were resolved by discussion.

All individual randomised controlled trials included in the meta-analyses were eligible, and the full text for the trials was systematically searched. If the full text was not available we excluded the trial. We excluded crossover or cluster trials and subgroup analyses.

Data extraction

A standardised data collection form was used to collect the general characteristics and centre status of each trial (single or multicentre) and to assess the risk of bias from the original reports of the trials. One reviewer extracted all data. A second reviewer independently duplicated data extraction and risk of bias assessment in a random sample of one fifth of the trials, and consensus was achieved by discussion.

General characteristics of individual trials

For each trial we recorded the general characteristics, such as year of publication, funding source (public, private, not reported), and number of patients randomly allocated in each group. We classified the experimental intervention as non-pharmacological or pharmacological, with a non-pharmacological intervention classified as any intervention that did not include an active substance.^{17 18} We also noted the type of comparator: placebo, waiting list, no treatment, usual care, or active treatment.

Two reviewers blinded to the trials and results of the meta-analysis identified the experimental and control groups for each eligible meta-analysis according to the objectives in the individual trials. If the two reviewers could not distinguish the experimental and control arms, the meta-analysis was excluded.

Assessment of single centre or multicentre status in individual trials

We initially relied on the self reporting of single centre or multicentre status for selected trials. If the status was not clearly reported in the original report, we systematically contacted the corresponding author up to three times. In case of no response we applied the rules that if the report stated both several ethics committees and different affiliations of authors then we classified the trial as multicentre, and if the report stated both a single ethics committee and a single author affiliation then we classified the trial as single centre. In all other cases we considered that the status was unclear and excluded the trial from the analysis.

Assessment of risk of bias in individual trials

As recommended by the Cochrane Collaboration we used the risk of bias tool to assess the risk of bias from the individual reports for each trial.¹⁹ The domains assessed separately were random sequence generation, allocation concealment, blinding, incomplete outcome data, and selective outcome reporting. According to the Cochrane handbook, the item dedicated to selective reporting should be assessed by comparing the protocol and the article to check whether all prespecified outcomes in the protocol are adequately reported in the article. Because no protocol was published and no trial was registered in our sample, we could not assess this item appropriately and decided to disregard it. Each domain was rated as low, high, or unclear for risk of bias (see web extra appendix 1). The domains for blinding and incomplete outcome data were assessed at the outcome level and thus corresponded to the selected outcome. We summarised the risk of bias for the selected outcome within each trial (across domains). The overall risk of bias was rated as low (low risk of bias for all domains), high (high risk of bias for one or more domains), or unclear (unclear risk of bias for one or more domains).

Results of individual trials

Because errors in data extraction are common for continuous outcomes,^{13 20} five statisticians or epidemiologists extracted data on outcomes from the original reports of trials. We extracted the number of analysed patients and, according to the reporting, the mean baseline, final, and change score values (and associated standard deviations) for both the experimental and control arms. The abstracted data were compared with those abstracted in the meta-analysis report. If the authors of the meta-analysis report used published data only and if differences were noted, a reviewer checked the results in the trial reports and in the meta-analysis report with the help of a statistician and epidemiologist if needed to reach consensus. If the meta-analysis report specified that the results concerned unpublished data, we used outcome data abstracted in the meta-analysis report. As standard deviations were missing in three meta-analyses (totalling seven trials,²¹ two trials,²² and one trial²³) we followed the imputation method used by the author of the original meta-analyses—that is, we imputed standard deviations from other studies in the same meta-analysis.²⁴

Statistical analysis

Estimation of treatment effect estimates within each meta-analysis

For each trial we analysed the continuous outcome according to the analytical method used in the Cochrane meta-analysis, by analysing final values or change scores as the outcome. We estimated intervention effects as standardised mean differences—the difference in mean outcome between groups divided by a pooled standard deviation within groups. Hedges’ correction for small sample size was applied, whereby we multiplied the effect measure by a correction factor. Because of differences in the direction of scales, some standardised mean differences were multiplied by −1 so that a negative effect size (standardised mean difference <0) always indicated a beneficial effect for the experimental intervention. If a trial with multiple intervention groups was included in a meta-analysis, we combined the groups for a single pairwise comparison.¹⁹ Within each meta-analysis we used DerSimonian and Laird random effects models to combine intervention effects across trials. To assess heterogeneity across trials we used the Cochran χ² homogeneity test, Higgins’ I² coefficient, and the between trial variance estimate τ².

Meta-epidemiological analysis

The meta-epidemiological analysis relied on previously described methodology.⁹ For each meta-analysis we estimated the difference in standardised mean differences between single centre and multicentre trials using random effects metaregression to incorporate heterogeneity between trials. Then we synthesised the differences in standardised mean differences across meta-analyses using a random effects meta-analysis model. Results are reported as the mean difference in standardised mean differences between single centre and multicentre trials, with associated 95% confidence intervals. A combined difference in standardised mean differences of less than 0 indicated that on average single centre trials showed larger treatment effects than did multicentre trials. We used the I² statistic, Cochran’s Q χ² test, and the between meta-analyses variance τ² to assess heterogeneity across differences in standardised mean differences. The type of intervention (pharmacological versus non-pharmacological) was a prespecified subgroup analysis. We carried out an interaction test using a random effects metaregression model to assess whether the difference in standardised mean differences varied by type of intervention.

Sensitivity analyses

To control for potential confounding we adjusted the metaregression models for important trial characteristics and re-evaluated the combined difference in standardised mean differences.

Because single centre trials may be more prone to publication bias than multicentre trials, we adjusted the metaregression model on the number of patients randomly allocated as a continuous variable, and as a binary variable with a cut-off of 100 patients per arm; a recent study²⁵ showed that results were more beneficial in trials with fewer than 100 patients per arm than in trials with more than 100 patients per arm.

To control for potential confounding by methodological quality and funding, we adjusted on the different domains of the risk of bias tool (random sequence generation and allocation concealment, blinding, incomplete outcome data), as well as the overall risk of bias within trials and the funding source (public, private, or not reported).

All analyses were done using Stata MP v10.0 with the metan and metareg subroutines.

Results

Characteristics of selected meta-analyses

Among 539 potentially eligible meta-analyses from the Cochrane database of systematic reviews, 26 were selected.^{21 22 23 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48} These meta-analyses included 306 trials, and 292 trials contributed to the analysis after exclusions (fig 1).

Fig 1 Selection of meta-analyses and randomised controlled trials

The meta-analyses concerned a wide range of medical areas. (See web extra appendix 2 for the characteristics of the meta-analyses.) All but three meta-analyses had a subjective outcome. The median number of trials included per meta-analysis was 9 (range 5-33), and the median number of patients analysed per meta-analysis was 719 (range 184-4683). The combined standardised mean difference varied from −1.45 to −0.04 (median −0.38). Fourteen meta-analyses (52%) showed substantial heterogeneity (I² ≥50%).

Sixteen meta-analyses involving 173 trials assessed non-pharmacological interventions. All but one concerned psychological or educational interventions. The median number of trials was 8 (range 5-33), and the median number of patients analysed was 546 (184-4683).

Ten meta-analyses involving 119 trials assessed pharmacological interventions. The median number of trials was 12 (range 5-27), and the median number of patients analysed was 1051 (310-2225).

Characteristics of single centre and multicentre trials

Among the 292 trials contributing to the analysis, 177 were of single centre status and 115 multicentre status. Information about the centre status of the trial was unclear in 21 articles, so the corresponding author was contacted, and information was obtained for nine. A further eight trials were classified using the number of ethics committees and the affiliations of authors. The four remaining trials were excluded from the analysis.

In total, 63% of single centre and 54% of multicentre trials assessed a non-pharmacological intervention. Multicentre trials tended to be published more recently than single centre trials. The median number of randomised patients was 50 (interquartile range 30-77) for single centre trials and 122 (60-235) for multicentre trials (P=0.02). More multicentre trials than single centre trials showed a low risk of bias for sequence generation (52% v 36%, P=0.15) and allocation concealment (30% v 12%, P=0.07); 59% of single centre trials and 64% of multicentre trials showed a high risk of bias for incomplete outcome data. Single centre trials had a lower rate of low overall risk of bias than did multicentre trials (3% v 9%, P=0.23; table).

Characteristics of single centre and multicentre randomised controlled trials. Values are numbers (percentages) unless stated otherwise

Characteristics	Single centre trials (n=177)	Multicentre trials (n=115)	P value*
Interventions:
Pharmacological	66 (37)	53 (46)	0.35
Non-pharmacological	111 (63)	62 (54)
Funding:
Public	60 (34)	42 (36)	0.13
Private	38 (21)	41 (36)
Not reported	79 (45)	32 (28)
Median (interquartile range) No of patients randomly assigned	50 (30-77)	122 (60-235)	0.02
No of patients randomly assigned:
≤50	90 (51)	22 (19)	<0.001
51-100	56 (32)	25 (22)
101-200	20 (11)	33 (29)
>200	11 (6)	35 (30)
Year of publication:
1957-89	35 (20)	9 (8)	0.17
1990-9	53 (30)	47 (41)
2000-8	89 (50)	59 (51)
Risk of bias tool:
Sequence generation
High risk of bias	0 (0)	1 (1)	0.15
Low risk of bias	63 (36)	60 (52)
Unclear risk of bias	114 (64)	54 (47)
Allocation concealment:
High risk of bias	22 (12)	13 (11)	0.07
Low risk of bias	22 (12)	34 (30)
Unclear risk of bias	133 (76)	68 (59)
Blinding:
High risk of bias	90 (51)	49 (43)	0.39
Low risk of bias	87 (49)	66 (57)
Unclear risk of bias	0 (0)	0 (0)
Incomplete outcome data:
High risk of bias	104 (59)	74 (64)	0.46
Low risk of bias	30 (17)	24 (21)
Unclear risk of bias	43 (24)	17 (15)
Overall risk of bias:
High risk of bias	141 (80)	96 (83)	0.23
Low risk of bias	5 (3)	10 (9)
Unclear risk of bias	31 (17)	9 (8)

*P values associated with cluster adjusted χ² or t tests.

Estimates of treatment effect differences between single centre and multicentre trials

Treatment effects were on average more beneficial in single centre trials than in multicentre trials (combined difference in standardised mean differences −0.09, 95% confidence interval −0.17 to −0.01, P=0.04; fig 2). The estimated differences in standardised mean differences were negative for 19 meta-analyses and positive for seven meta-analyses. Heterogeneity across individual meta-analyses was low (I²=0%, between meta-analysis variance τ²=0.00).

Fig 2 Difference in intervention effect estimates between single centre and multicentre randomised controlled trials. A combined difference in standardised mean differences <0 indicates that single centre trials yielded larger estimates of intervention effect than did multicentre trials

Subgroup analysis

For the 16 meta-analyses assessing non-pharmacological interventions, the combined difference in standardised mean differences was −0.08 (95% confidence interval −0.18 to 0.01, P=0.07, I²=0%, between meta-analysis variance τ²=0.00). For the 10 meta-analyses assessing pharmacological interventions, the combined difference in standardised mean differences was −0.11 (95% confidence interval −0.29 to 0.08, P=0.27, I²=0%, between meta-analysis variance τ²=0.00). The test of interaction between non-pharmacological and pharmacological trials was not significant (P=0.84).

Sensitivity analyses

Figure 3 shows the results of sensitivity analyses. Adjustment for the different domains of the risk of bias tool and the overall risk of bias yielded similar estimates, with slightly wider confidence intervals, crossing 0 for sequence generation, blinding, incomplete outcome data, and overall risk of bias (combined difference in standardised mean differences for overall risk of bias −0.09, 95% confidence interval −0.17 to 0.00). Adjustment on funding yielded consistent results (combined difference in standardised mean differences −0.09, 95% confidence interval −0.18 to −0.01). Adjustment for the number of patients randomly allocated attenuated the difference between single centre and multicentre trials: −0.06 (−0.16 to 0.04) as a continuous variable and −0.08 (−0.17 to 0.01) as a binary variable, with the cut-off of 100 patients per arm.

Fig 3 Sensitivity analyses showing difference in intervention effect estimates between single centre and multicentre trials adjusted for sample size, domains of risk of bias tool, and funding

Discussion

We carried out a meta-epidemiological study of 26 meta-analyses of 292 randomised controlled trials comparing estimates of intervention effect for single centre and multicentre trials assessing a continuous outcome. Our results showed on average slightly larger intervention effect estimates in single centre trials than in multicentre trials, with low heterogeneity across individual meta-analyses. Similar results were found in the prespecified subgroups of non-pharmacological and pharmacological interventions. Results tended to be consistent after adjusting for risk of bias and funding source. Adjustment for the number of patients randomly allocated resulted in an attenuation of this difference.

The magnitude of the observed difference in standardised mean differences between single centre and multicentre trials may seem small. The average difference corresponded to about one quarter of a typical effect found for interventions in our collection of meta-analyses: the combined standardised mean difference varied from −1.45 to −0.04 across the 26 meta-analyses, with a median of −0.38.

Strengths and limitations of the study

Our selection of meta-analyses concerned a large number of recently published meta-analyses covering a wide range of medical areas, for a more generalisable sample than when focusing on a particular topic. We selected meta-analyses published in the Cochrane database of systematic reviews because they are of higher methodological quality, are better reported, and have fewer conflicts of interest than do non-Cochrane meta-analyses.^{14 15 16} Overall we found low heterogeneity across individual meta-analyses. We carried out several sensitivity analyses by adjusting metaregression models on important covariates, which is not commonly done in meta-epidemiological studies. Our results tended to be consistent after adjustment for sample size, risk of bias, and funding source.

Our study contains some limitations. Firstly, our search strategy may not have been exhaustive, but our aim was to provide a representative sample of meta-analyses. We excluded updates of meta-analyses because, although they are the most complete, they might have less detail than the original report. Nevertheless, this choice should not have biased our results. Secondly, our sensitivity analyses to control for meta-confounding may also have limitations. The risk of bias domains were poorly reported and discrepancies may exist between the reported quality and what actually occurred.⁴⁹ In particular, we were unable to appropriately assess the risk of selective outcome reporting bias because none of the selected trials was registered and study protocols were not available, which precluded assessing whether prespecified (primary and secondary) outcomes were reported. Moreover, the dimensions of within trial risk of bias are possibly associated with each other and with other trial characteristics, such as single centre status. However, methods to correct for multiple dimensions of quality require further development.

Comparison with other studies

Heterogeneity of intervention effect estimates between single centre and multicentre trials has been poorly explored⁵⁰ and is not taken into account when assessing the risk of bias or the level of evidence in trials.¹⁰ In 1986 a study⁵⁰ assessed a sample of 246 comparative clinical trials of cancer (not necessarily randomised) and found by a random effects model adjusted for sample size that single centre trials tended to show larger effects for survival than did multicentre trials.⁵⁰ More recently, a meta-epidemiological study¹¹ of 48 meta-analyses showed significantly larger intervention effect estimates for binary outcomes in single centre trials than in multicentre trials, which persisted after adjustment for the number of patients randomly allocated and risk of bias. The two collections of meta-analyses differed in terms of medical specialties and types of outcomes. In fact the collection involving binary outcomes contained a high representation of cardiovascular research, and we selected the most objective outcomes (such as all cause mortality or the result of a biological test if reported), but in the collection involving continuous outcomes, several meta-analyses concerned psychological and behavioural research, and most outcomes were subjective.

Possible mechanisms

The larger intervention effects we observed for single centre trials than for multicentre trials may be explained by several mechanisms. The first may be due to “small study effects”: the tendency for smaller studies in a meta-analysis to show larger intervention effects.²⁵ As expected, we found that the single centre trials were of a smaller sample size than the multicentre trials. Our results were slightly attenuated after adjusting for the number of patients randomly allocated as a continuous variable. Nevertheless, when adjusting for the number of patients randomly allocated as a binary variable, with the previously recommended cut-off of 100 patients per arm,²⁵ we obtained similar estimates, although the confidence interval was slightly wider and crossed 0. Publication bias could also contribute to the observed differences in treatment effect estimates between single centre and multicentre trials. The relations between centre status and publication bias were explored in some studies^{51 52 53}: two found no significant association between single centre status and publication bias,^{51 53} but a more recent study found multicentre status independently associated with full publication.⁵² The association of single centre and multicentre trials and intervention effect may also be confounded by methodological quality. We found that more multicentre trials were at low risk of bias for the risk of bias domains of sequence generation and allocation concealment than single centre trials. After adjusting for each domain of the risk of bias tool as well as overall risk of bias, we obtained similar estimates with slightly wider confidence intervals, some crossing 0. The larger intervention effect could also be due to different mechanisms for selection of a more homogeneous (highly selected) population in single centre than in multicentre trials, standardised interventions, and higher expertise of teams in single centre trials.⁵⁴ Further studies are needed to explore the role and impact of these different mechanisms, but probably several reasons act together to contribute to the larger intervention effect observed in single centre trials.

Conclusions and implications

Intervention effect estimates were on average slightly larger in single centre randomised controlled trials than in multicentre randomised controlled trials assessing continuous outcomes. The reasons for these larger effect sizes in single centre trials need to be explored in further studies.

What is already known on this topic

• For many doctors the results of single centre trials are more debatable than those of multicentre trials

• A recent meta-epidemiological study of binary outcomes found that single centre trials showed larger intervention effects than did multicentre trials

• Results from trials of binary outcomes may not be extrapolated to trials of continuous outcomes because such trials often differ in medical condition, risk of bias, and sample size

What this study adds

• Intervention effect estimates were slightly larger in single centre randomised trials of continuous outcomes than in multicentre randomised trials: mean difference in standardised mean differences −0.09 (95% confidence interval −0.17 to −0.01)

• Similar estimates were obtained after adjusting for risk of bias and funding source but were slightly attenuated after adjusting for sample size

Cite this as: BMJ 2012;344:e813

Web Extra. Extra material supplied by the author

Appendix 1: definitions used for assessing risk of bias

Appendix 2: characteristics of included meta-analyses