Outcomes for patients with the same disease treated inside and outside of randomized trials: a systematic review and meta-analysis

Abstract

Background:

It is unclear whether participation in a randomized controlled trial (RCT), irrespective of assigned treatment, is harmful or beneficial to participants. We compared outcomes for patients with the same diagnoses who did (“insiders”) and did not (“outsiders”) enter RCTs, without regard to the specific therapies received for their respective diagnoses.

Methods:

By searching the MEDLINE (1966–2010), Embase (1980–2010), CENTRAL (1960–2010) and PsycINFO (1880–2010) databases, we identified 147 studies that reported the health outcomes of “insiders” and a group of parallel or consecutive “outsiders” within the same time period. We prepared a narrative review and, as appropriate, meta-analyses of patients’ outcomes.

Results:

We found no clinically or statistically significant differences in outcomes between “insiders” and “outsiders” in the 23 studies in which the experimental intervention was ineffective (standard mean difference in continuous outcomes −0.03, 95% confidence interval [CI] −0.1 to 0.04) or in the 7 studies in which the experimental intervention was effective and was received by both “insiders” and “outsiders” (mean difference 0.04, 95% CI −0.04 to 0.13). However, in 9 studies in which an effective intervention was received only by “insiders,” the “outsiders” experienced significantly worse health outcomes (mean difference −0.36, 95% CI −0.61 to −0.12).

Interpretation:

We found no evidence to support clinically important overall harm or benefit arising from participation in RCTs. This conclusion refutes earlier claims that trial participants are at increased risk of harm.

When people are asked to participate in a randomized controlled trial (RCT), it is natural for them to ask several questions in return. How safe are these treatments? How many extra visits and tests must I undergo? Will the researchers keep my family doctor informed about what’s going on? What outcomes are to be measured, and do they include ones that are of interest to me as a patient?

These multiple questions can be summarized as follows: Would I fare better being treated within the trial (as an “insider”) or in routine clinical care outside it (as an “outsider”)? Patients may ask this question in 1 of 2 ways. The first is highly specific: “Am I better off receiving this specific treatment as an insider or as an outsider?” Alternatively, they might ask a more general question: “Am I better off having my illness managed, regardless of the specific treatment I would receive, as an insider or as an outsider?” These questions are highly appropriate, and both deserve to be asked and answered,^1,2 especially given that nonsystematic reviews have suggested a possible “inclusion benefit” from participating in trials.³

These 2 specific patient questions are analogous to those posed by researchers asking whether treatments do more good than harm when applied under “ideal” circumstances (in explanatory trials) or in the “real world” of routine health care (in pragmatic trials). Vist and colleagues answered the explanatory question when their earlier review⁴ found no advantage or disadvantage from receiving the same treatment inside or outside an RCT. Left unanswered, however, was the broader, more pragmatic question. In our experience, trial participants are often offered new, as-yet-untested treatments that would not be available to them outside the trial. This review looks at the dilemma faced by these patients, which needs to be addressed before general conclusions can be drawn about trial safety.

Methods

Data sources and searches

We searched the following databases: MEDLINE (1966 to November 2010), Embase (1980 to November 2010), Cochrane Central Register of Controlled Trials (CENTRAL; 1960 to last quarter of 2010) and PsycINFO (1880 to November 2010). The search strategy for each database is available upon request to the corresponding author. Studies were eligible for inclusion if they reported the same set of outcomes for “insiders” and “outsiders,” either simultaneously or within 2 months, where “insiders” were patients with a particular diagnosis who entered an RCT (whether treated with the intervention or a comparator) and “outsiders” were patients with the same diagnosis who did not enter the RCT. To validate our search, we compared our yield with the list of articles reviewed by Vist and colleagues.⁴

Study selection

Working in pairs, we reviewed the resulting titles and abstracts to screen for eligibility. Two reviewers independently screened the full text of eligible articles, with an independent third adjudicator resolving disagreements. Agreement was summarized with a weighted kappa coefficient.

Data extraction

Our primary outcome was mortality, and secondary outcomes included patient-reported or other clinically important outcomes. We calculated the relative risk (RR), unless count data were not reported, in which case we extracted the authors’ RR. We used adjusted RRs whenever they were reported.⁵ When RRs could not be calculated, we assumed that the reported odds ratios (ORs) approximated the RR for low event-rate outcomes.

For continuous outcomes, we extracted mean between-group differences and their standard deviations. We created rules for calculating missing outcomes using various statistical measures that were reported (Table 1).

Table 1:

Assumptions and imputations used to calculate data if missing from published report

Data needed	Data available	Assumptions/imputations
SD of the difference	SE of the difference	Multiply SE by square root of sample size
	Confidence interval around the difference	For n ≥ 100, assume standard normal distribution For n < 100, assume t distribution
SE of the difference	p value for mean difference	Convert p value to t value at same degrees of freedom; divide mean difference by t value
Final score	Baseline and change scores	Add or subtract the change score from baseline value
SD of final scores	SD of baseline and change scores	Sum baseline and change variances
SD	Range	No appropriate conversion possible

Note: SD = standard deviation, SE = standard error.

Prespecified causes of heterogeneity

We used the I² statistic to measure the extent of heterogeneity between studies, where I² values of 25%, 50% and 75% indicated low, medium and high heterogeneity, respectively.⁶ In addition, we constructed a priori hypotheses to potentially explain between-study heterogeneity, based on differences in types of outcomes, methodologic quality, types of care provided, potential for detection bias (due to differential follow-up or use of better diagnostic tools), potential for exclusion bias (if patients were excluded after enrolment because of characteristics related to outcome), potential for selection bias (due to imbalance of baseline characteristics), medical specialty and treatments provided.

In particular, we proposed 6 subgroups to explain observed heterogeneity due to treatment effect:

1. when the randomized experimental intervention given to “insiders” was effective (i.e., the outcome was statistically significantly superior to the comparator), and “outsiders” received that same intervention or comparator

2. when the randomized experimental intervention was effective, and “outsiders” received that same effective intervention only (without the comparator that was provided within the RCT)

3. when the randomized experimental intervention was effective, and “outsiders” received the less effective comparator intervention only (without the experimental intervention provided within the RCT)

4. when the randomized experimental intervention was effective, and “outsiders” received a different intervention (this subgroup acted as a positive control for the current analysis, since we anticipated better outcomes in the RCT group)

5. when the randomized experimental and comparator interventions generated equivalent outcomes, with no further subdivision of this group (because any differences in outcomes between those treated inside and outside the RCT could be attributed to a trial effect)

6. when insufficient information was provided about the effectiveness of the treatment in the trial and/or insufficient details were provided about the interventions received by “outsiders”

Data synthesis and analysis

Statistical calculations were performed with SPSS (version 20).⁷ Forest plots and funnel plots were created using Review Manager (version 5.1).⁸ When event counts were available, we used the Mantel–Haenszel method to estimate overall RR.⁹ If a study had a zero event rate in one group, we added a 0.5 correction to all cells. If only estimates of effect size and standard errors were provided, we used the generic inverse-variance meta-analysis function of Review Manager 5.1. We used the random-effects model to summarize outcomes.⁹

We first separated the studies into 2 groups according to whether randomization was applied in determining whether potential participants would be “insiders” or “outsiders.” Next, we separated studies by type of outcome: continuous or dichotomous, with the latter being further subdivided as nonmortality or mortality.

We created a funnel plot and conducted a sensitivity analysis to determine the stability of our conclusions.

Results

Summary of evidence

Following elimination of duplicate records and exclusions on the basis of initial screening and full-text review, 147 articles met our eligibility criteria and provided sufficient information to be included in our analysis (Figure 1).^10–156 Details for the 576 articles excluded after full-text review, including reasons for exclusion, are available upon request. The eligibility of the remaining 74 articles was uncertain, and they were not included in the analysis.

Figure 1:

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow chart of studies identified and included in the analysis.^10–156 The reasons for exclusions at screening and full-text review are available upon request to the corresponding author.

For full-text screening, the calculated average of the weighted kappa for eligibility was 0.68. There was 83% raw agreement between reviewers in the data-extraction phase for outcomes.

In 5 of the 147 eligible studies, patients were randomly assigned to become “insiders” and “outsiders.”^{38,41,86,87,141} In the remaining 142 studies, patients became part of the “outsiders” group for a variety of reasons. Table 2 presents the details about each included study.

Table 2:

Characteristics of included studies

Study	No. of “insiders”	No. of “outsiders”	Specialty	Intervention	Care setting
Akaza et al. 199510	107	13	Oncology	Vaccination	Hospital
Amar et al. 199711	70	40	Surgery	Anti-arrhythmic drugs	Hospital
Andersson et al. 200312	24	8	Family	Counselling	Home
Antman et al. 198513	42	42	Oncology	Chemotherapy	Cancer centre
Ashok et al. 200214	229	45	Ob/gyn	Abortion	Hospital
Bain et al. 200115	36	62	Anesthesia	Sedatives	Operating room
Bakker et al. 200016	113	24	Psychiatry	Counselling	Outpatient clinic
Balmukhanov et al. 198917	108	287	Oncology	Radiation	Hospital
Bannister et al. 200118	202	38	Anesthesia	Analgesics	Operation room
Bedi et al. 200019	85	164	Family	Counselling	Family clinic
Bell and Palma 200020	59	56	Ob/gyn	Exercise program	Unclear
Bhattacharya et al. 199821	92	68	Ob/gyn	Longer hospital stay	Hospital
Biasoli et al. 200822	52	41	Oncology	Chemotherapy	Hospital
Biederman et al. 198523	24	18	Psychiatry	Drugs	Inpatient
Bijker et al. 200224	268	155	Oncology	Excision	Unclear
Blichert-Toft et al. 198825	619	136	Oncology	Mastectomy	Surgical departments
Blumenthal et al. 199726	66	38	Cardiology	Exercise	Hospital
Boesen et al. 200727	258	137	Oncology	Psychoeducation	Outpatient clinic
Boezaart et al. 199828	240	136	Anesthesia	Drugs	Private hospital
Brinkhaus et al. 200829	902	3 888	Allergy	Acupuncture	Unclear
Caplan and Buchanan 198430	29	46	ID	Drugs	Hospital
CASS 198431	779	1 309	Cardiology	Surgery	Unclear
Chauhan et al. 199232	38	15	Ob/gyn	Amnio-infusion	Labour unit
Chesebro et al. 198333	351	183	Internal	Anticoagulant	Unclear
Chilvers et al. 200134	98	207	Family	Counselling v. drugs	Outpatient
Clagett et al. 198435	29	28	Surgery	Surgery	Unclear
Clapp et al. 198936	115	85	ID	Drugs	Pediatric hospital
Clemens et al. 199237	20 744	21 943	ID	Vaccine	Research centre
Cooper et al. 199738	–	–	Ob/gyn	Surgery	Hospital
Cowchock et al. 199239	20	13	Ob/gyn	Drugs	Unclear
Creutzig et al. 199340	31	25	Oncology	Radiation	Unclear
Dahan et al. 198641	–	–	Research design	Informed consent	Unclear
Dalal et al. 200742	100	84	Cardiology	Rehabilitation	Hospital, home
Decensi et al. 200343	116	29	Oncology	Drugs	Hospital
Detre et al. 199944	343	299	Cardiology	Surgery	Hospital
Eberhardt et al. 199646	43	37	Rheumatology	Drugs	Hospital
Edsmyr et al. 197847	18	9	Urology	Drugs	Unclear
Ekstein et al. 200248	91	1 202	Cardiology	Surgery	Hospital
Emery et al. 200349	168	49	Ob/gyn	Counselling	Hospital
Euler et al. 200550	58	14	Pediatrics	Diet	Unclear
Feit et al. 200051	1 169	1 336	Cardiology	Surgery	Hospital
Forbes and Collins 200052	102	88	Gastrointestinal	Drugs	Hospital
Franz et al. 199553	179	62	Nutrition	Diet	Unclear
Gall et al. 200754	46	41	Gastrointestinal	Follow-up	Hospital
Girón et al. 201055	24	45	Psychiatry	Family intervention	Mental health centre
Goodkin et al. 198756	27	24	Neurology	Drugs	Unclear
Gossop et al. 198657	20	40	Addiction	Inpatient treatment	Unclear
Grant et al. 200858	299	375	Gastrointestinal	Surgery	Hospital
Gunn et al. 200059	122	308	Pediatrics	Home support	Hospital
Helsing et al. 199860	47	97	Oncology	Supportive care	Unclear
Henriksson and Edhaq 198661	91	9	Urology	Surgery	Unclear
Heuss et al. 200462	74	40	Gastrointestinal	Sedation	Hospital
Hoh et al. 199863	13	39	Nutrition	Diet	Hospital
Howard et al. 200964	44	28	Psychiatry	Crisis houses	Hospital
Howie et al. 199765	77	63	Ob/gyn	Abortion	Hospital
Jena et al. 200866	2 792	10 410	Alternative	Acupuncture	Unclear
Jensen et al. 200367	897	294	Geriatrics	Hormones	Hospital
Kane 198868	59	116	Orthopedics	Bone growth stimulator	Unclear
Karande et al. 199969	63	57	Ob/gyn	IVF	Infertility clinic
Kayser et al. 200870	31	44	Travel	Drugs	Unclear
Kendrick et al. 200171	394	50	Technology	Radiography	General practice or hospital
Kieler et al. 199872	526	4 801	Ob/gyn	Ultrasonography	Antenatal care clinic
King et al. 200073	165	106	Psychiatry	Counselling	Unclear
Kirke et al. 199274	351	106	Ob/gyn	Folic acid	Unclear
Koch-Henriksen et al. 200675	224	74	Neurology	Drugs	Unclear
Lansky and Vance 198376	55	59	Psychology	Diet and exercise	Unclear
Lichtenberg et al. 200877	217	153	Psychiatry	Case management	Unclear
Lidbrink et al. 199578	20 000	7 785	Oncology	Mammography	Unclear
Link et al. 199179	36	77	Oncology	Drugs	Unclear
Liu et al. 200980	169	163	Alternative	Salvia	Delivery room
Lock et al. 201081	40	303	Surgery	Tonsillectomy	ENT department
Loeffler et al. 199745	100	21	Oncology	Radiotherapy	Hospital
Luby et al. 200282	162	79	ID	Antibacterial soap	Home
Macdonald et al. 200783	5	48	Nephrology	Drugs	Unclear
MacLennan et al. 198584	96	73	Ob/gyn	Relaxin	IVF clinic
MacMillan et al. 198685	107	49	Psychiatry	Drugs	Unclear
Mahon et al. 199686	–	–	Respirology	Drugs	Hospital
Mahon et al. 199987	–	–	Respirology	Drugs	Primary care
Marcinczyk et al. 199788	54	29	Vascular surgery	Endarterectomy	Hospital
Martin 199489	46	54	Gastrointestinal	Antacid	Hospital
Martínez-Amenos et al. 199090	589	133	Family	Education	Primary care
Masood et al. 200291	96	14	Urology	Nitrous oxide – oxygen	Urology department
Matilla et al. 200392	137	166	ENT	Surgery	Study clinic
Mayo Group 199293	71	87	Vascular surgery	Endarterectomy	Unclear
McCaughey et al. 199894	19	13	Pediatrics	Hormone	Hospital
McKay et al. 199895	101	51	Psychology	Day hospital	Hospital
McKay et al. 199596	40	80	Psychology	Day hospital	Addiction recovery unit
Melchart et al. 200297	26	80	Alternative	Acupuncture	Hospital
Moertel et al. 198498	62	10	Oncology	Chemo + radiation	Hospital
Mori et al. 200699	712	158	Gastrointestinal	Endoscopy	Hospital
Morrison et al. 2002100	454	302	Cardiology	Surgery	Hospital
Nagel et al. 1998101	115	95	Ob/gyn	Amniocentesis	Hospital
Neldam et al. 1986102	978	349	Ob/gyn	Fetal heart monitor	Hospital
Nicolaides et al. 1994103	488	812	Ob/gyn	Chorionic villus sampling	Research centre
Ogden et al. 2004104	285	47	Orthopedics	Shock wave treatment	Unclear
Palmon et al. 1996105	50	10	Critical care	Carbon dioxide monitor	Neuroradiology centre
Panagopoulou et al. 2009106	148	66	Psychology	Diary writing	Clinic
Paradise et al. 1984107	42	28	Surgery	Tonsillectomy	Hospital
Petersen et al. 2007108	79	33	Orthopedics	Hip replacement	Hospital
Raistrick et al. 2005109	174	225	Addiction	Drugs	Addiction recovery unit
Reddihough et al. 1998110	19	22	Physiotherapy	Education	Unclear
Rigg et al. 2000111	455	237	Anesthesia	Analgesia	Hospital
Rørbye et al. 2005112	105	727	Ob/gyn	Abortion	Hospital
Rosen et al. 1987113	98	44	Anesthesia	Nitrous oxide	Hospital
Salisbury et al. 2002114	253	129	Family	School-based clinics	Unclear
Sesso et al. 2002115	22 071	11 152	Cardiology	ASA	Unclear
Shain et al. 1989116	155	98	Ob/gyn	Contraception	Unclear
Smith and Arnesen 1990117	1 214	270	Internal	Warfarin	Cardiology centre
Smuts et al. 2003118	16	37	Pediatrics	Diet	Unclear
Stecksén-Blicks et al. 2008119	115	64	Dentistry	Lozenges	Clinic
Stern et al. 2003120	555	1 788	Ob/gyn	Anticoagulants	Hospital
Stith et al. 2004121	19	4	Psychology	Couple therapy	Unclear
Stockton and Mengersen 2009122	57	21	Rehab	Physiotherapy	Hospital
Strandberg et al. 1995123	910	489	Cardiology	Health checks	Hospital
Suherman et al. 1999124	83	29	Ob/gyn	Contraception	Unclear
Sullivan et al. 1982125	144	25	Oncology	Radiotherapy	Unclear
Sundar et al. 2008126	136	45	ID	Drugs	Inpatient unit
Taddio et al. 2006127	98	20	Pediatrics	Analgesics	Hospital
Tanai et al. 2009128	100	19	Oncology	Drugs	Hospital
Tanaka et al. 1994129	30	10	Anesthesia	Drugs	Unclear
Taplin et al. 1986130	63	30	Dermatology	Permethrin cream	Unclear
Tenenbaum et al. 2002131	3 122	380	Cardiology	Drugs	Hospital
Toprak et al. 2005132	30	15	Ob/gyn	Hormone replacement	Clinic
Underwood et al. 2008133	187	271	Geriatrics	Ibuprofen	Primary care
Urban et al. 1999134	55	24	Cardiology	Early revascularization	Unclear
Van et al. 2009136	40	45	Psychiatry	Therapy	Unclear
van Bergen et al. 1995135	350	587	Cardiology	Anticoagulant	Centre
Verdonck et al. 1995137	69	37	Oncology	Chemotherapy	Unclear
Vind et al. 2009138	256	297	Geriatrics	Fall prevention	Unclear
Walker et al. 1986139	98	37	Surgery	Antibiotics	Unclear
Wallage et al. 2003140	178	28	Ob/gyn	Anesthesia	Hospital
Watzke et al. 2010141	180	97	Psychiatry	Counselling	Inpatient unit
Welt et al. 1981142	23	40	Ob/gyn	Drugs	Unclear
West et al. 2005143	86	322	Critical care	Magnesium sulphate	Unclear
Wetzner et al. 1979144	34	64	Radiology	Contrast	Unclear
Wieringa-de Waard et al. 2002146	122	305	Ob/gyn	Evacuation	Clinic
Williford et al. 1993147	395	199	Nutrition	Diet	Unclear
Witt et al. 2006a148	543	2 481	Alternative	Acupuncture	Unclear
Witt et al. 2006b149	3 036	4 686	Alternative	Acupuncture	Unclear
Witt et al. 2006c150	2 518	3 901	Alternative	Acupuncture	Unclear
Witt et al. 2008151	57	21	Alternative	Acupuncture	Unclear
Woodhouse et al. 1995152	194	145	Cardiology	Adrenaline dose	Hospital
World Health Organization 1988145	40	32	Ob/gyn	Contraception	Unclear
Wyse et al. 1991153	1 672	318	Cardiology	Anti-arrhythmic drugs	Unclear
Yamamoto et al. 1992154	31	92	Gastrointestinal	Esophageal dilator	Unclear
Yamani et al. 2005155	23	33	ID	Vaccine	Unclear
Yersin et al. 1996156	20	10	Addiction	Counselling	Unclear

Note: ASA = acetylsalicylic acid, CASS = Coronary Artery Surgery Study, chemo = chemotherapy, ENT = ear, nose and throat, Family = family medicine, ID = infectious diseases, “insider” = patient receiving treatment within a randomized controlled trial, IVF = in vitro fertilization, Ob/gyn = obstetrics and gynecology, “outsider” = patient receiving treatment via routine clinical care outside the randomized controlled trial.

We analyzed a total of 48 continuous outcomes and 99 dichotomous outcomes; of the dichotomous outcomes, 74 were nonmortality outcomes, 4 were recurring outcomes (such as relapse rates), and 21 were mortality outcomes.

Risk of bias

Sources of risk of bias are detailed by individual study in Appendix 1 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.131693/-/DC1). In terms of detection bias, about two-thirds of the studies (n = 100) employed identical follow-up strategies for “insiders” and “outsiders.” In terms of exclusion bias affecting “insiders,” 67 studies had no exclusions, 1 study employed a deliberate but appropriate exclusion, and 74 studies inappropriately excluded “insiders” unequally between treatment groups; for the remaining 5 studies, the details were unclear. Forest plots based on subgroups created for each of these sources of bias did not change the results described below.

Replication of earlier studies

As a method of calibrating our search strategies and statistical methods, we carried out analyses of our dataset that were restricted to “insiders” and “outsiders” receiving identical treatments. These restricted analyses replicated the results of previous studies by Vist and colleagues⁴ and Gross and associates.¹⁵⁷

Outcomes for studies with participants not randomized as “insiders” or “outsiders”

Our initial pooled analyses revealed a high degree of between-study heterogeneity (p < 0.001, I ² = 84% for studies with dichotomous mortality outcomes; p < 0.001, I² = 70% for studies with dichotomous nonmortality outcomes; p < 0.001, I ² = 88% for studies with continuous outcomes). In total, mortality was determined for 53 714 “insiders” and 25 817 “outsiders” (see Table 3 and Appendix 2, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.131693/-/DC1). Dichotomous nonmortality outcomes were reported for 30 253 “insiders” and 30 000 “outsiders” (see Table 4 and Appendix 3, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.131693/-/DC1). We present the results of our nonrandomized continuous outcomes and randomized comparisons according to treatment effects, by presenting the subgrouping that left the least amount of remaining heterogeneity. All other forest plots are available upon request.

Table 3:

Summary of meta-analyses for studies with mortality as an outcome, without randomization of potential participants as “insiders” v. “outsiders” (subgroups based on effectiveness of trial treatment)

Subgroup	No. of trials	No. of events/no. of patients		Weight, %	RR (95% CI)	I2, %
		RCT	Cohort
Trial treatment effective, same treatment and comparator given to “outsiders”	3	273/2 000	251/2 447	15.2	1.30 (0.78 to 2.16)	79
Trial treatment effective, treatment only given to “outsiders”	1	39/47	76/97	7.0	1.06 (0.90 to 1.25)	NA
Trial treatment effective, comparator only given to “outsiders”	1	53/62	7/10	5.0	1.22 (0.80 to 1.86)	NA
Trial treatment effective, neither treatment nor comparator given to “outsiders”	2	377/2 124	116/759	12.4	1.13 (0.43 to 2.94)	96
Trial treatment ineffective	9	478/22 306	472/10 328	44.2	0.73 (0.50 to 1.05)	92
Trial effect or treatment given unknown	5	640/27 175	192/12 176	16.2	0.83 (0.59 to 1.18)	60
Overall	21	1 860/53 714	1 114/25 817	100.0	0.92 (0.78 to 1.07)	84

Note: CI = confidence interval, NA = not applicable, RCT = randomized controlled trial, RR = relative risk.

Table 4:

Summary of meta-analyses for studies with dichotomous nonmortality outcomes, without randomization of potential participants as “insiders” v. “outsiders” (subgroups based on effectiveness of trial treatment)

Subgroup	No. of trials	No. of patients		Weight, %	RR (95% CI)	I2, %
		RCT	Cohort
Trial treatment effective, same treatment and comparator given to “outsiders”	9	1 316	1 768	11.9	1.06 (0.81 to 1.40)	54
Trial treatment effective, treatment only given to “outsiders”	3	382	168	4.6	1.68 (0.80 to 3.56)	84
Trial treatment effective, comparator only given to “outsiders”	1	589	133	3.3	0.76 (0.62 to 0.92)	NA
Trial treatment effective, neither treatment nor comparator given to “outsiders”	6	369	269	8.0	0.99 (0.61 to 1.63)	77
Trial treatment ineffective	37	5 513	4 915	60.3	0.96 (0.89 to 1.04)	58
Trial effect or treatment given unknown	13	22 084	22 747	12.0	1.06 (0.65 to 1.70)	83
Overall	69	30 253	30 000	100.0	0.99 (0.92 to 1.08)	70

Note: CI = confidence interval, NA = not applicable, RCT = randomized controlled trial, RR = relative risk.

Results for clinically relevant subgroups

The results for continuous outcomes are summarized by subgroup in Table 5 (see also Appendix 4, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.131693/-/DC1).

Table 5:

Summary of meta-analyses for studies with continuous outcomes, without randomization of potential participants as “insiders” v. “outsiders” (subgroups based on effectiveness of trial treatment)

Subgroup	No. of trials	No. of patients		Weight, %	Standardized mean difference (95% CI)	I2, %
		RCT	Cohort
Trial treatment effective, same treatment and comparator given to “outsiders”	7	6 626	2 293	19.0	0.04 (−0.04 to 0.13)	37
Trial treatment effective, treatment only given to “outsiders”	3	1 391	5 072	9.4	0.51 (0.21 to 0.82)	95
Trial treatment effective, comparator only given to “outsiders”	4	5 794	9 035	11.8	0.16 (0.07 to 0.25)	74
Trial treatment effective, neither treatment nor comparator given to “outsiders”	9	649	188	11.3	−0.36 (−0.61 to −0.12)	43
Trial treatment ineffective	23	5 940	11 927	45.4	−0.03 (−0.10 to 0.04)	29
Trial effect or treatment given unknown	2	137	69	3.1	0.35 (0.02 to 0.68)	0
Overall	48	20 537	28 584	100.0	0.04 (−0.04 to 0.12)	88

Note: CI = confidence interval, RCT = randomized controlled trial.

There were 7 studies in which the randomized experimental intervention given to “insiders” (n = 6626) was effective, and “outsiders” (n = 2293) received that same intervention or the comparator. The heterogeneity was low to moderate (p = 0.2, I ² = 37%), and the pooled result indicated neither significant harm nor significant benefit attributable to being an “insider” or an “outsider” (standardized mean difference 0.04, 95% confidence interval [CI] −0.04 to 0.13).

There were 3 studies in which the randomized experimental intervention (given to 1391 “insiders”) was effective, and the 5072 “outsiders” received only that same effective intervention. In this subgroup, there was a high degree of heterogeneity (p < 0.001, I² = 95%).

There were 4 studies in which the randomized experimental intervention was effective, and “outsiders” received only the less effective comparator. In these studies, the 5794 “insiders” (those assigned to receive the active intervention or comparator) experienced a positive effect of the intervention, but the 9035 “outsiders” were offered only the ineffective comparator. In this subgroup, there was also a high degree of heterogeneity (p = 0.01, I² = 74%).

There were 9 studies in which the randomized experimental intervention had a positive effect inside the RCT, but “outsiders” received a completely different intervention or comparator. For these studies, results could be pooled for the 649 “insiders” and 188 “outsiders” (standardized mean difference −0.36, 95% CI −0.61 to −0.12, p = 0.08, I ² = 43%). In this subgroup, “insiders” fared statistically significantly better than “outsiders.”

The largest subgroup consisted of 23 studies in which the randomized experimental and comparator interventions generated equivalent outcomes. In this subgroup, the 5 940 “insiders” and 11 927 “outsiders” were given both treatments, only the control or only the experimental treatment, or completely different interventions. Heterogeneity among these studies was low to moderate (p = 0.10, I ² = 29%). The pooled result revealed neither net harm nor net benefit for “insiders” compared with “outsiders” (standardized mean difference −0.03, 95% CI −0.1 to 0.04).

For the final subgroup of 2 studies, it was unclear whether there was a treatment effect or which interventions the “outsiders” received. We requested additional information from the study authors, but as of the date of publication, were still awaiting this clarification.

Outcomes for studies with participants randomized as “insiders” or “outsiders”

In 5 studies, potential participants were randomly assigned to become “insiders” or “outsiders.” One of these studies used a continuous outcome, with no reported difference between the 180 “insiders” and 97 “outsiders” (95% CI −0.22 to 0.27). The remaining 4 studies reported dichotomous nonmortality outcomes, with a moderate degree of heterogeneity (p = 0.06, I ² = 60%). Their overall pooled effect indicated neither harm nor benefit when patients were treated inside or outside a trial (RR 0.94, 95% CI 0.56 to 1.57).

Additional analyses

Our investigation into publication bias showed a lack of smaller studies (both positive and negative) in our study. Because the included studies were symmetric around the pooled estimate, we are confident that our estimates are valid.

Our sensitivity analysis confirmed the robust nature of our imputations. Removing the studies with imputed outcomes had no significant effect on our results. Similarly, the results were not affected by clinical specialty.

Interpretation

Our study has confirmed the earlier findings of Vist and colleagues⁴ and Gross and associates,¹⁵⁷ who reported that when trial participants (“insiders”) and nonparticipants (“outsiders”) receive the same treatments, they experience similar outcomes. As such, there is neither a “trial advantage” nor a “guinea pig disadvantage” of participating in an RCT. Furthermore, we have shown that even when “insiders” and “outsiders” are offered different interventions, there is no disadvantage to trial participation.

Our findings do not support the theory of “inclusion benefits,” “protocol effects” or “care effects” proposed by other authors.^3,158 We found no differences in outcomes that could be attributed to health care workers providing additional care to “insiders,” the setting in which “insiders” were treated or the closer follow-up and attention that “insiders” receive. Had there been better care because physicians were following strict study protocol, a difference would have been detected between the groups for whom treatments were identical and would have been amplified within the subgroup of studies in which detection bias and expertise bias were most probable.

As expected, our subanalysis of “insiders” and “outsiders” who received the same treatments confirmed the results of the Vist and Gross reviews.^4,157 However, we suggest that their insistence on identical interventions for patients inside and outside of an RCT answered only a narrow, explanatory question. For our review, we posed a more pragmatic question: Will patients fare better being treated within a trial (as “insiders”) or in routine clinical care outside it (as “outsiders”), regardless of the treatment received? In other words, will they be “sacrificial guinea pigs,” or, conversely, will they enjoy an “inclusion benefit”? Or will they fare the same inside the RCT or outside it? Our pragmatic study supports the last of these options, that patients will, in general, fare just as well regardless of whether they are “insiders” or “outsiders.”

Stiller¹⁵⁹ reported a beneficial effect on mortality for “insiders.” However, that conclusion was based on simply counting the number of studies in which “insiders” had lower mortality than “outsiders,” ignoring the size of each study. As such, smaller studies (which are more prone to type II error) were weighted the same as much larger studies. Our random-effects meta-analysis took into account the size and weight of each study, and we found no such benefit from trial participation.

Limitations

Although 68% of the studies included here employed identical follow-up protocols for both “insiders” and “outsiders,” some studies did not explicitly state whether “outsiders” included all eligible patients or only those for whom data could be obtained. If “outsiders” are more likely to become lost to follow-up, in part because they have died or suffered other adverse events, true trial advantages might be missed.

Conclusion

We found no evidence to support either clinically important harm or clinically important benefit when patients’ illnesses were managed inside or outside an RCT. These results can inform discussions between clinicians and the patients to whom they are offering entry into peer-reviewed, ethically conducted RCTs. These results are also relevant to the policies, procedures and actions of institutions, ethics committees and granting agencies that permit and support the execution of RCTs.

Our findings and conclusions are only as good as the publication base of relevant RCTs, and we look forward to the day when the proposals of Vickers¹⁶⁰ and Altman and Cates¹⁶¹ are fully realized, with all trials registered and reported and with raw trial data made readily available. When that day arrives, our study should be repeated to determine the validity of the conclusions reached here.