Survival Benefit Associated With Participation in Clinical Trials of Anticancer Drugs

Key Points

Question

Is patient participation in cancer drug trials associated with a survival benefit?

Findings

In this systematic review and meta-analysis of 39 studies (85 comparisons), participation in trials by patients with cancer was associated with greater survival benefit compared with routine care (hazard ratio [HR], 0.76). However, survival benefit was not significantly greater when only high-quality studies were pooled (HR, 0.9) or when the sample was adjusted for possible publication bias (HR, 0.94).

Meaning

After accounting for biases and confounders, cancer clinical trial participation was not associated with a survival benefit.

Abstract

Importance

Many cancer clinical investigators view clinical trials as offering better care for patients than routine clinical care. However, definitive evidence of clinical benefit from trial participation (hereafter referred to as the participation effect) has yet to emerge.

Objective

To conduct a systematic review and meta-analysis of the evidence examining whether patient participation in cancer trials was associated with greater survival benefit compared with routine care.

Data Sources

Studies were found through PubMed and Embase (January 1, 2000, until August 31, 2022), as well as backward and forward citation searching.

Study Selection

Studies were included that compared overall survival of trial participants and routine care patients.

Data Extraction and Synthesis

Data extraction and methodological quality assessment were completed by 2 independent coders using Covidence software. Data were pooled using a random-effects model and analyzed based on the quality of the comparison between trial participants and routine care patients (ie, extent to which studies controlled for bias and confounders).

Main Outcomes and Measures

The hazard ratio (HR) for overall survival of trial participants vs routine care patients.

Results

Thirty-nine publications were included, comprising 85 comparisons of trial participants and routine care patients. The meta-analysis revealed a statistically significant overall survival benefit for trial participants (HR, 0.76 [95% CI, 0.69-0.82]) when all studies were pooled, regardless of design or quality. However, survival benefits diminished in study subsets that matched trial participants and routine care patients for eligibility criteria (HR, 0.85 [95% CI, 0.75-0.97]) and disappeared when only high-quality studies were pooled (HR, 0.91 [95% CI, 0.80-1.05]). They also disappeared when estimates were adjusted for potential publication bias (HR, 0.94 [95% CI, 0.86-1.03]).

Conclusions and Relevance

Many studies suggest a survival benefit for cancer trial participants. However, these benefits were not detected in studies using designs addressing important sources of bias and confounding. Pooled results of high-quality studies are not consistent with a beneficial effect of trial participation on its own.

This systematic review and meta-analysis examines whether patient participation in cancer trials is associated with greater survival benefit compared with routine care.

Introduction

Many people believe that patients achieve better clinical outcomes because of participation in clinical trials.^{1,2,3,4,5,6,7,8} This phenomenon is commonly called the trial effect, which is often attributed to closer monitoring and access to new treatments in trials. However, the assertion that trials confer medical benefits for participants rests on observational methodologies that are notoriously prone to bias and confounding.^{1,2,3,4,9,10,11} These observational methodologies are precisely the forms of evidence that motivate the conduct of rigorous trials.

Two reviews of cancer clinical trials based mostly on studies published in the 1990s found inconclusive evidence that participation in trials confers survival benefits.^1,5 Two other major systematic reviews in other disease areas have been published,^12,13 and neither showed evidence that patients in trials have better outcomes than patients outside of trials.

Numerous reports supporting a trial effect have been published since these reviews.^{14,15,16,17,18,19,20} Changes in trial practices since the 1990s, such as improved patient monitoring or greater inclusivity, might influence survival benefit for trial participation. Recent studies also vary in the extent to which they control for factors that might produce a spurious suggestion of survival benefit. For example, survival benefits associated with trial participation may reflect the confounding influence of trials selecting for patients with better prognoses, rather than the benefits of closer monitoring.

The primary aim of this study was to conduct a systematic review and meta-analysis of the evidence examining whether patient participation in cancer trials (hereafter referred to as trial participants) was associated with greater survival benefit when compared with patients who did not participate in trials (hereafter referred to as routine care patients), taking into consideration various sources of bias and confounding. The study assessed pooled overall survival based on subgroups related to quality of the comparison between trial participants and routine care patients (eg, accounting for trial eligibility, treatment effect, prognostic confounders) to explore design features associated with detection of survival benefits in trial participants.

Methods

Theoretical Framework

We defined treatment effect as the effects of trial participation on outcomes that are mediated by assignment to the experimental intervention in the trial. We defined participation effect as the effects of trial participation that are not mediated by assignment to the experimental intervention in the trial. We refer to the combination of these 2 as the trial effect, which includes all outcome differences between trial participants and routine care patients that are attributable to trial participation (eTable 1 and eTable 2 in Supplement 1). The present study is primarily aimed at isolating the participation effect, which we defined as outcome differences (eg, in overall survival time) attributed to trial participation that are unrelated to receiving an investigational intervention and not the result of confounding or measurement errors. Participation effects in trials might arise because of better management of medication adverse effects or improved medication adherence.

Search Strategy and Selection Criteria

We performed searches of PubMed and Embase (for articles published until August 31, 2022) for studies comparing survival outcomes for trial participants and routine care patients in cancer, limiting our search to studies published on or after January 1, 2000. We limited our search to 2000 and onward because previous systematic reviews have included older publications.^5,13 Also, older studies are not representative of the current landscape of clinical trials and often lack the detailed information needed for the present systematic review. Reference lists of found articles were also searched for eligible articles. The following keywords and variants were used to identify relevant articles: cancer, oncology, clinical trial, retrospective cohort, trial participation, trial effect, participation bias, nonparticipant, and nontrial (full list provided in eTable 3 in Supplement 1). Because the literature on trial effect uses different terminologies, we also conducted backward citation searches (searching publications cited in publications we found) and forward citation searches (searching publications that cited publications we found).

Publications were first screened based on titles and abstracts and then full texts were screened by 2 authors (R.I. and H.M.) for studies meeting the following inclusion criteria: (1) use of hazard ratios (HRs) to compare overall survival in a group of trial participants to a group of routine care patients (regardless of whether trial participant groups derived from a randomized trial), (2) treatment includes a drug/biologic, and (3) studies were conducted in patients with cancer. We excluded publications that were (1) focused on trials examining surgical procedures or indirect interventions (eg, programs); (2)commentaries, editorials, letters, or other nonresearch articles; and (3) non–English-language studies (eTable 4 in Supplement 1). Screening was completed using Covidence software.²¹ Disagreements were resolved through consensus.

Data Extraction

We extracted information from all studies for the following domains: patient demographics, treatment characteristics, and various quality items deemed important for adjustment when measuring participation effects (see below). We also extracted overall survival HRs for trial participants and routine care patients (receiving the same treatment, if available), regardless of whether the former derived from a treatment or control arm. When multiple HRs were available for a comparison, we extracted the HR that reflected the most adjustments for quality factors. Following extraction, study authors were contacted for missing information. All studies were extracted by 2 independent coders with disagreements reconciled by discussion.

Quality scoring in meta-analyses of observational studies using standard methods (eg, ROBINS-I) involves conceptual and practical difficulties^22,23,24 and is not customized to discern factors that specifically bear on estimating trial and participation effects. Also, some items in such tools involve many different components that are likely to be at play with routine care patient groups (eg, bias in selection of participants into studies might reflect differences in eligibility, prognostic factors, or consent in the trial participant and routine care patient groups). Instead, we began our study by creating a directed acyclic graph (eFigure 1 in Supplement 1) to identify factors that could cause a participation effect as well as factors that might confound or bias estimates thereof. Sixteen factors associated with confounding or bias were identified. From this, we created a 16-point scoring scale, assigning 1 quality point for each factor addressed in the primary study (eTable 5 in Supplement 1). Of note, some primary studies were not focused on measuring participation effects; our scale reflected quality with respect to estimating participation effects, not the primary objectives of original studies. A leave-one-out meta-analysis of the quality factors is included in eTable 6 in Supplement 1. Quality factors included differences between trial participants and routine care patients in cancer treatment, eligibility criteria, timeframe, demographics (eg, age, sex, race and ethnicity), and medical history (comorbidities, cancer stage, histology, performance status, and line of treatment). To examine the impact of quality scores on estimates of participation effects, we categorized primary studies into 3 similarly sized subgroups of low (≤6 points in our score), medium (7 points), and high (≥8 points) quality levels. Additional post hoc analyses include grouping primary studies into 2 and 5 quality score subgroups and are reported in eFigures 2 and 3 in Supplement 1.

Statistical Analysis

Because studies in our sample involved different indications and drugs, a standard meta-analysis would not be possible. Instead, meta-analytic methods were used to explore the impacts of quality factors (eg, accounting for treatment and eligibility) and general study characteristics (eg, sponsorship, study location) on pooled effects and heterogeneity.²⁵ For this, a DerSimonian and Laird random-effects model was used to pool survival hazard ratios of trial participants vs routine care patients.²⁶ This model is appropriate when pooling estimates from heterogeneous studies employing multiple designs and cancer types. All statistical analyses were completed using R version 4.3.0 (R Foundation).²⁷ We used funnel plots, Begg²⁸ and Egger²⁹ tests, and the trim-and-fill method³⁰ to explore potential publication biases.

This systematic review followed the PRISMA reporting guidelines (eTable 7 in Supplement 1).³¹ A protocol was preregistered on Open Science Framework (https://osf.io/b7h9w).³² Other information, including the dataset and codebook, are available on Open Science Framework.

Results

Following screening, 39 studies (85 total comparisons) were eligible for inclusion in the meta-analysis (Figure 1). Of these studies, 32 comparisons comparing trial participants and routine care patients aimed to measure the trial effect. The median sample sizes for the trial participant and routine care patient groups were 209 and 409 patients, respectively (Table 1). Characteristics of individual publications are available in eTable 8 and quality scores of individual studies are available in eTable 9 in Supplement 1. Examples of studies included in the sample are provided in the Box,^{33,34,35,36,37} along with an explanation of their quality.

Figure 1. Selection of Studies in the Systematic Review and Meta-Analysis.

Thirty-nine studies were included, which involved 85 comparisons.

Table 1. Study Characteristics of 85 Included Comparisons.

Study characteristic	No. (%)a
Year of publication
2000-2005	2 (2)
2006-2011	7 (8)
2012-2017	33 (39)
2018-2022	43 (51)
Study location
North America	64 (75)
Asia	9 (11)
Europe	9 (11)
Australia	3 (4)
Cancer
Hematologic	18 (21)
Breast	14 (16)
Lung	12 (14)
CNS	6 (7)
Prostate	6 (7)
Melanoma	5 (6)
Pancreatic	4 (5)
Otherb	20 (24)
Advanced/metastatic patients	28 (33)
Patient populationc
Adult	75 (89)
Pediatric	5 (6)
Mixed	4 (5)
Routine care group institution source
Single institution	21 (25)
Multiple institutions	64 (75)
Routine care group data source
Registry	67 (79)
Medical records	18 (21)
Sample size, median (IQR)
Trial participants	209 (96-397)
Routine care patients	409 (173-1523)
Age, median (IQR), yc,d
Trial participants	59 (53.3-62.0)
Routine care patients	62 (57-67)
Male sex, median (IQR), %c,e
Trial participants	57 (0-70.5)
Routine care patients	54.5 (0-67.5)
Female sex, median (IQR), %c,e
Trial participants	42 (28.5-99.0)
Routine care patients	44.5 (32.5-100.0)

Abbreviation: CNS, central nervous system.

^a Percentages may not add to 100 due to rounding.

^b Other cancer types included bladder, cervical, colorectal, esophageal, gastric, head and neck, kidney, ovarian, and mixed solid tumors.

^c Excludes studies with missing data.

^d Unweighted medians of reported means or medians.

^e Unweighted medians of reported percentages.

Box. Examples of Included Studies and Results.

• Abdel-Rahman (2019)³⁷ compared survival of patients with localized prostate cancer treated in clinical trials and patients registered in the Surveillance, Epidemiology, and End Results Program database. This study was deemed high quality: it accounted for age, sex, race and ethnicity, comorbidities, stage, performance status, line of treatment, treatment, eligibility, and timeframe. The study did not show evidence of a survival difference between patients treated in a clinical trial (n = 397) vs routine care (n = 1718) (hazard ratio [HR], 0.79 [95% CI, 0.45-1.39]; quality score = 12 points).

• Tanai et al (2011)³⁶ compared survival of patients with unresectable or recurrent gastric cancer treated with chemotherapy in trials and patients who were offered trial participation using medical records. This study was deemed high quality: it accounted for age, sex, race and ethnicity, stage, histology, performance status, line of treatment, treatment, eligibility, timeframe, and data source for patient groups. They found no evidence of a survival difference in patients treated in a clinical trial (n = 190) vs patients treated in routine care who refused trial participation (n = 96) (HR, 0.83 [95% CI, 0.62-1.10]; quality score = 12 points).

• Le Du et al (2016)³⁵ was scored as medium quality. It compared survival of patients with breast cancer in and out of clinical trials using medical records, and it accounted for age, sex, race and ethnicity, comorbidities, line of treatment, eligibility, timeframe, data source for patient groups, but not performance status, histology, stage, or treatment. They found no evidence of a survival difference in patients treated in a clinical trial (n = 285) vs patients treated in routine care (n = 367) (HR, 0.89 [95% CI, 0.72-1.10]; quality score = 7 points).

• Elumalai et al (2022)³⁴ compared survival of patients treated with docetaxel for metastatic castration-resistant prostate cancer in and out of clinical trials using medical records. This study was low quality with respect to estimation of participation effects. It accounted for sex, performance status, treatment, and timeframe, but not for factors including age, race and ethnicity, comorbidities, stage, histology, line of treatment, eligibility, or data source for patient groups. They found a survival benefit for trial participants (n = 2070) compared with routine care patients (n = 178) (HR, 0.57 [95% CI, 0.48-0.68]; quality score = 3 points).

• Mayers et al (2001)³³ compared survival of patients with breast carcinoma who participated in a clinical trial with those who did not using medical records. This study was low quality with respect to estimation of participation effects. It accounted for sex, stage, treatment, timeframe, data source for patient groups, but not for factors including age, race and ethnicity, comorbidities, histology, performance status, line of treatment, or eligibility. They reported a survival benefit for trial participants (n = 160) compared with routine care patients (n = 519), but results were not statistically significant (HR, 0.77 [95% CI, 0.57-1.05]; quality score = 3 points).

The original pooled HR for all included studies without regard to quality subgroups was 0.76 (95% CI, 0.69-0.82), suggesting a statistically significant survival benefit for trial participants in the highly heterogeneous sample (I² = 88%; Figure 2). When studies were grouped according to their overall quality score based on susceptibility to bias or confounding, the lowest scoring group reported the largest survival benefit for trial participants (HR, 0.64 [95% CI, 0.58-0.72]) and high heterogeneity (I² = 83%). The intermediate group reported results between the low and high groups (HR, 0.85 [95% CI, 0.73-0.98]; I² = 53%), and the highest scoring group reported no significant survival benefit (HR, 0.91 [95% CI, 0.80-1.05]) and showed high heterogeneity (I² = 89%) (Figure 2). Other groupings by quality produced consistent trends toward null estimates with greater quality (eFigures 2 and 3 in Supplement 1).

Figure 2. Pooled Overall Survival by Quality Adjustment Scores.

All studies are listed in the References list at the end of this article. Quality was assessed on a 16-point scale with higher scores indicating less potential for confounding or bias. Studies were classified as low-quality (≤6 points), medium-quality (7 points), and high-quality (≥8 points). See eTable 5 in Supplement 1 for factors used in scoring and eTable 9 in Supplement 1 for quality scores of individual studies. Some studies had multiple comparisons and, to differentiate them, superscript letters are noted by each comparison. The distinctions can be found in eTable 8 in Supplement 1.

Significant evidence of publication bias was found using 2 of the 3 methods deployed. A funnel plot (eFigure 4 in Supplement 1) and Egger test (P = .007) suggested possible publication bias against studies failing to show a participation effect. The Begg test did not show publication bias (P = .48). The trim-and-fill method added 30 comparisons to the 85 (n = 115) with a random effects pooled HR regressing to 0.94 (95% CI, 0.86-1.03).

All studies matched trial participants and routine care patients by cancer type, but varied in the extent to which they addressed other confounders and biases. This might explain the observed survival benefit for trial participants in the overall pooled estimate (Table 2). Overall, high heterogeneity is a possible indication of studies that were lower-quality for estimating participation effects. For instance, studies that lacked information on whether trial participants and routine care patients were matched on trial eligibility were the most heterogeneous and had the strongest effects (HR, 0.76 [95% CI, 0.63-0.92]; Q = 266.5), whereas studies known to have matched on trial eligibility had less heterogeneity and a higher HR (0.85 [95% CI, 0.75-0.97]; Q = 37.0). Similarly, studies that did not report whether they included individuals who chose not to participate in the trial were included in the routine care group (Q = 653.9) were also highly heterogeneous. In all cases but 1 (line of treatment), studies that accounted for prognostic confounders (another set of quality factors) produced smaller estimates of survival benefit for trial participants.

Table 2. Results of Subgroup Analyses of Comparisons by Various Characteristics That Indicate Quality of Comparison in Publications.

Characteristics	No. of comparisons	Pooled HR random model		Heterogeneity		Significance P valuea
		HR (95% CI)	P value	Q	Q P value
All comparisons	85	0.76 (0.69-0.82)	<.001	688.6	<.001	NA
Patient characteristics
Accounts for age	53	0.85 (0.77-0.93)	<.001	301.7	<.001	.002
Does not account for age	32	0.66 (0.59-0.75)	<.001	204.2	<.001
Accounts for sex	57	0.85 (0.77-0.93)	<.001	348.0	<.001	<.001
Does not account for sex	28	0.64 (0.56-0.72)	<.001	173.7	<.001
Accounts for race/ethnicity	42	0.89 (0.79-1.00)	.056	276.4	<.001	<.001
Does not account for race and ethnicity	43	0.68 (0.61-0.75)	<.001	268.0	<.001
Accounts for comorbidities	14	0.81 (0.66-1.00)	.05	214.7	<.001	.40
Does not account for comorbidities	71	0.74 (0.68-0.81)	<.001	349.9	<.001
Accounts for stage	47	0.87 (0.78-0.97)	.01	291.1	<.001	<.001
Does not account for stage	38	0.67 (0.60-0.74)	<.001	217.9	<.001
Accounts for histology	34	0.94 (0.82-1.07)	.33	246.5	<.001	<.001
Does not account for histology	51	0.68 (0.62-0.75)	<.001	286.2	<.001
Accounts for performance status	23	0.80 (0.68-0.94)	.01	256.1	<.001	.39
Does not account for performance status	62	0.74 (0.67-0.81)	<.001	320.2	<.001
Accounts for line of treatment	43	0.72 (0.64-0.82)	<.001	575.0	<.001	.15
Does not account for line of treatment	42	0.81 (0.73-0.90)	<.001	112.0	<.001
Accounts for trial eligibility	17	0.85 (0.75-0.97)	.01	37.0	.002	.17
Does not account for trial eligibility	52	0.73 (0.66-0.81)	<.001	244.1	<.001
Missing data on trial eligibility	16	0.76 (0.63-0.92)	.004	266.5	<.001
Routine care group does not include trial refusers	2	0.96 (0.73-1.26)	.76	0.1	.74	.004
Routine care group includes trial refusers	14	0.93 (0.83-1.04)	.18	19.2	.12
Missing data on the inclusion of trial refusers	69	0.73 (0.66-0.81)	<.001	653.9	<.001
Treatment characteristics
Accounts for treatment effect (same treatments)	45	0.69 (0.62-0.77)	<.001	291.8	<.001	.01
Does not account for treatment effect (different treatments)	30	0.87 (0.76-0.99)	.04	58.3	<.001
Missing data on treatment effect	10	0.88 (0.69-1.11)	.28	191.8	<.001
Setting characteristics
Similar time period for comparison groups	59	0.80 (0.72-0.88)	<.001	460.1	<.001	.08
Different time period for comparison groups	26	0.69 (0.60-0.78)	<.001	149.8	<.001
Compares routine care group to a single trialb	54	0.73 (0.65-0.82)	<.001	301.4	<.001	.32
Compares routine care group to multiple trialsb	31	0.79 (0.70-0.90)	<.001	322.4	<.001
Other characteristics
Routine care group data source is registries	67	0.75 (0.67-0.82)	<.001	632.8	<.001	.42
Routine care group data source is medical records	18	0.80 (0.70-0.91)	.001	53.7	<.001
Accounts for the same trial and routine care group sourcesc	29	0.82 (0.74-0.90)	<.001	70.8	<.001	.18
Does not account for the same trial and routine care group sourcesc	56	0.74 (0.65-0.83)	<.001	617.3	<.001
Trial sample size of ≥200	48	0.75 (0.67-0.84)	<.001	613.1	<.001	.73
Trial sample size of <200	37	0.77 (0.69-0.85)	<.001	69.1	<.001

^a Shows whether there is a significant difference between items in each subgroup (eg, in the first category, the P value of .002 shows that there is a statistically significant difference in survival estimates for comparisons that did vs did not account for age.

^b Studies that compared routine care patient groups to the same trial only included 1 trial in the trial group, whereas comparing to multiple trials included more than 1 trial in the trial participant group.

^c Studies accounting for the same trial and routine care sources indicated that both groups were from the same source. For example, the Surveillance, Epidemiology, and End Results (SEER) Program covers all the US, so it is not the same location, but if both trial and routine care groups are from the SEER database, then they accounted for similar group sources.

Subgroup analyses based on different study characteristics were also conducted, without regard for quality indicators. Comparisons from studies in the US (n = 31) showed an HR of 0.87 (95% CI, 0.75-1.02), while those conducted in other countries (n = 54) had an HR of 0.72 (95% CI, 0.65-0.79) (eTable 10 in Supplement 1).

Discussion

In this systematic review and meta-analysis of 39 studies comparing outcomes among patients with cancer participating in clinical trials vs those receiving routine care, we found that typical pooled analyses suggest that trial participation is associated with greater survival benefit. However, survival benefits for participation diminish or disappear in studies that account for various sources of bias and confounding. Accounting for factors such as eligibility and some prognostic confounders reduced the magnitude of overall participation effects. For example, no statistically significant trial effect was observed for studies that accounted for comorbidities, histology, or race and ethnicity. When a subgroup analysis was conducted based on quality score to isolate participation effects, there was a greater apparent survival benefit in low-quality studies than high-quality studies. The high-quality studies made more efforts to reduce heterogeneity between trial participants and routine care patients.

Because preregistries are not widely used for observational studies, there are no direct ways to test for publication bias. However, indirect statistical tests suggest a disproportionate number of small studies indicating large survival benefits for trial participants, which is consistent with publication bias. Statistically correcting for potential publication bias produced an HR that was inconsistent with a significant survival benefit for trial participants.

Previous systematic reviews, including those focused on cancer, have not detected clear evidence that patients in randomized trials have better outcomes than patients outside them.^1,5,9,12,13 Despite this, many factors may contribute to the perception that patients have better outcomes in trials. One is that participants in trials often experience better care processes, including more frequent imaging. Another is the notorious efficacy-effectiveness gap. Trials often show survival outcomes that exceed those in clinical care.^38,39 Indeed, the literature on efficacy-effectiveness gaps complements our findings: prognostic variables such as performance status that appear to explain better outcomes in trials, when adjusted for in comparisons of trial participants and routine care patients, diminish apparent participation effects. A third factor is the regular publication of studies, similar to many in the current sample, that suggest a participation effect but do not account for relevant confounders. Similar to prior studies,^1,5 the current study suggests that methodological rigor and quality continues to present a challenge for some publications asserting estimates of the participation effect.

This study leaves many questions about participation effects unresolved. Some studies suggest that patients have improved quality of life, greater cost savings, or benefit from incidental findings when participating in trials.^40,41,42 Other studies have suggested that patients have worse quality of life outcomes.^43,44 The current protocol set out to measure such benefits. However, outcomes such as quality of life or anxiety are often not measured in trials, much less in routine care patients. It was not possible to identify a critical mass of studies on patient-reported outcomes to include in the meta-analysis. The analysis also does not address whether participation effects might occur with interventions involving greater skill with administration (eg, surgical procedures).

Limitations

This meta-analysis has limitations. First, many studies in the review were missing data or provided unclear information leading to exclusions and/or difficulties in interpreting studies. Authors were contacted to mitigate these difficulties. Due to incomplete descriptions on how trial participants and routine care patients were matched, it was also difficult to determine how well quality factors were implemented. For example, treatments were often very poorly described and therefore treatment similarity was not always clear for studies claiming to have matched treatments between groups. Second, the scale used for measuring study quality was created expressly for this study. Although the scoring scale was systematic and reflected sources of bias for trial effect studies, scales with different sets of factors, factor prioritization with weighted scores, or score cut-offs might produce findings that differ from those reported above. However, the main purpose of stratifying by quality was to explore relationships between low adjustment and detection of participation effects.

Third, as previously noted,¹ the absence of standard terminologies makes it extremely difficult to conduct literature searches for primary reports of the participation effect. It cannot be ruled out that some reports were not captured in the search. Fourth, the study provides a picture of participation benefit associated with current research practices. The possibility that other trial approaches might produce different results cannot be excluded. For example, trials often select for patients that have fewer comorbidities or better performance status. Participation effects might be more manifest where trials relax such criteria. Fifth, any attempt to study the benefits of trial participation is subject to the fact that there is no random assignment among trial participants and routine care patients. Even ideally designed studies may be confounded, and such confounders could either mask or amplify participation effect estimates. The best that can be concluded is that participation effect estimates grow narrower the more analyses account for various confounders. It cannot be ruled out that benefits of participation might be revealed were it possible to conduct a randomized trial testing the benefits of participating in a randomized trial.

Conclusions

In this meta-analysis, evidence that cancer trial participation results in survival benefits was mostly driven by studies that do not account for factors that could bias or confound such estimates. When analysis was restricted to studies that account for such factors (eg, eligibility, prognostic confounders), effects regress toward a null effect. These findings may strike some trial advocates as discouraging, given how hard they work to improve patient outcomes within trials. However, a more reassuring interpretation is that there is no evidence that excluding patients from trials due to geography, nonavailability of trials in their condition, or ineligibility deprives them of survival opportunities.

Educational Objective: To identify the key insights or developments described in this article.

1. The authors refer to a phenomenon called the “trial effect.” What is the trial effect?

   1. An increase in adverse events beyond what might otherwise be anticipated, likely on account of regular assessment throughout the trial period.

   2. Improved outcomes for both placebo and treatment groups due to systematic exclusion of sicker patients who may not be able to complete therapy.

   3. The belief that participation in clinical trials yields better outcomes, potentially on account of closer monitoring and access to new treatments.

2. What were the results of this meta-analysis of 85 comparisons across 39 studies?

   1. High-quality studies demonstrated a statistically meaningful survival benefit (hazard ratio, 0.64), while low-quality studies did not (hazard ratio, 0.91).

   2. The original pooled hazard ratio for all included studies was 0.76, suggesting a statistically significant survival benefit for those participating in a clinical trial.

   3. Trial participants experienced more adverse events, including incidental mortality, than patients undergoing routine care.

3. How do the authors interpret their findings?

   1. Evidence for a trial effect is mostly driven by studies that did not adequately account for factors that could bias or confound results.

   2. That trial participation resulted in survival benefit suggests the need to augment routine care processes to more closely resemble the care provided as part of clinical trials.

   3. While there was evidence of publication bias, statistical correction for publication bias continued to yield a meaningful survival benefit for trial participants.

Supplement 1.

eResults

Supplement 2.

Data sharing statement