Key Points
Question
How similar are dialysis-dependent patients recruited to large, multicenter randomized clinical trials compared with the general dialysis-dependent population?
Findings
In this meta-analysis of 189 trials including 80 104 participants, trial participants were significantly younger, more likely to be male, and less likely to have diabetes or diabetic nephropathy than patients in the US national registry. Moreover, the mortality rate of dialysis-dependent patients recruited to large, multicenter randomized trials was substantially lower than that of registry patients, both overall and when only studies recruiting participants from the United States were considered.
Meaning
These findings imply that caution should be exercised when generalizing results from clinical trials to the broader dialysis-dependent patient population.
Abstract
Importance
Systematic differences between patients included in randomized clinical trials (RCTs) and the general patient population may influence the generalizability of RCT findings. Comprehensive national registries of patients with end-stage kidney disease who are undergoing dialysis provide a unique opportunity to compare trial and real-world patient cohorts.
Objective
To determine if participants in large, multicenter dialysis trials were similar to the general population undergoing dialysis in terms of age, comorbidities, and mortality rate.
Data Sources
MEDLINE, PubMed, and the Cochrane Central Register of Controlled Trials were systematically searched on January 6, 2017, for studies published from January 1, 2007, to December 31, 2016. Data sources were published manuscripts, supplementary material, and trial registration information. Data on the general population undergoing dialysis were derived from the US Renal Data System (USRDS). Data were analyzed from March 17 to July 22, 2018.
Study Selection
Randomized clinical trials enrolling only participants undergoing dialysis for end-stage kidney disease with 100 or more adult participants from 2 or more sites.
Data Extraction and Synthesis
Abstract screening and data extraction were performed independently by 2 researchers. Data were pooled using a random-effects model.
Main Outcomes and Measures
The primary outcome was difference in mean age between the RCT and USRDS populations. Secondary outcomes included differences in mortality rate and comorbidities.
Results
The search identified 186 RCTs, enrolling 79 104 participants. Compared with the 2011 USRDS population, RCT participants were younger (mean age, 58.9 years; 95% CI, 58.3-59.5 years vs 61.2 years; P < .001), more likely to be male (58.9%; 95% CI, 57.6%-60.1% vs 55.7%; P < .001), and have coronary artery disease (26.9%; 95% CI, 22.2%-31.7% vs 17.7%; P < .001) and less likely to have diabetes (40.2%; 95% CI, 36.7%-43.6% vs 44.2%; P = .03) or heart failure (19.6%; 95% CI, 15.1%-24.0% vs 29.8%; P < .001). The mortality rate per 100 patient-years during trial participation was less than half that of the USRDS population (8.9; 95% CI, 7.8-10.0 vs 18.6; P < .001). The differences in age, mortality, and coronary artery disease remained when studies recruiting only from the United States were considered. Diabetes was more common in RCT participants from the United States than in the registry population.
Conclusions and Relevance
Participants in large, multicenter RCTs of patients with end-stage kidney disease undergoing dialysis are younger, have a different pattern of comorbidities, and have a lower mortality rate than the general population of patients undergoing dialysis. This finding has implications for the generalization of trial results to the broader patient population and for future trial design.
This meta-analysis of 189 studies involving 80 104 participants examines whether participants in large, multicenter dialysis trials were similar to the general population undergoing dialysis in terms of age, comorbidities, and mortality rate.
Introduction
The single or pooled results of randomized clinical trials (RCTs) are considered the highest level of evidence for treatment efficacy in modern medicine.1 For the results of an RCT to be generalizable to a specific patient population, the study cohort and population of interest must be sufficiently similar in clinically relevant characteristics so that the intervention effect from the RCT can be reasonably expected to be replicated in the population of interest. Such studies are crucial to the development of clinical practice guidelines and to inform health economic and health service provision decisions. However, studies in other specialties have found that participants in RCTs are often not representative of the broader patient population in which the intervention is likely to be used.2,3,4 This finding has implications for clinicians and health policy because intervention efficacy may be less than expected for some patient groups, such as older and frailer patients who tend to be underrepresented in clinical trials.3,5
The relevance of study generalizability depends, in part, on the study aims. Early phase and “explanatory” studies may not prioritize generalizability, as their aim is to provide proof of efficacy under ideal conditions. In contrast, generalizability is important for later phase and pragmatic trials that aim to determine if an intervention is effective in general clinical practice. Such trials are typically larger, recruit from multiple sites, and may have broader inclusion and exclusion criteria to increase the generalizability of the results.6 Nephrology RCTs are fewer in number, smaller, and more likely to be phase 1 or 2 trials compared with other disciplines in internal medicine,7,8 raising the possibility that they may have limited generalizability. Although RCTs in kidney transplant recipients have been shown to enroll younger participants than the general population in the United States of patients who have received kidney transplants (suggesting important limitations to generalizability),9 to our knowledge, this question has not been addressed in the dialysis population.
We aimed to assess the generalizability of large, multicenter RCTs among patients undergoing dialysis by comparing prespecified measures of generalizability (age, comorbidities, and mortality rate) between trial participants and patients in the largest dialysis registry, the United States Renal Data System (USRDS). We also aimed to determine if particular study factors (such as design, size, and sponsorship type) were associated with these measures of generalizability.
Methods
We undertook a systematic search of MEDLINE, PubMed, and the Cochrane Central Register of Controlled Trials for randomized studies (including randomized crossover trials and cluster randomized trials), published from January 1, 2007, to December 31, 2016, enrolling participants undergoing maintenance dialysis for end-stage kidney disease at the time of randomization. Both hemodialysis and peritoneal dialysis were included. Studies were included if they recruited participants from at least 2 sites (as defined by the study authors) and if at least 100 participants were randomized. Studies enrolling participants younger than 18 years, with acute kidney injury, or with—or about to receive—a kidney transplant were excluded. No language restriction was used. The review protocol was registered (PROSPERO ID: CRD42018090862).
After removal of duplicates, titles and abstracts were screened for inclusion or exclusion by 2 of 3 of us (B.S., A.H., and K.T.) with discrepancies adjudicated by the third. Articles were then reviewed independently by 2 of the 3 of us. Inclusion and exclusion criteria were assessed and, if included, the reviewer proceeded to data extraction. Supplementary eAppendices, eTrial registrations, and secondary publications were also consulted if complete data were not present in the primary study manuscript. A single cohort of participants could be included only once; where multiple publications from the same cohort of participants were identified, the earliest publication with results was identified as the primary publication and was considered the primary source of information. Data discrepancies were resolved by discussion among the reviewers after data extraction was completed.
Data Items
Data were extracted based on predefined study characteristics, participant characteristics, and mortality. Study characteristics included trial registration number, type of study sponsor, country of sponsor, type of randomized study, type of intervention, primary end point(s), follow-up time (planned and observed), dialysis modalities, number of randomized participants, and number of recruiting sites and countries. Study sponsor information was obtained from trial registration information where available and, for commercial sponsors, the country of origin was determined by the location of the organization’s headquarters. Otherwise, if sponsorship was not stated, the allocation to commercial or noncommercial sponsor category was made based on the presence of a commercial entity in the funding or acknowledgments statements. Baseline participant characteristics included age, sex, albumin level, hemoglobin level, dialysis modality, dialysis vintage, type of vascular access, erythropoietin-stimulating agent use, cause of primary renal disease, and proportion with comorbid diabetes or cardiovascular disease. Means and SDs were preferred to medians and interquartile range. Mortality was recorded where available. Mortality rate (per 100 patient-years) was preferred. If reported per study group, a weighted mean was calculated. If no rate was reported, an actuarial mortality rate was calculated from study duration and number of recorded deaths (eAppendix in the Supplement). Studies that did not report on participant deaths were excluded from mortality analyses.
Comparison Registry
The 2011 cohort of the USRDS was chosen a priori as the reference registry because it is the largest single registry (with annual records for more than 500 000 dialysis recipients), makes comprehensive data available online, and represents the country likely to provide the largest share of study participants. The year 2011 was chosen for its position at the midpoint of the systematic review period. Information was sourced from data files provided with the 2013 and 2015 Annual Data Reports (available at https://www.usrds.org/archive.aspx).10,11,12,13,14
Outcomes
The primary outcome was the age of RCT participants. Secondary outcomes included a variety of patient characteristics: sex, comorbidities such as diabetes and cardiovascular disease, cause of renal disease, hemoglobin level, albumin level, type of vascular access, and study mortality. Several outcomes and variables were selected a priori for unadjusted and adjusted analyses to determine which study characteristics were associated with generalizability. Three participant characteristics—age, prevalence of diabetes, and mortality rate—were selected as surrogate measures of generalizability in these adjusted analyses. Five study characteristics—sponsor type (commercial, noncommercial, or unknown), type of study (parallel-group, cluster-randomized, or crossover), number of participants, number of recruiting sites, and year of publication—were selected as factors associated with these generalizability outcomes.
Statistical Analysis
For each study, overall participant baseline characteristics were calculated by combining treatment allocation subgroup means or medians using weighting by subgroup size. Pooled SDs were calculated using the formula provided by Woodward.15 The weighted mean SD from all studies was used to impute the SD for studies that reported mean values without SD. Age, albumin level, and hemoglobin level were assumed to be normally distributed, permitting median values to be considered equivalent to means and the estimation of SD from the interquartile ranges (eAppendix in the Supplement).16 Summary statistics were estimated by the random effects model of DerSimonian and Laird,17 applied to both continuous and categorical variables (those with >2 categories were analyzed as a series of dichotomous outcomes).
Primary and secondary outcomes were compared with the equivalent USRDS data by 1-sample t tests, where the USRDS value was considered the reference value and the test statistic was obtained from a random-effects RCT meta-analysis. Categorical variables were analyzed by 1-way analysis of variance and continuous variables were analyzed by linear regression. Multivariable linear regression analysis was performed to determine the association between the 5 study characteristics and 3 primary generalizability outcomes. Where relevant, individual observations within statistical models were weighted by number of study participants. Statistical analysis was performed using Stata, version 15.0 (StataCorp). All P values were from 2-sided tests, and results were deemed statistically significant at P < .05.
Results
Study Selection
eTable 1 and eFigure 2 in the Supplement contain all 189 studies in this meta-analysis. The database search on January 6, 2017, returned 5229 records. After removal of duplicates and screening of title and abstract, 545 full-text articles were obtained. Of these, 356 were excluded, leaving 186 studies, enrolling 79 104 participants, to be included in the analysis (Figure).
Figure. Study Selection.
Study Characteristics
Studies were predominantly open-label (119 [64.0%]) and parallel group (170 [91.4%]), with a surrogate primary outcome (126 [67.7%]) and randomized participants receiving only hemodialysis (149 [80.1%]) (Table 1). The most common class of intervention was dialysis practice change (41 [21.7%]), encompassing a variety of interventions such as catheter locks, dressings, topical preparations, and novel dialysis membranes. A minority of studies (35 [18.8%]) used upper age exclusion criteria. Among these studies, the median upper age limit was 75 years (range, 65-90 years).
Table 1. Study Characteristics.
| Characteristic | Trials, No. (%) (N = 186) |
|---|---|
| Sponsor | |
| Commercial | 95 (51.1) |
| Noncommercial | 82 (44.1) |
| Unknown | 9 (4.8) |
| Type of study | |
| Parallel group | 170 (91.4) |
| Cluster | 11 (5.9) |
| Crossover | 5 (2.7) |
| Multicountry | 50 (26.9) |
| Countries included, median (IQR), No. | 7.5 (3-11) |
| Intervention | |
| Dialysis practice change | 41 (22.0) |
| Oral pharmaceutical | 39 (21.0) |
| Phosphate binder | 32 (17.2) |
| Erythropoietin-stimulating agent | 24 (12.9) |
| Injectable pharmaceutical | 19 (10.2) |
| Other | 13 (7.0) |
| Dietary | 6 (3.2) |
| Fistula or graft related | 5 (2.7) |
| Physical therapy | 3 (1.6) |
| Increased hours or frequency of dialysis | 2 (1.1) |
| Other surgical procedure or device | 2 (1.1) |
| Type of primary outcome | |
| Surrogate | 126 (67.7) |
| Clinical eventa | 48 (25.8) |
| Cardiovascular event(s) | 12 (6.5) |
| Mortality | 20 (10.8) |
| Complication of dialysis | 23 (12.4) |
| Other | 5 (2.7) |
| Patient reported | 10 (5.4) |
| Other | 2 (1.1) |
| Included modality | |
| Hemodialysis | 149 (80.1) |
| Both hemodialysis and peritoneal dialysis | 21 (11.3) |
| Peritoneal dialysis | 16 (8.6) |
| Incident participants only | 10 (5.4) |
| Primary outcome statistically significantb | 136 (73.1) |
| Participants, median (IQR), No. | 212 (149-339) |
| Sites, median (IQR), No. | 16 (5-42) |
| Planned duration of follow-up, median (IQR), mo | 7 (3-12) |
| Blinding | |
| Open-label | 119 (64.0) |
| Double-blind | 60 (32.3) |
| Single-blind | 7 (3.8) |
| Upper age limit specified | 35 (18.8) |
Abbreviation: IQR, interquartile range.
Subcategories are overlapping owing to coprimary or composite end points.
Based on author-specified criteria for significance. Achievement of noninferiority was considered significant where this was the stated primary outcome.
Participants were recruited from 58 countries, and 50 studies (26.9%) recruited from multiple countries (Table 1). The United States was the most frequently represented country, with 59 studies (32.3%) including at least 1 site from the United States and 42.0% of all sites (2587 of 6161) and 28.0% of study sponsors (52 of 186) being located in the United States (eTables 2 and 3 in the Supplement). Based on median values, the typical study had 212 participants from 16 sites in a single country and a follow-up time from randomization to final data collection of 7 months. A total of 133 trials (71.5%) reported their trial registration in the manuscript and 96 of these 133 trials (72.2%) were registered with ClinicalTrials.gov. Sponsorship could be determined for 177 trials (95.2%), with 95 (51.1%) being sponsored by a commercial entity.
Comparison of All Participants With USRDS
Compared with USRDS in 2011, study participants were significantly younger (58.9 years; 95% CI, 58.3-59.5 years vs 61.2 years; P < .001), more likely to be male (58.8%; 95% CI, 57.5%-60.0% vs 55.7%; P < .001), and less likely to have diabetes (40.4%; 95% CI, 36.9%-43.8% vs 44.2%; P = .04), diabetes as a cause of renal failure (27.4%; 95% CI, 24.9%-29.9% vs 44.2%; P < .001), or hypertension as a cause of renal failure (20.7%; 95% CI, 18.3%-23.0% vs 29.0%; P < .001) or to be undergoing dialysis with a catheter (12.9%; 95% CI, 10.3%-15.5% vs 21.2%; P < .001) (Table 2). Although study participants had a lower prevalence of heart failure, the prevalence of coronary artery disease, cerebrovascular disease, and peripheral vascular disease was higher. Mean albumin levels were higher in study participants, although this was in comparison with patients with incident dialysis in the USRDS (as prevalent patient albumin levels are not reported). Mortality among study participants could be determined in 67.2% (125 of 186) of studies. In these studies, the mortality rate was less than half of that reported in the USRDS for 2011 (8.9 per 100 patient-years; 95% CI, 7.8-10.0 vs 18.6 per 100 patient-years; P < .001).
Table 2. Study Participant Characteristics Compared With USRDS.
| Characteristic | USRDS Value, %a | All RCTs, Mean (95% CI) | Trials, No. | P Value | US-Predominant RCTs, Mean (95% CI) | Trials, No. | P Value | US-Only RCTs, Mean (95% CI) | No. | P Value |
|---|---|---|---|---|---|---|---|---|---|---|
| Total No. of trials | 186 | 46 | 37 | |||||||
| Age, y | 61.2 | 58.9 (58.3-59.5) | 184 | <.001 | 58.2 (57.4-59.0) | 46 | <.001 | 58.6 (57.7-59.4) | 37 | <.001 |
| Male sex | 55.7 | 58.9 (57.6-60.1) | 183 | <.001 | 56.6 (54.6-58.7) | 44 | .38 | 55.4 (53.0-57.7) | 35 | .82 |
| Comorbid diabetes | 44.2 | 40.2 (36.7-43.6) | 90 | .03 | 54.9 (52.4-57.5) | 24 | <.001 | 56.1 (53.6-58.5) | 20 | <.001 |
| Primary renal disease | ||||||||||
| Diabetic nephropathy | 44.2 | 27.3 (24.8-29.9) | 88 | <.001 | 40.8 (38.1-43.5) | 20 | .03 | 41.9 (38.6-45.3) | 13 | .21 |
| Hypertension or vascular | 29.0 | 20.5 (18.2-22.9) | 77 | <.001 | 32.8 (28.5-37.1) | 20 | .10 | 31.6 (26.5-36.7) | 13 | .33 |
| Glomerulonephritis | 9.5 | 25.6 (22.6-28.7) | 83 | <.001 | 10.8 (8.4-13.2) | 16 | .32 | 8.9 (6.4-11.5) | 11 | .64 |
| Cystic kidney disease | 2.6 | 5.4 (4.6-6.2) | 50 | <.001 | 3.2 (2.6-3.9) | 10 | .09 | 2.8 (2.1-3.6) | 6 | .59 |
| Comorbidities at start of RRT, all modalities (2011-2013) | ||||||||||
| Heart failure | 29.8 | 19.6 (15.1-24.0) | 31 | <.001 | 28.9 (22.9-35.0) | 10 | .78 | 27.7 (20.9-34.6) | 6 | .57 |
| Coronary artery disease | 17.7 | 26.9 (22.2-31.7) | 34 | <.001 | 37.7 (34.0-41.5) | 9 | <.001 | 38.4 (31.1-45.6) | 6 | .003 |
| Cerebrovascular disease | 8.8 | 11.1 (9.6-12.5) | 30 | .004 | 12.9 (9.3-16.5) | 9 | .06 | 13.0 (7.5-18.5) | 5 | .21 |
| Peripheral vascular disease | 12.0 | 15.2 (12.3-18.0) | 26 | .04 | 15.4 (9.0-21.8) | 9 | .33 | 15.0 (7.7-22.3) | 6 | .46 |
| Hemoglobin, g/dL | ||||||||||
| Prevalent hemodialysis | 11.00 | 11.01 (10.83-11.19) | 91 | .91 | 11.26 (11.04-11.48) | 16 | .04 | 11.07 (10.74-11.40) | 10 | .69 |
| Prevalent peritoneal dialysis | 10.83 | .05 | .002 | .19 | ||||||
| Albumin, g/dL | ||||||||||
| Incident, all modalities (2011-2013) | 3.20 | 3.78 (3.73-3.82) | 79 | <.001 | 3.80 (3.72-3.88) | 17 | <.001 | 3.77 (3.70-3.84) | 13 | <.001 |
| Vascular access | ||||||||||
| Catheter | 21.2 | 12.9 (10.3-15.5) | 22 | <.001 | 18.6 (12.0-25.2) | 9 | .47 | 20.2 (11.0-29.5) | 7 | .84 |
| Mortality rate, per 100 patient-years (2011-2013) | 18.6 | 8.9 (7.8-10.0) | 125 | <.001 | 10.3 (7.3-13.3) | 33 | <.001 | 10.6 (6.9-14.3) | 25 | <.001 |
Abbreviations: RCT, randomized clinical trial; RRT, renal replacement therapy; USRDS, United States Renal Data System.
SI conversion factors: To convert hemoglobin to grams per liter, multiply by 10.0; albumin to grams per liter, multiply by 10.0.
All USRDS values are for prevalent dialysis patients in 2011, unless otherwise specified.
Comparison of Studies Recruiting From the United States With USRDS
The disparity in mean age between individuals in the USRDS and the study population remained when considering only the 46 US-predominant studies (ie, those with at least 50% of their sites in the United States) (58.2 years; 95% CI, 57.4-59.0 years vs 61.2 years; P < .001). In contrast with the overall RCT cohort, the prevalence of diabetes was higher than in the USRDS population (54.9%; 95% CI, 52.4%-57.5% vs 44.2%; P < .001) and mean hemoglobin level was significantly higher than in the USRDS prevalent hemodialysis population (11.26 g/dL; 95% CI, 11.04-11.48 g/dL vs 11.00 g/dL; P = .04 [to convert to grams per liter, multiply by 10.0]). The mortality rate remained significantly lower (10.3 per 100 patient-years; 95% CI, 7.3-13.3 vs 18.6 per 100 patient-years; P < .001).
When the analysis was further restricted to studies recruiting only from the United States, the difference in age and mortality between trial participants and the 2011 USRDS population remained significant. This finding was despite a higher burden of comorbid diabetes and coronary artery disease in the study population compared with the registry population (Table 2).
Influence of Study Factors on Participant Age, Comorbid Diabetes, and Mortality
Participant age and prevalence of diabetes were significantly higher in cluster-randomized trials compared with parallel group and crossover studies, although mortality rates did not differ between these study types (Table 3). There were no differences in these participant characteristics between studies with commercial or noncommercial sponsors. Similarly, neither study size nor the number of recruiting sites was associated with participant age, proportion with diabetes, or study mortality rate. Between 2007 and 2016, the prevalence of diabetes in trial participants increased (coefficient, 0.020; 95% CI, 0.009-0.031; P < .001), while the mortality rate decreased (coefficient, –0.617; 95% CI, –0.978 to –0.256; P = .001), with no significant change in mean participant age (coefficient, 0.173; 95% CI, –0.027 to 0.373; P = .13) (eFigure 1 in the Supplement). Adjusted analyses suggested that noncommercial studies were associated with a significantly higher mortality rate (2.60; 95% CI, 0.09-5.10; P = .04), as were studies published earlier (–0.59; –0.97 to –0.21; P = .001) (Table 4). No study characteristics were significantly associated with patient age or prevalence of diabetes.
Table 3. Unadjusted Analysis of Participant Age, Prevalence of Diabetes, and Mortality by Sponsor and Study Typea.
| Characteristic | Age, Median (95% CI), y | P Value | Diabetes, % (95% CI) | P Value | Mortality Rate, per 100 Patient-Years (95% CI) | P Value |
|---|---|---|---|---|---|---|
| Sponsor | ||||||
| Commercial | 59.3 (58.5-60.1) | .22 | 44.9 (40.1-49.7) | .99 | 8.7 (7.6-9.9) | .54 |
| Noncommercial | 60.1 (59.2-61.0) | 44.9 (40.5-49.3) | 9.5 (7.3-11.7) | |||
| Study type | ||||||
| Cluster-randomized | 61.9 (60.7-63.0) | <.001 | 56.7 (50.3-63.0) | <.001 | 7.0 (0.9-13.1) | .75 |
| Parallel group | 58.8 (58.1-59.4) | 38.3 (35.2-41.4) | 9.0 (7.9-10.1) | |||
| Crossover | 59.1 (54.8-63.5) | 20.2 (26.8-37.7) | 6.4 (0.0-13.7) |
Results from 1-way analysis of variance.
Table 4. Adjusted Analysis of Age, Prevalence of Diabetes, and Mortality Ratea.
| Characteristic | Age, Coefficient (95% CI) | P Value | Diabetes, Coefficient (95% CI) | P Value | Mortality Rate, Coefficient (95% CI) | P Value |
|---|---|---|---|---|---|---|
| Sponsor (reference: commercial) | ||||||
| Noncommercial | 0.91 (−0.73 to 2.55) | .27 | −0.022 (−0.098 to 0.054) | .56 | 2.63 (0.13 to 5.12) | .04 |
| Study type (reference: parallel group) | ||||||
| Cluster-randomized | 1.03 (−2.29 to 4.36) | .54 | 0.042 (−0.091 to 0.175) | .53 | −0.24 (−7.49 to 5.01) | .70 |
| Crossover | 1.59 (−3.30 to 6.49) | .52 | −0.209 (−0.434 to 0.016) | .07 | −0.19 (−7.33 to 6.96) | .96 |
| Year of publication | 0.05 (−0.20 to 0.30) | .72 | 0.007 (−0.006 to 0.019) | .30 | −0.62 (−1.00 to −0.24) | .001 |
| No. of participants | 0.0002 (−0.001 to 0.001) | .81 | 2 × 10−5 (−2 × 10−5 to 7 × 10−5) | .30 | 5 × 10−5 (−0.003 to 0.003) | .97 |
| No. of sites | 0.002 (−0.015 to 0.020) | .79 | −3 × 10−4 (−9 × 10−4 to 3 × 10−4) | .35 | 0.01 (−0.01 to 0.04) | .36 |
Results from weighted linear regression model.
Discussion
These results show that participants in large, multicenter trials recruiting patients undergoing dialysis are younger and more likely to be male than the general population of patients in the United States undergoing dialysis. Moreover, their pattern of comorbidities differs, with a lower prevalence of diabetes, diabetic nephropathy, and hypertensive nephropathy, but a higher prevalence of cardiovascular disease. The estimated mortality rates in trial participants were much lower than in the USRDS, suggesting that trial participants are healthier on average than the general population of patients undergoing dialysis. Although some of the differences observed between the RCT and registry cohorts were small, when considered collectively these differences may imply that the magnitude of treatment effects assumed from the evidence base may not be realized when those interventions are applied to a more representative cohort of patients. This finding emphasizes the importance of clinicians and policymakers carefully considering the generalizability of particular RCT results to their own distinct populations. It also supplies a clear rationale for increasing the effort to produce pragmatic RCTs. Such trials, for example, the TiME (Time to Reduce Mortality in ESRD) trial of longer hemodialysis sessions,18 are characterized by an effort to maximize generalizability by the minimization of inclusion and exclusion criteria, the use, where possible, of routinely collected data and an emphasis on real-world practice.19
Our results are in line with research in renal transplant recipients, where a difference in age between registry and trial populations has been described.9 Similar conclusions have also been drawn in other disciplines. For example, studies in cardiology, mental health, and oncology have been found to recruit younger patients with fewer high-risk characteristics compared with real-world cohorts3,20 and a meta-analysis of cardiology RCTs has found unselected registry patients to be twice as likely to die as those participating in a clinical trial.21 Some difference in mortality between RCT cohorts and the general population receiving dialysis should be expected, as many interventions in nephrology may not be relevant to patients with clearly limited life expectancy (eg, tight phosphate control, extended dialysis hours, or intensive cardiovascular risk reduction). Despite this fact, the magnitude of the difference between RCT cohorts and the registry population in our study was large. Even when restricted to studies recruiting predominantly or only in the United States, which were more similar in comorbidity burden to the USRDS population, the RCT mortality rate was still more than 40% lower than the registry population. The observed mortality rate of 8.9 per 100 patient-years in RCT cohorts was also substantially lower than that reported for dialysis cohorts in Europe (19.2)22 and Australia (13.3),23 although clearly higher than that reported in Japan (4.7).24 Reasons for the difference in mortality are likely to include study design (eg, inclusion and exclusion criteria and recruitment methods),3,25 volunteer bias, and differences between study and nonstudy sites.
Despite the difference in mortality, we found that study cohorts tended to have a higher burden of cardiovascular disease than the general population undergoing dialysis. Although differences in comorbidity ascertainment between USRDS and clinical trials may contribute to this difference, it may also reflect the inclusion of RCTs targeting cardiovascular disease in dialysis recipients (and so specifically recruiting participants at elevated cardiovascular risk). The coexistence of high-risk prognostic features with lower mortality raises the possibility that, compared with nonparticipants, study participants have other positive prognostic characteristics (eg, a history of treatment compliance or stable, rather than recently active, cardiovascular disease) that were not measured in this analysis. Moreover, previous research has found study participants to differ from the general patient population in a range of socioeconomic characteristics that have the potential to influence their underlying risk of adverse outcomes or response to therapy.26
Our analysis suggested that commercial sponsorship and more recent year of publication were independently associated with lower mortality. This finding raises the possibility that commercially sponsored RCTs in nephrology are less generalizable consistent with the bias described in commercially sponsored RCTs in other fields.27 However, this hypothesis was not supported by our finding that sponsor type had no association with participant age or proportion of patients with diabetes. It remains plausible that the association between sponsor type and mortality reflects other aspects of study design and target population that were not included in our model. The association with year of publication may reflect the reduction in mortality among patients undergoing dialysis seen in USRDS data from 2007 to 2015. It is not consistent with an improvement in RCT generalizability in the past decade, which would be expected to manifest as an increase in study mortality rates with time as the gap between trial and real-world populations narrowed. We also found that, at least in univariate analysis, cluster-randomized studies were more generalizable in terms of age and proportion with diabetic nephropathy, but not in terms of mortality rate. However, the hypothesis that cluster randomized trials were more generalizable was not supported in multivariable analysis.
This review also identifies areas for improvement in dialysis trial methods. Almost 1 in 5 studies included an age-based exclusion criteria, some as low as 65 years. Avoiding unjustified age-based exclusion criteria represents a simple way to improve study generalizability. Approximately two-thirds of large trials were open-label, and while many interventions in dialysis are difficult to blind, a lack of double-blinding is associated with exaggeration of the intervention effect, particularly where the outcome is subjective.28 There is also growing recognition of the importance of patient-centered outcomes—whether these are clinical events or patient-reported outcomes.29 However, the primary outcome in two-thirds of included studies was a surrogate rather than an event of importance to the patient. This finding accords with an earlier survey of hemodialysis randomized trials (without a study size limitation) that found mortality to be reported in only 20% of studies, cardiovascular diseases to be reported in 12%, and quality of life to be reported in 9% of studies.30 We also demonstrated a disproportionate number of male study participants in the RCT population. Although this finding may reflect differences in the sex distribution of patients in countries outside of the United States undergoing dialysis,31 it is an area that may warrant further research.
On the face of it, recruitment and follow-up of patients with end-stage kidney disease, who are perhaps uniquely dependent on a particular health care institution for their treatment and who mostly attend 3 times per week, should be straightforward. However, our results confirm the ongoing challenge of small (median number of participants, 212) and short-term (median follow-up, 7 months) trials in nephrology, particularly considering that this survey specified only studies randomizing at least 100 participants. Although, in part, this finding is likely to reflect institutional factors such as the commercial priorities of large institutions that provide dialysis and the relative shortage of research funding (both governmental and philanthropic) for nephrology,32 it may also be associated with more fundamental difficulties inherent in the specialty. For instance, clinicians may be reluctant to engage patients who are already undertaking burdensome treatment in research; in addition, studies often require lengthy follow-up periods to observe clinically meaningful outcomes. Fortunately, the nephrology community has now begun to explore innovative research methods that integrate trial conduct with registries and administrative data with the aim of reducing costs and the burden on participants while also improving the size and quality of RCTs.33
Limitations
An important limitation of the present study is that the comparison was restricted to USRDS. The initial aim of comparing study participant characteristics with those of patients in multiple end-stage kidney disease registries foundered owing to the limitations and wide variation in reported parameters across different registries. We suspect, however, that similar differences would be identified if our data were compared directly with European, Japanese, or Canadian registries where published data suggest that the mean age of dialysis patients is similar to or higher than that in the United States.34,35,36 In addition, while the characteristics of study participants recruited from 1 global region may differ from those recruited elsewhere, evidence from such studies still informs the practice of clinicians and guideline authors internationally. More than one-quarter of RCTs in this analysis were international, with the median number of participating countries being 7.5. Moreover, meta-analysis frequently results in pooling of study population data from disparate regions, which presents a challenge for clinicians and policymakers who are required to make decisions based on available evidence. Our results need not discourage the use of evidence from a variety of sources, but do serve as a reminder that caution is required when generalizing evidence from any source to a local population.
The primary limitation of this study was the lack of individual patient data. The heterogeneity in study reporting means that the analysis was limited to a few key population characteristics and comorbidities. Even within these categories, we were reliant on reported aggregate data and could not independently verify their accuracy. There were also a substantial number of manuscripts for which we were unable to determine if any participants died. We suspect that this lack of clarity reflects an assumption by authors that readers would assume a lack of deaths in short studies of stable study participants. This finding leads us to believe that the true rate of within-study mortality may be lower than our analysis suggests. The limitations of study reporting also prevented us from exploring more nuanced indicators of participant health such as comorbidity indices, functional metrics, or measures of treatment adherence. We were also unable to describe socioeconomic characteristics such as race, ethnicity, income, or educational level. Important differences in these characteristics between participants and the general population have been reported in other disciplines.37 The inclusion criteria for this review (≥100 participants and ≥2 sites), while necessarily arbitrary, were designed to identify larger and later-phase studies more likely to be considered generalizable. The finding that increasing study size and site number was not associated with higher participant age, proportion of patients with diabetes, or mortality (our chosen surrogate measures of generalizability) supports our selection criteria in that higher values for inclusion would not have resulted in a more generalizable cohort. Finally, we did not make adjustment for multiple comparisons and so our conclusions beyond the primary outcome of age are necessarily tentative.
Conclusions
Large multicenter trials in dialysis enroll younger participants with a different pattern of comorbidities and substantially lower mortality than the general population of individuals undergoing dialysis. The limitations of randomized evidence when applied to older or frailer populations should be recognized and efforts to broaden the generalizability of randomized trials are warranted.
eTable 1. Included Manuscripts and Study Characteristics
eTable 2. Highest 10 Countries by Study Participation, Number of Sites and Study Sponsor
eTable 3. Number of Studies per Country
eFigure 1. Linear Regression of Year of Publication by Participant Age, Prevalence of Diabetes and Study Mortality Rate
eFigure 2. Year of Publication of Included Studies (N = 189)
eAppendix. Methods
References
- 1.Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines, 1: introduction—GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383-394. doi: 10.1016/j.jclinepi.2010.04.026 [DOI] [PubMed] [Google Scholar]
- 2.Rothwell PM. External validity of randomised controlled trials: ‘to whom do the results of this trial apply?’. Lancet. 2005;365(9453):82-93. doi: 10.1016/S0140-6736(04)17670-8 [DOI] [PubMed] [Google Scholar]
- 3.Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials. 2015;16:495. doi: 10.1186/s13063-015-1023-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Golomb BA, Chan VT, Evans MA, Koperski S, White HL, Criqui MH. The older the better: are elderly study participants more non-representative? a cross-sectional analysis of clinical trial and observational study samples. BMJ Open. 2012;2(6):e000833. doi: 10.1136/bmjopen-2012-000833 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Susukida R, Crum RM, Ebnesajjad C, Stuart EA, Mojtabai R. Generalizability of findings from randomized controlled trials: application to the National Institute of Drug Abuse Clinical Trials Network. Addiction. 2017;112(7):1210-1219. doi: 10.1111/add.13789 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Thorpe KE, Zwarenstein M, Oxman AD, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ. 2009;180(10):E47-E57. doi: 10.1503/cmaj.090523 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Inrig JK, Califf RM, Tasneem A, et al. The landscape of clinical trials in nephrology: a systematic review of Clinicaltrials.gov. Am J Kidney Dis. 2014;63(5):771-780. doi: 10.1053/j.ajkd.2013.10.043 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Strippoli GFM, Craig JC, Schena FP. The number, quality, and coverage of randomized controlled trials in nephrology. J Am Soc Nephrol. 2004;15(2):411-419. doi: 10.1097/01.ASN.0000100125.21491.46 [DOI] [PubMed] [Google Scholar]
- 9.Blosser CD, Huverserian A, Bloom RD, et al. Age, exclusion criteria, and generalizability of randomized trials enrolling kidney transplant recipients. Transplantation. 2011;91(8):858-863. doi: 10.1097/TP.0b013e31820f42d9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.United States Renal Data System . 2015 USRDS Annual Data Report: Epidemiology of Kidney Disease in the United States: Vol 2, Chap 4: Vascular Access. Bethesda, MD: National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2015. [Google Scholar]
- 11.United States Renal Data System . 2015 USRDS Annual Data Report: Epidemiology of Kidney Disease in the United States: Reference Table C (Patient Characteristics). Bethesda, MD: National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2015. [Google Scholar]
- 12.United States Renal Data System . 2015 USRDS Annual Data Report: Epidemiology of Kidney Disease in the United States: Reference Table H (Mortality and Causes of Death). Bethesda, MD: National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2015. [Google Scholar]
- 13.United States Renal Data System . USRDS 2013 Annual Data Report: Atlas of Chronic Kidney Disease and End-Stage Renal Disease in the United States: Vol 2, Chap 1: Incidence, Prevalence, Patient Characteristics, and Modalities. Bethesda, MD: National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2013. [Google Scholar]
- 14.United States Renal Data System . USRDS 2013 Annual Data Report: Atlas of Chronic Kidney Disease and End-Stage Renal Disease in the United States: Vol 2, Chap 2: Clinical Indicators and Preventive Care. Bethesda, MD: National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2013. [Google Scholar]
- 15.Woodward M. Epidemiology: Study Design and Data Analysis. 3rd ed. Hoboken, NJ: CRC Press; 2013. doi: 10.1201/b16343 [DOI] [Google Scholar]
- 16.Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. http://www.handbook.cochrane.org. Updated March 2011. Accessed August 31, 2018.
- 17.DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177-188. doi: 10.1016/0197-2456(86)90046-2 [DOI] [PubMed] [Google Scholar]
- 18.Dember LM, Lacson E Jr, Brunelli SM, et al. The TiME Trial: a fully embedded, cluster-randomized, pragmatic trial of hemodialysis session duration. J Am Soc Nephrol. 2019;30(5):890-903. doi: 10.1681/ASN.2018090945 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;375(5):454-463. doi: 10.1056/NEJMra1510059 [DOI] [PubMed] [Google Scholar]
- 20.Evans A, Kalra L. Are the results of randomized controlled trials on anticoagulation in patients with atrial fibrillation generalizable to clinical practice? Arch Intern Med. 2001;161(11):1443-1447. doi: 10.1001/archinte.161.11.1443 [DOI] [PubMed] [Google Scholar]
- 21.Laursen PN, Holmvang L, Lønborg J, et al. Comparison between patients included in randomized controlled trials of ischemic heart disease and real-world data: a nationwide study. Am Heart J. 2018;204:128-138. doi: 10.1016/j.ahj.2018.05.018 [DOI] [PubMed] [Google Scholar]
- 22.de Jager DJ, Grootendorst DC, Jager KJ, et al. Cardiovascular and noncardiovascular mortality among patients starting dialysis. JAMA. 2009;302(16):1782-1789. doi: 10.1001/jama.2009.1488 [DOI] [PubMed] [Google Scholar]
- 23.Australia & New Zealand Dialysis & Transplant Registry . Chapter 3: mortality in end stage kidney disease. http://www.anzdata.org.au/anzdata/AnzdataReport/39thReport/c03_deaths_v2.1_20170418.pdf. Accessed March 31, 2018.
- 24.Tsuruta Y, Nitta K, Akizawa T, et al. Association between allopurinol and mortality among Japanese hemodialysis patients: results from the DOPPS. Int Urol Nephrol. 2014;46(9):1833-1841. doi: 10.1007/s11255-014-0731-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Hordijk-Trion M, Lenzen M, Wijns W, et al. ; EHS-CR Investigators . Patients enrolled in coronary intervention trials are not representative of patients in clinical practice: results from the Euro Heart Survey on Coronary Revascularization. Eur Heart J. 2006;27(6):671-678. doi: 10.1093/eurheartj/ehi731 [DOI] [PubMed] [Google Scholar]
- 26.Busija L, Tao LW, Liew D, et al. Do patients who take part in stroke research differ from non-participants? implications for generalizability of results. Cerebrovasc Dis. 2013;35(5):483-491. doi: 10.1159/000350724 [DOI] [PubMed] [Google Scholar]
- 27.Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326(7400):1167-1170. doi: 10.1136/bmj.326.7400.1167 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Savović J, Jones H, Altman D, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012;16(35):1-82. doi: 10.3310/hta16350 [DOI] [PubMed] [Google Scholar]
- 29.Standardised Outcomes in Nephrology . The SONG Handbook. Version 1.0. http://songinitiative.org/reports-and-publications/. Published June 1, 2017. Accessed July 6, 2018.
- 30.Sautenet B, Tong A, Williams G, et al. Scope and consistency of outcomes reported in randomized trials conducted in adults receiving hemodialysis: a systematic review. Am J Kidney Dis. 2018;72(1):62-74. doi: 10.1053/j.ajkd.2017.11.010 [DOI] [PubMed] [Google Scholar]
- 31.Hecking M, Bieber BA, Ethier J, et al. Sex-specific differences in hemodialysis prevalence and practices and the male-to-female mortality rate: the Dialysis Outcomes and Practice Patterns Study (DOPPS). PLoS Med. 2014;11(10):e1001750. doi: 10.1371/journal.pmed.1001750 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Bryan L, Ibrahim T, Zent R, Fischer MJ. The kidney research predicament. J Am Soc Nephrol. 2014;25(5):898-903. doi: 10.1681/ASN.2013121313 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Dember LM, Archdeacon P, Krishnan M, et al. Pragmatic trials in maintenance dialysis: perspectives from the Kidney Health Initiative. J Am Soc Nephrol. 2016;27(10):2955-2963. doi: 10.1681/ASN.2016030340 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Wong MM, McCullough KP, Bieber BA, et al. Interdialytic weight gain: trends, predictors, and associated outcomes in the International Dialysis Outcomes and Practice Patterns Study (DOPPS). Am J Kidney Dis. 2017;69(3):367-379. doi: 10.1053/j.ajkd.2016.08.030 [DOI] [PubMed] [Google Scholar]
- 35.European Renal Association–European Dialysis and Transplant Association. ERA-EDTA Registry Annual Report 2013. Amsterdam, the Netherlands: Academic Medical Center, Department of Medical Informatics; 2015. [Google Scholar]
- 36.Canadian Institute for Health Information . Treatment of end-stage organ failure in Canada, Canadian Organ Replacement Register, 2007. to 2016: data tables, end-stage kidney disease and kidney transplants. https://www.cihi.ca/sites/default/files/document/corr_ar-kidney-data-tables-en.xlsx. Accessed March 19, 2019.
- 37.Coakley M, Fadiran EO, Parrish LJ, Griffith RA, Weiss E, Carter C. Dialogues on diversifying clinical trials: successful strategies for engaging women and minorities in clinical trials. J Womens Health (Larchmt). 2012;21(7):713-716. doi: 10.1089/jwh.2012.3733 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eTable 1. Included Manuscripts and Study Characteristics
eTable 2. Highest 10 Countries by Study Participation, Number of Sites and Study Sponsor
eTable 3. Number of Studies per Country
eFigure 1. Linear Regression of Year of Publication by Participant Age, Prevalence of Diabetes and Study Mortality Rate
eFigure 2. Year of Publication of Included Studies (N = 189)
eAppendix. Methods