Lessons for Successful Study Enrollment from the Veterans Affairs/National Institutes of Health Acute Renal Failure Trial Network Study

Abstract

Background and objectives: Design elements of clinical trials can introduce recruitment bias and reduce study efficiency. Trials involving the critically ill may be particularly prone to design-related inefficiencies.

Design, setting, participants, & measurements: Enrollment into the Veterans Affairs/National Institutes of Health Acute Renal Failure Trial Network Study was systematically monitored. Reasons for nonenrollment into this study comparing strategies of renal replacement therapy in critically ill patients with acute kidney injury were categorized as modifiable or nonmodifiable.

Results: 4339 patients were screened; 2744 fulfilled inclusion criteria. Of these, 1034 were ineligible by exclusion criteria. Of the remaining 1710 patients, 1124 (65.7%) enrolled. Impediments to informed consent excluded 21.4% of potentially eligible patients. Delayed identification of potential patients, physician refusal, and involvement in competing trials accounted for 4.4, 2.7, and 2.3% of exclusions. Comfort measures only status, chronic illness, chronic kidney disease, and obesity excluded 11.8, 7.8, 7.6, and 5.9% of potential patients. Modification of an enrollment window reduced the loss of patients from 6.6 to 2.3%.

Conclusions: The Acute Renal Failure Trial Network Study's enrollment efficiency compared favorably with previous intensive care unit intervention trials and supports the representativeness of its enrolled population. Impediments to informed consent highlight the need for nontraditional acquisition methods. Restrictive enrollment windows may hamper recruitment but can be effectively modified. The low rate of physician refusal acknowledges clinical equipoise in the study design. Underlying comorbidities are important design considerations for future trials that involve the critically ill with acute kidney injury.

Elements in the design of randomized clinical trials (RCT) can introduce systematic bias in patient recruit-ment and reduce efficiency of patient accrual. Because of the complexities of care of critically ill patients, for whom concurrent treatment trials are prevalent, rapid changes in patient census occur, informed consent limitations exist, and high mortality rates prevail, RCT involving the critically ill may be particularly prone to design-related inefficiencies and bias in patient enrollment. Such bias can result in an exaggeration or diminution of treatment effects and ultimately confuse rather than inform clinical practice and health care policy (1).

Important information regarding the representativeness of clinical trial participants can be gleaned by evaluating characteristics of enrolled and nonenrolled individuals. Comparison of the two cohorts can provide a yardstick for gauging the external validity of a trial's findings (2), although a detailed description of the clinical characteristics of nonenrolled individuals is often omitted or ignored in published reports (3,4).

The Consolidated Standards of Reporting Trials (CONSORT) statement calls for transparency in study design and improved quality of reporting (2,5). Its recommendation to detail exclusion criteria allows for quantification of the effect of each criterion on patient enrollment. A cumulative assessment of reasons for exclusion yields valuable information about the trial's generalizability.

In keeping with the CONSORT statement, we detailed the process of patient selection for the Veterans Affairs/National Institutes of Health Acute Renal Failure Trial Network (ATN) Study, a multicenter RCT that compared two strategies of renal replacement therapy (RRT) in critically ill patients with acute kidney injury (AKI) (6). We quantified the study's enrollment rate and reasons for exclusion using a screening log, which allowed us to assess the generalizability of the randomly assigned patients and to determine the impact of a change in exclusion criteria.

Materials and Methods

A detailed description of the ATN Study has been previously published (6). In brief, the ATN Study compared a less intensive with a more intensive management strategy for RRT (hemodialysis and hemodiafiltration) in critically ill patients with AKI. The primary study outcome was 60-d all-cause mortality. A cumulative screening log was compiled weekly by the data coordinating center using the data reported on study patient eligibility screening forms.

Potentially eligible patients were critically ill patients who met inclusion criteria of (1) a clinical diagnosis of AKI secondary to acute tubular necrosis, (2) AKI (Δ serum creatinine [SCr] ≥2 mg/dl for men (≥1.5 mg/dl for women) during a period of ≤4 d or oliguria >24 h), (3) age ≥18 yr, and (4) need for RRT.

Potentially eligible patients were excluded when they had chronic kidney disease (CKD; defined as baseline SCr >2 mg/dl in men or 1.5 mg/dl in women); were kidney transplant recipients, incarcerated prisoners, pregnant, or morbidly obese (weight >128.5 kg); were not candidates for support with RRT; were moribund or had care limited to comfort measures only (CMO); or were not expected to survive 28 d as a result of their underlying chronic medical illness. Physician refusal, concurrent participation in another interventional trial, delayed identification by study personnel (i.e., receipt of more than one intermittent hemodialysis session or >24 h of continuous RRT), or delayed start of initial RRT beyond the enrollment window (initial window limit: blood urea nitrogen [BUN] >60 mg/dl for ≥48 h; subsequently amended to BUN >100 mg/dl for ≥72 h after meeting the definition of AKI) also resulted in exclusion.

Patients who met all inclusion and no exclusion criteria constituted the fully eligible cohort. Fully eligible study patients were not enrolled when they or their proxies were unable or unwilling to provide informed consent. The enrollment rate was calculated as the ratio of enrolled and randomly assigned patients to fully eligible patients. Potentially eligible patients who met all inclusion criteria were evaluated for reasons for nonenrollment. The more frequent reasons for nonenrollment were broadly categorized as either modifiable or nonmodifiable; the percentage of potentially eligible patients who were excluded for each reason was determined.

Results

In 44 mo, 4339 patients were screened, 2744 of whom satisfied all inclusion criteria (Figure 1). Of these potentially eligible patients, 1034 (37.7%) were ineligible as a result of the presence of one or more exclusion criteria. Of the remaining 1710 fully eligible patients (met all inclusion and no exclusion criteria), 1124 were enrolled, representing 25.9% of all patients screened, 41.0% of potentially eligible patients, and 65.7% of fully eligible patients, respectively. The reasons for rejection of potentially eligible patients are shown in Figure 1 and Table 1.

Figure 1.

Algorithm and reasons for patient enrollment or exclusion. Patients may have had more than one exclusion criterion. *A total of 593 potentially eligible patients before modification of enrollment window and 2151 potentially eligible patients afterward.

Table 1.

Percentage of potentially eligible patients who were rejected by reason for exclusion^a

Reasons	%
Modifiable
lack of consent	21.4
patient/surrogate refusal	10.9
surrogate unavailable	7.0
enrollment window
original/postrevision	6.6/2.3
delayed notification of study personnel	4.4
physician refusal	2.7
other study	2.3
Nonmodifiable
CMO/moribund	11.8
chronic illness	7.8
CKD	7.6
obesity	5.9

^aCKD, chronic kidney disease; CMO, comfort measures only.

Limitations in acquiring informed consent were the most common barrier to enrollment. A total of 586 patients were excluded because of absence of informed consent, representing 21.4% of potentially eligible patients and 34.3% of the fully eligible cohort. Because fewer than 10% of patients had decision-making capacity as the result of their acute illness, surrogate consent was required for the majority of enrolled patients. Nonavailability of a surrogate to provide informed consent excluded 193 patients (7.0% of potentially eligible patients and 11.2% of all fully eligible patients) and constituted nearly one third (32.9%) of patients who were not enrolled as a result of absence of informed consent. A total of 298 potentially eligible patients (or their surrogates) declined to participate in the study (10.9% of potentially eligible patients and 17.4% of the fully eligible cohort).

An eligibility window limiting the time frame for patient recruitment was identified as a small but significant barrier to enrollment. Modification of the window after 9 mo of patient accrual reduced its negative impact on patient recruitment, decreasing the loss of potentially eligible patients from 6.6 to 2.3%.

Delayed identification of potential study patients was responsible for 4.4% of missed enrollments in the potentially eligible cohort. Noninvestigator physician refusal to permit participation of the patient in the trial was unusual, accounting for the exclusion of only 2.7% of the potentially eligible cohort. Similarly, involvement in competing trials was an uncommon barrier to enrollment, excluding only 2.3% of potentially eligible patients.

Nonmodifiable exclusion criteria, including CMO status, chronic underlying medical illness, CKD, and morbid obesity, precluded enrollment of notable fractions of potentially eligible patients (11.8, 7.8, 7.6, and 5.9%, respectively). Patients who were excluded for obesity had a mean body weight at screening of 154.3 ± 28.1 kg. The mean SCr of those who were excluded as a result of the presence of CKD was 2.9 ± 1.9 mg/dl.

Discussion

Difficulty in the interpretation of RCT results as a result of obscurity in trial design has been raised as a significant concern in the methodologic literature of clinical research (7). The initial CONSORT statement called for improved transparency in study design (5), and even more explicit criteria for describing patient recruitment and reasons for exclusion were specified in the CONSORT II revision to improve further the readers’ understanding of a trial's patient population (1). Closer scrutiny of patient recruitment and enrollment efficiency in clinical trials has also been encouraged for practical and ethical reasons, because these data can be informative regarding feasibility of ongoing trials, timeliness of study completion, minimization of trial costs, and optimal use of limited resources (8).

In most trials, a marked surplus of potential patients are screened for eligibility to offset impediments to enrollment (9). In the past decade, recruitment efficiencies (defined as the percentage of eligible patients enrolled) for large, interventional critical care RCT conducted have most commonly been reported to be <20% (10–14) with a few notable exceptions (15,16). In contrast, the ATN Study demonstrated excellent enrollment efficiency, recruiting 41% of potentially eligible patients and randomly assigning 66% of eligible patients. A relatively high enrollment efficiency not only is beneficial from the standpoint of research costs, but also suggests greater generalizability of a trial's results.

Gauging the true efficiency of reported trials has been difficult because of imprecision or absence of reporting of the trial enrollment process (3,7). Adoption of a common lexicon and a requirement for quantifying not only enrolled patients but also those screened, determined potentially eligible, and then determined truly eligible would facilitate transparency of trial design and calculation of trial efficiency and enhance interpretation of study results, particularly with regard to their generalizability.

Use of the CONSORT reporting criteria has highlighted the issue of patient exclusions as a barrier to study enrollment. A recent analysis of two series of large clinical oncology trials in the post-CONSORT era revealed not only that the number of exclusion criteria had nearly doubled over the past two decades but also that this increase had occurred without a clear rationale and was largely due to perpetuation of exclusion criteria in sequential trials (11). Consequently, exclusion criteria considered for the ATN Study were carefully developed and closely scrutinized for their adverse impact on patient enrollment.

We used a screening log, as proposed by others, to pinpoint criteria that were highly leveraged with respect to patient accrual and evaluated corrective strategies to optimize enrollment (11,17). Of the criteria that adversely affected enrollment efficiency, the nine most frequent were examined and broadly categorized as either modifiable or nonmodifiable barriers to enrollment.

Modifiable Barriers to Study Recruitment

Limitations in acquiring informed consent were the most significant modifiable barriers to enrollment, contributing to the exclusion of 21.4% of potentially eligible patients. One third of these nonenrolled patients were excluded because the patient lacked decision-making capacity and the appropriate surrogate decision maker was not available to provide consent. These results are similar to observations in other critical care trials. In the Acute Respiratory Distress Syndrome (ARDS) Network's Fluid and Catheter Therapy Trial (FACTT) Study, 10% of potentially eligible patients were precluded from enrollment as a result of lack of consent, with 58% of these due to nonavailability of the patient's proxy (18). Similarly, in the observational, multicenter Program to Improve Care in Acute Renal Disease (PICARD) study, absence of a proxy to provide consent accounted for up to 40% of excluded patients, documenting that such obstacles to recruitment are not unique to RCT (19).

The validity of written informed consent in critical care has been debated, especially when obtained via proxy (20,21); however, signature documentation of informed consent is a regulatory requirement in the United States and the majority of other countries (22). Sufficient lead time to reasonably seek written informed consent from the patient or a proxy is therefore a prudent design consideration in future trials that involve the critically ill to avoid the unnecessary exclusion of otherwise eligible patients. In addition, the incorporation of alternative strategies for documentation of signature for informed consent, such as electronic transmittal, can facilitate the ability to enroll patients with acutely impaired decision-making capacity by surrogates who are unable to be present at the study site during limited windows of eligibility.

The design of the ATN Study specified an “enrollment window” in an attempt to achieve a degree of uniformity in the timing of initiation of RRT for enrolled patients. Periodic audits of the ATN Study screening log highlighted the negative impact that this enrollment window was having on patient accrual. It became readily apparent that there was no consensus in practice outside of the study setting regarding the indications for initiation of RRT. Patients were excluded both because of failure to fulfill the enrollment window's criteria for initiation of renal support (early initiation) and because of failure to initiate therapy within 48 h of meeting the study definition of acute renal failure and reaching a BUN of >60 mg/dl (late initiation). Nearly 7% of potentially eligible patients in the trial's first 10 mo were excluded as a result of this enrollment window. The mean BUN at the initiation of RRT in these excluded patients was 71 mg/dl, with 25% having a BUN >103 mg/dl. Revision of the enrollment criteria, eliminating the “window” but excluding patients in whom the BUN was >100 mg/dl for >72 h after meeting the study definition of AKI, removed an unintended barrier to patient recruitment while increasing the generalizability of the study population. After implementation of this change, fewer than 3% of potentially eligible patients were excluded as a result of prolonged untreated azotemia.

Enrollment windows are criteria of precision and account for the greatest increase in eligibility criteria reported in the past several decades (8). A recent meta-analysis of acute stroke trials highlighted the importance of the enrollment window as a potential barrier to enrollment; a stringent window was the primary criterion that predicted reduced recruitment efficiency (23). Although the merits of a precise enrollment window may be debated, at a minimum, a clear biologic basis should be evident before invoking a restrictive enrollment window as an exclusion criterion.

Although elimination of the enrollment window was beneficial from the standpoint of patient recruitment, changing eligibility criteria during the conduct of a study does introduce the possibility of bias and may necessitate modification of the study analytic plan. In this case, the elimination of the enrollment window occurred relatively early in the course of the study, after enrollment of <20% of the overall study population. Because enrollment by treatment arm was well balanced both before and after this change was made, it is unlikely to have introduced bias as a result of imbalance in the application of the change across the two treatment arms.

Delayed identification of potentially eligible patients by study personnel as a result of late notification by treating physicians excluded 4.4% of potentially eligible patients. Privacy constraints under the Health Insurance Portability and Accountability Act of 1996 can limit identification of potential study patients and restrict screening of potentially eligible patients by investigators lacking a waiver of authorization. Late notification of study personnel increases the time constraints associated with obtaining informed consent and can preclude the opportunity to present a study to patients or their families while naive to therapeutic interventions. It also leads to disproportionate attrition of patients with the greatest mortality and can result in a “death before consent” bias (19).

Noninvestigator physician refusal was an infrequent obstacle to enrollment, accounting for nonenrollment of only 2.7% of potentially eligible patients. In contrast, physician refusal was the second most common cause of nonenrollment of patients in two recent critical care trials, accounting for 11 and 16% of nonenrolled patients, respectively (12,13), whereas in a trial of catheter management in the critically ill, treating physician refusal accounted for 50% of patient nonenrollment (15). Because treating physician “buy-in” was expected to be an important determinant of the ATN Study success, multiple strategies reported to foster protocol acceptance by noninvestigator practitioners were used at study sites, including involvement of multiple disciplines in the study team, repeated educational sessions for intensive care unit (ICU) staff, provision of research information, stakeholder involvement in protocol development, prospective protocol agreement from local clinicians, simplicity in study design, and ease in protocol implementation (11,17). In addition, our low physician refusal rate supports the premise of widespread acknowledgment of clinical equipoise in the study's design by noninvestigator physicians.

Involvement in competing trials was an uncommon barrier to enrollment. Only a small fraction (2.3%) of potentially eligible ATN patients were excluded as a result of enrollment in concurrent interventional trials. In contrast, other investigators have reported that participation in competing studies excluded 13% of potential patients (423 of 3245) (12). This markedly lower rate may reflect that AKI commonly excludes patients from other studies that are conducted in the ICU setting.

Nonmodifiable Barriers to Enrollment

Various comorbidities constituted the nonmodifiable barriers to enrollment. As the most extreme example of comorbidity, moribund patients and/or patients who were classified as CMO status accounted for 11.8% of potentially eligible patient exclusions, although co-selection of these criteria in screening may have yielded an overestimation of their effects on enrollment. Exclusion for hopelessness in critical care trials has been reported to account for 4 to 31% of patients (12,13,16). Exclusion as a result of hopelessness may not only affect available patient pools but also may introduce survivorship bias and affect expected event rates and mortality, making this an important consideration in trial design (19). In designing the ATN Study, exclusion of these patients was considered appropriate because dialysis is futile as a life-preserving therapy in these settings.

Underlying life-limiting chronic illness excluded 7.8% of potentially eligible patients. The rationale for this exclusion criterion is that management of RRT would have no impact on 60-d mortality, the primary study end point. This rate of study exclusion as a result of underlying chronic illness is similar to the rate of 10.6% of screened patients reported in another recent critical care study (13). Although it may be argued that exclusion of these patients reduces the generalizability of study results, the inclusion of such patients could dilute the intervention's effect and inflate the number of patients required to detect a significant difference between groups, thereby decreasing study feasibility (24).

Nearly 8% of potentially eligible patients were excluded from the ATN Study because of preexisting moderate to severe CKD (defined as a baseline serum creatinine >2 mg/dl in men and >1.5 mg/dl in women). Although data from the PICARD study suggested that acute on chronic disease may be common, representing up to 30% of hospitalized patients with AKI (25), the exclusion of patients with CKD from the ATN Study was deemed appropriate given differences in the natural history of acute on chronic kidney disease as compared with de novo AKI, with both lower mortality risk and lower probability of recovery of renal function in acute on chronic disease (25). Although exclusion of patients with moderate to advanced CKD may decrease the generalizability of the study's findings, only 208 (7.6%) of the 2744 potentially eligible patients were excluded on this basis.

Morbid obesity excluded 5.9% of potentially eligible patients. Because the stipulated dosing of continuous venovenous hemodiafiltration was indexed to body mass and the maximum combined flow rate for dialysate and replacement fluid that is provided by the continuous RRT equipment that is most commonly used at participating study sites at the initiation of the study was limited to 4.5 L/h, the delivery of an effluent flow of 35 ml/kg per h, as required in the intensive arm, was precluded for patients with morbid obesity. Thus, the exclusion of patients who weighed >128.5 kg was necessitated by the constraints of the existing medical technology at the study's initiation.

Conclusions

Enrollment efficiency in the ATN Study compared favorably with other intervention trials previously reported in the ICU and supports the representativeness of the study's enrolled population. Periodic audits of a screening log permitted early identification of important barriers to enrollment and provided a gauge of the effectiveness of criteria modification on subsequent patient accrual. Impediments to acquiring informed consent were the major modifiable barriers to patient recruitment and highlight the need for nontraditional methods of obtaining and documenting informed consent in future critical care trials.

The enrollment window was identified as an unintended barrier to patient recruitment and was readily modified to facilitate patient accrual. Excellent study acceptance, or buy-in, by noninvestigator colleagues was evidenced by the low rate of noninvestigator physician refusal and supports the premise of widespread acknowledgment of clinical equipoise in the study design. Underlying comorbidities were the basis for most nonmodifiable barriers to enrollment and should be serious design considerations in future trials that involve the critically ill.

Disclosures

None.