Subject Eligibility Criteria Can Substantially Influencethe Results of Alcohol-Treatment Outcome Research

Abstract

Objective:

Most alcohol-treatment studies exclude some patients from participation based on particular criteria (e.g., comorbid illegal drug abuse, homelessness). The current study evaluated whether such eligibility criteria can change the outcome results a study obtains.

Method:

Five widely used treatment research eligibility criteria—(1) psychiatric problems, (2) medical problems, (3) social-residential instability, (4) low motivation/noncompliance, and (5) drug problems—were applied to two samples of real-world alcohol patients whose outcomes were known. Comparing outcomes of the samples with and without the application of eligibility criteria produced estimates of bias in outcome results, as well as an assessment of change in statistical power.

Results:

Medical and psychiatric eligibility criteria produced a moderate bias in outcome estimates (e.g., a 10% or less change in outcome results). In contrast, social-residential instability, low motivation/noncompliance, and drug use produced a large (e.g., up to an 18% change) to a very large (e.g., up to a 51% change) bias in outcome estimates. Sensitivity analyses showed that these biases are even larger if eligibility criteria are operationalized in a broad rather than a narrow fashion. Contrary to expectation, eligibility criteria did not produce their theoretically expected benefit of increased statistical power.

Conclusions:

Researchers who use eligibility criteria should do so judiciously and interpret outcome results in light of potential bias introduced by the ineligibility of some patients for study enrollment. Efforts to integrate findings across treatment outcome studies should also consider how conclusions might be affected by the eligibility criteria used in different research areas.

Across medical specialties, many patients seen in clinical practice do not meet eligibility criteria for the treatment research that ostensibly guides their care. Hlatky and colleagues (1984), for example, found that only 4%-13% of a sample of cardiac patients would have met eligibility criteria for three influential trials of coronary artery bypass surgery. High rates of patient ineligibility for treatment research have also been found for cancers (e.g., Martin et al., 1984), Alzheimer's disease (Schneider et al., 1997), and panic disorder (Mavissakalian and Guo, 2002). Parallel findings have recently emerged in the alcohol field, showing that most treatment studies have extensive eligibility criteria, wherein a large proportion of patients are ineligible for research participation. In addition, those findings showed that eligibility criteria produce treatment research samples that differ substantially in baseline demographics and problem characteristics from typical alcohol patients (Blanco et al., submitted for publication; Humphreys, 2003; Humphreys and Weisner, 2000; Humphreys et al., 2005, 2007; Moncrieff and Drummond, 1998). The present study assessed whether and in what way this situation might change the outcome results of alcohol-treatment research projects.

Whether eligibility criteria influence the results of alcohol-treatment research is important to evaluate for at least two reasons: (1) from a clinical viewpoint, if it can be demonstrated that the outcomes of a treatment in a selected research sample also apply in those patients who were ineligible for study participation, basing clinical practice on the study's results is more logically compelling and ethically acceptable; and (2) from a research perspective, efforts to draw integrative conclusions about the effectiveness of treatments (e.g., systematic reviews, meta-analyses, and clinical practice guidelines) must be informed as to whether differences in results across outcome studies can be produced by application of various eligibility criteria. For example, in trying to judge the effectiveness of a treatment that has been shown quite effective in a trial that excluded patients with comorbid psychopathology—but that was ineffective in a trial that did not—it would be helpful to know if the difference in outcomes is attributable to the variation in eligibility criteria or to some other factor.

If eligibility criteria do in fact change outcome results, the direction of any bias (i.e., the difference in outcome results observed when exclusion criteria are applied vs if they were not) is difficult to predict. As our previous work has shown (Humphreys et al., 2005), the most common effect of eligibility criteria in the alcohol-treatment outcome literature has been to remove patients with poor prognoses from research samples (e.g., excluding patients with various medical and psychiatric comorbidities and those lacking stable living arrangements). On this basis, one might reasonably hypothesize that eligibility criteria cause many investigators to overstate the effectiveness of their treatments in real-world clinical practice. However, because the most troubled patients have the greatest room to improve, their exclusion from research could just as plausibly lead to an understatement of the value of alcohol treatment. Such effects are not unknown in the field. The New Orleans Homeless Substance Abuse Project found that, when clinical staff attempted to “cream” a research sample by covertly excluding more seriously ill patients, the resulting ceiling effect for possible improvement led the research project to conclude that the program was less effective than it really was in practice (Devine et al., 1997).

The other pathway through which eligibility criteria might affect outcome results is by increasing statistical power. Eligibility criteria should reduce sample heterogeneity, which in turn should make it more likely that the study finds a treatment effect when it is present (Lipsey, 1990). This benefit is indisputable at the level of statistical theory; whether it is realized in practice in alcohol-treatment research has yet to be examined empirically.

Given that eligibility criteria are widely used in alcohol-treatment research and in some cases are indispensable (e.g., when a certain type of patient may be harmed by the treatment being evaluated), assessing whether and in what way the criteria may influence outcome results is essential to determining what conclusions can be drawn from individual treatment-research projects and the treatment-outcome literature as a whole. In what we believe is the first effort to evaluate this issue empirically, we examine here the potential impact of several widely used eligibility criteria on the outcome findings of alcohol-treatment research.

Method

The approach of this study was to operationalize eligibility criteria and apply them to unselected samples of alcohol-treatment seekers whose outcomes were known. This allows comparison of the outcomes of patients who would be eligible for participation in a treatment outcome study under different criteria (the selected sample) versus outcomes for the entire unselected, real-world sample.

Treatment samples

Patient data were drawn from two research studies of individuals seeking treatment for substance-use disorders: (1) the Target Cities study (Guydish and Claus, 2002) and (2) the Washington State system evaluation (Carney et al., 2000). Because neither study used any eligibility criteria, samples of these studies represent the real world of service delivery to which treatment research aspires to generalize. In both studies, research interviewers at baseline and at 6-month follow-up assessed patients with the Addiction Severity Index (ASI), a valid and reliable assessment of medical, employment/support, drug, alcohol, family/social, legal, and psychiatric problems (McLellan et al., 1980, 1985). For the present study, all patients in these two studies who were rated by a research interviewer as being in need of alcohol treatment (vs treatment for only an illicit drug problem), according to the ASI, were included.

The Washington State sample (n = 502) of alcohol treatment seekers was drawn from 13 treatment programs around the state. About two thirds (62.0%) of participants were male; 7.3% were black; 34.1% were employed; and 26.7% had an ASI psychiatric severity score above .50, which is a marker for very serious psychopathology (Bovasso et al., 2001).

The Target Cities sample (n = 943) of individuals seeking treatment for alcohol-use disorders was drawn from 12 treatment programs throughout the United States. Although approximately the same percentage of the Target Cities sample was male and employed, as in the Washington State sample (61.9% and 33.1%, respectively), a much higher proportion of this sample was black (69.6%), and a somewhat lower proportion exceeded the threshold for serious psychopathology (16.2%).

Eligibility criteria and their operationalization

In a prior study, we reliably coded the eligibility criteria used by researchers in 683 alcohol-treatment outcome studies conducted between 1970 and 1998 (Humphreys et al., 2005). The database included all studies that had a follow-up, had at least five participants per condition, had at least some participants who were age 18 or older, and were published in English. The resultant collection of studies comprised both randomized trials and nonrandomized studies and included research conducted in Canada, Europe, Australia, and the United States. Coding revealed the prevalence of alcohol-treatment researchers' use of various eligibility criteria examined over a 28-year period (Humphreys et al., 2005). In a follow-up study, we examined six of the most widely used of these eligibility criteria and found that a number of them changed the demographic composition of research samples (Humphreys et al., 2007).

The present study examined the potential impact on treatment outcome of these same six criteria individually and in combination, with the exception of neurological criteria (e.g., organic brain syndrome). We found that this criterion excludes so few patients (e.g., about 1%) from research participation that it could not affect study outcomes even if the ineligible patients had extremely different outcomes than the rest of the sample.

The five widely used treatment research eligibility criteria were operationalized using the ASI in the same fashion as in our previous work. We operationalized each criterion in both more exclusive and less exclusive forms (Table 1), examining the effects of the less exclusive forms in our primary analyses and the more exclusive forms in sensitivity analyses.

Table 1.

More and less exclusive operationalizations of prevalent eligibility criteria using items from the Addiction Severity Index (ASI)

Basis for exclusion		Operationalization
Psychiatric problems	Less exclusive	At least three of the following: No. of times treated as inpatient for psychiatric problems > 1
		Experienced hallucinations, lifetime (yes/no)
		Experienced serious thoughts of suicide, past 30 days (yes/no)
		Attempted suicide, lifetime (yes/no)
		Been prescribed medication for psychiatric problems, past 30 days (yes/no)
	More exclusive	At least one of above
Medical problems	Less exclusive	ASI medical composite > 0.66 (>2.5 SD above mean)
	More exclusive	ASI medical composite > 0.58 (>2.0 SD above mean)
Drug use	Less exclusive	Any one of the following: No. of times overdosed on drugs > 0
		No. of days experienced drug problems, past 30 days, ≥ 5
		Used any of the following drugs at least 15 days of the past 30: heroin, methadone, opiates/analgesics, barbiturates, sedatives, cocaine, amphetamines, cannabis, hallucinogens, inhalants
	More exclusive	Any one of the following: No. of times overdosed on drugs > 0
		No. of days experienced drug problems, past 30 days, ≥ 1
		Used any of the above drugs at least 5 days of the past 30
Unmotivated/noncompliant	Less exclusive	Patient's rating of the importance of receiving alcohol-abuse treatment was <3 on a scale of 0 = not at all, 1 = slightly, 2 = moderately, 3 = considerably, 4 = extremely.
	More exclusive	Patient's rating of the important of treatment using the above scale was <4; or during interview, was patient obviously hostile? (yes/no)
Social and residential stability	Less exclusive	Any two of the following: Was usually unemployed during the past 3 years
		Has no stable living arrangement
		Is living in a controlled environment
	More Exclusive	Any one of above

Analysis strategy

The primary outcome of interest was improvement in the ASI alcohol composite score from baseline to follow-up. We compared how two different samples scored on this outcome measure, namely all treatment seekers and the subset that would have been eligible for research participation under various eligibility criteria. The former represents how treatment performs in the real world of service delivery, and the latter represents how treatment performs in the research world, where only a subset of patients is studied. How different these two numbers are depends on the size of the excluded group and the degree to which its outcomes differ from those of the eligible group. Thus, if an eligibility criterion rules out a large proportion of patients, a study's results could be affected even if the ineligible patients have only modestly different outcomes than do eligible patients. In contrast, if a criterion only makes a small proportion of patients ineligible, this will not affect a study's outcome results unless those ineligible patients have markedly different outcomes than the eligible patients.

There is no direct statistical test for whether the means of the selected and total samples differ, mainly because the assumption of independence is not met (i.e., all the ineligible patients by definition are in the unselected sample). However, it is possible to determine the magnitude of bias that may result from application of the criteria. We applied each eligibility criterion to both the Target Cities and Washington State data and plotted the 95% confidence intervals for the mean outcome for the selected and overall samples. We quantify the magnitude of the observed bias simply with the mean difference. For example, if an overall sample of unselected patients had an ASI alcohol improvement of .30, whereas a subsample selected under a particular eligibility criteria had a score of .33, this would be a bias of +10%, meaning that the removal of potential subjects by the eligibility criteria raised the obtained outcome results by 10% (from .30 to .33). In contrast, if under a different eligibility criterion the selected sample had an outcome of .24, this would be a bias of -20%, because the effect of applying the criteria was to lower the study's estimate of treatment effectiveness by 20% (from .30 to .24).

Results

Overall improvement in ASI alcohol scores was .196 (95% confidence interval [CI]: .178-.213) for the Target Cities sample and .279 (95% CI: .251-.307) for the Washington State sample. The mean (95% CI) ASI alcohol improvement scores for the samples selected by the five exclusion criteria and the most common combinations are presented in Tables 2 and 3. The differences between the total and selected means for individual criteria are presented in Table 2. Figure 1 shows the amount of positive or negative bias introduced by individual application of the five widely used treatment research eligibility criteria in each sample. The social-residential criteria resulted in upward bias in both samples. The amount of upward bias was 2.5% in the Target Cities sample and 15.6% in the Washington State sample.

Table 2.

Change in Addiction Severity Index alcohol composite score among full treatment sample and samples selected by single eligibility criteria in Target Cities (TC) and Washington State (WA) data

Location	Total mean (95% CI)	Social-res. selected (95% CI)	Noncomp. selected (95% CI)	Drug selected (95% CI)	Medical selected (95% CI)	Psychiatric selected (95% CI)
TC	.196 (.178-.213)	.200 (.176-.225)	.284 (.263-.306)	.116 (.088-.144)	.187 (.167-.206)	.197 (.173-.221)
WA	.279 (.251-.308)	.323 (.282-.364)	.295 (.265-.326)	.299 (.240-.358)	.251 (.220-.282)	.251 (.210-.299)

Notes: Res. = residential; noncomp. = noncompliance/unmotivated; CI = confidence interval.

Table 3.

Change in Addiction Severity Index alcohol composite among full treatment sample and samples selected by multiple eligibility criteria in Target Cities (TC) and Washington State (WA) data

Location	Total mean (95% CI)	Psychiatric + Unmotivated selected (95% CI)	Psychiatric + medical selected (95% CI)	Unmotivated + medical selected (95% CI)	Psychiatric + unmotivated + medical selected (95% CI)
TC	.196 (.178-.213)	.306 (.277-.336)	.190 (.164-.215)	.280 (.256-.305)	.299 (.267-.330)
WA	.279 (.251-.307)	.279 (.229-.330)	.231 (.184-.278)	.264 (.230-.288)	.256 (.204-.308)

Note: CI = confidence interval.

Figure 1.

Effect of eligibility criteria on change in Addiction Severity Index alcohol composite score: Amount and direction of bias introduced into effect estimates by individual eligibility criteria in the Target Cities sample (white) and in the Washington State sample (black)

The unmotivated/noncompliance criterion also produced upward bias in both samples: 45.3% in the Target Cities data and 5.8% in the Washington State sample. The drug-eligibility criteria had contrasting biases in each sample, causing pronounced downward bias (40.5%) in outcome estimates in the Target Cities sample versus a moderate (7.2%) upward bias in the Washington State sample. The medical eligibility criteria somewhat lowered effectiveness estimates in both samples (by 4.5% in the Target Cities sample and 10.1% in the Washington State sample). Finally, the psychiatric eligibility criteria had no effect on the Target Cities data but resulted in downwardly biased outcomes in the Washington State sample by 8.7%.

In terms of statistical power, the size of the CI around the treatment effect estimate was .035 in the Target Cities sample and .057 in the Washington State sample. Contrary to expectation, every eligibility criteria produced larger, rather than smaller, CIs in both samples.

We then examined the impact of the most common combinations of two and three eligibility criteria (Table 3 and Figure 2). The simultaneous application of the psychiatric and unmotivated/noncompliance criteria produced 56.7% upward bias in outcomes in the Target Cities data. In the Washington State data, the combination of criteria had no effect on outcomes, suggesting that the modest negative bias and modest positive bias, introduced individually, were effectively traded off to yield an unbiased estimate.

Figure 2.

Effect of eligibility criteria on change in Addiction Severity Index alcohol composite score: Amount and direction of bias introduced into effect estimates by multiple eligibility criteria in the Target Cities sample (white) and in the Washington State sample (black)

The simultaneous application of the psychiatric and medical criteria downwardly biased outcome estimates in both samples, with the effect being smaller in the Target Cities sample (3.1% worse outcomes) than in the Washington State sample (17.4% worse outcomes). The simultaneous application of the unmotivated/noncompliance and medical criteria generated substantial positive bias (a 43.4% increase) in the Target Cities outcome estimates and a small negative bias (a5.8% decrease) in the Washington State outcome estimate.

Combining the psychiatric, unmotivated/noncompliance, and medical criteria caused substantial positive bias of the effect estimate (a 51% increase) in the Target Cities data and a negative bias of 7% in the Washington State data. Overall, in no case did the combinations of criteria yield dramatic surprises. Therefore, their effect could be roughly, if not perfectly, estimated simply by adding the results of individual criteria.

The statistical power results for the combination of criteria paralleled those for each individual criterion. For every set of criteria, the CI was larger for selected samples in both the Target Cities sample (size of CI: .035 in full sample vs .049-.063 for selected samples) and in the Washington State sample (size of CI: .056 in full sample vs .058-.104 in selected samples).

Finally, we examined sensitivity of the results to different operationalizations of the criteria and to using outcomes other than improvement in ASI alcohol composite scores. When we repeated all analyses using the more exclusive forms of the criteria (i.e., those that made it more difficult for patients to qualify for the study), the magnitude of the biases observed in the primary analyses was almost uniformly amplified. We also repeated all analyses using ASI items on improvement in days of alcohol intoxication and improvement in days of alcohol-related problems as outcomes. The findings of these analyses mirrored the primary analyses (all sensitivity analyses results are available from the authors).

Discussion

Of the five widely used alcohol-treatment research eligibility criteria examined here in two large samples of real-world treatment seekers, two had moderate effects (psychiatric and medical criteria), one introduced large upward bias in one sample and modest upward bias in the other sample (social-residential stability), and two had little biasing effect in one sample but produced extremely large biases (unmotivated/noncompliance being positive bias, drug eligibility criterion being negative bias). Again, by bias, we refer specifically to the difference between the outcomes of research samples recruited from alcohol-treatment programs versus the outcomes of all patients in those programs. The results do not speak to the difference between research results and the outcomes of all people with alcohol problems, because even real-world treatment data sets can reflect the outcomes of only the small and unrepresentative subsample of that population.

What are the scientific implications of the fact that eligibility criteria can introduce large and not easily predictable bias in alcohol-treatment outcome results? Eligibility criteria could produce extremely deceptive results in any given single group evaluation. For example, if the true effect of a treatment being evaluated were a 35% improvement in alcohol problems, application of low motivation/noncompliance exclusion criteria could boost this rate to a much more impressive 50%, whereas application of a drug-problem exclusion criteria could drop it to a disappointing 20%. However, if the study had a comparison condition, conclusions about the relative effectiveness of treatment should still be valid, because the bias introduced by eligibility criteria would (barring interaction effects) be constant across the treatment and control condition.

Matters become more complex when looking across studies that use different eligibility criteria. Even if a research synthesis restricts focus to studies that have comparison groups, unknown variation can be introduced by differences in eligibility criteria (e.g., Does treatment X produce 10% better outcomes than treatment Y across the literature because it is better or because researchers who study treatment X tend to exclude homeless patients from research, whereas those who study treatment Y do not?). This problem is amplified by the facts that eligibility criteria are often poorly described and that the abstract and discussion sections of treatment outcome studies almost never qualify their conclusions with specific reference to the eligibility criteria the study used (Humphreys et al., 2005).

Because some scientists associate eligibility criteria with randomized trials, we hasten to point out that our results have no bearing on the question of whether randomized clinical trials per se are more or less generalizable than other study designs. Many observational studies have extensive eligibility criteria (Humphreys et al., 2005), whereas some of the largest clinical trials in medicine have enrolled virtually every single patient (Peto et al., 1993). Randomized trialsare essential to progress in medical research, and it would be inappropriate to consider the present results as casting any doubts on their value.

Turning from questions of generalizability to those of power, we were surprised to find that the CIs around treatment effects in the selected samples were consistently larger, rather than smaller, than those in the full treatment samples. This may be the result of the reduction in sample size. The more worrisome interpretation is that the most prevalent eligibility criteria in the alcohol-treatment research field produce increased, rather than decreased, sample heterogeneity. A weakness of our data is that it provides no way to test which of these explanations is correct here.

Our results on statistical power and on bias document some apparent costs of eligibility criteria that have not, to our knowledge, been given attention in the alcohol-treatment literature. Yet, we remain convinced that eligibility criteria are essential to the treatment research enterprise. Most obviously, some treatments have the potential to harm some patients, and eligibility criteria are the best mechanism to prevent such unethical instances in clinical research. Therefore, we make some recommendations to maximize the benefits of eligibility criteria while minimizing their problems.

Foremost, researchers should acknowledge that eligibility criteria have some disadvantages, which implies that they should be used only with good justification. In a recent review of randomized clinical trials in influential medical journals, Van Spall and colleagues (2007) found that slightly less than half of all criteria used in trials were strongly justified by the research team. Lack of such justification suggests that researchers hold the value of eligibility criteria to be self-evident. Eligibility criteria have their place, but this recognition should be balanced by the recognition that they can introduce bias in outcome estimates and can reduce the demographic representation of samples (Humphreys and Weisner, 2000; Humphreys et al., 2007) in alcohol-treatment research.

Second, eligibility criteria should be clearly defined and be no more exclusive than ethical and feasibility concerns demand. As previously mentioned, whether an eligibility criterion affects outcome results depends on two variables: (1) the proportion of patients excluded and (2) the characteristics of those patients. In a previous study, we found that neurological criteria (e.g., excluding patients with organic brain syndrome) exclude about 1% of patients across a range of alcohol-treatment systems. They, therefore, pose little threat to validity regardless of how different the outcomes of such patients are than the included patients. This general principle could be applied more frequently to those criteria that tend to introduce substantial bias into outcome results. Simply stated, our sensitivity analysis showed that the less restrictive the criteria are, the better the chance is that outcome estimates will be unbiased. Many studies apply sweeping exclusion criteria, such as “any drug problems” or “any emotional problems that might affect treatment” (Humphreys et al., 2005). If such criteria were refined, for example, by saying “a current DSM-IV diagnosis of heroin or cocaine dependence” (American Psychiatric Association, 1994) in the former case or “active hallucinations or suicidal intent” in the latter case, outcome results would be more generalizable and more easily replicable.

Third, we would argue that a particularly high burden of proof should be met before alcohol-treatment outcome studies exclude patients who also use illegal drugs. This study demonstrated that drug-related criteria can produce very large bias in outcome estimates, and our prior study showed that such criteria have the further serious disadvantage of tending to decrease the proportion of women and black persons who are eligible for alcohol-treatment research participation (Humphreys and Weisner, 2000; Humphreys et al., 2007). It is worth noting as well that no more than 20% of Americans seeking treatment for a substance-use disorder use only alcohol (Substance Abuse and Mental Health Services Administration, 2006). Focusing only on such patients in treatment studies reduces the applicability of treatment research for the other 80% of patients.

Evidence that a widely used research practice might have some disadvantages is not likely to generate unbridled enthusiasm in the field. In documenting the potential bias introduced by eligibility criteria, we are not attempting to discourage alcohol-treatment research as an enterprise. Rather, we hope that the findings here will aid treatment researchers in determining what design features are likely to give the most unbiased, most highly powered estimate of treatment effect and what tradeoffs are involved in recruitment strategies that enroll relatively higher or lower percentages of patients into clinical research.