Generalizability of Clinical Trials for Cannabis Dependence to Community Samples

Abstract

There is growing concern that results of tightly controlled clinical trials may not generalize to broader community samples. To assess the proportion of community-dwelling adults with cannabis dependence who would have been eligible for a typical cannabis dependence treatment study, we applied a standard set of eligibility criteria commonly used in cannabis outcome studies to a large (N=43,093) representative US adult sample interviewed face-to-face, the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC). Approximately eighty percent (80%) of the community sample of adults with a diagnosis of cannabis dependence (n=133) would be excluded from participating in clinical trials by one or more of the common eligibility criteria. Individual study criteria excluded from 0% to 41.0% of the community sample. Legal problems, other illicit drug use disorders, and current use of fewer than 5 joints/week excluded the largest percentage of individuals. These results extend to cannabis dependence concerns that typical clinical trials likely exclude most community dwelling adults with the disorder. The results also support the notion that clinical trials tend to recruit highly selective samples, rather than adults who are representative of typical patients. Clinical trials should carefully evaluate the effects of eligibility criteria on the generalizability of their results. Even in efficacy trials, stringent exclusionary criteria could limit the representativeness of study results.

1. Introduction

Concerns have emerged as to whether results from tightly controlled trials, generalize to patients commonly seen in community settings (Blanco et al., 2008a; Braslow et al., 2005; Humphreys and Weisner, 2000; Institute of Medicine, 1998; Spall et al., 2007). It has been suggested that some exclusion criteria in clinical trials are overly restrictive, provide little additional patient safety or internal validity (Humphreys et al., 2005; Humphreys and Weisner, 2000; Spall et al., 2007), and severely limit the generalizability of study results. The National Institute on Drug Abuse has consistently stressed the need to increase the generalizability of clinical trials (National Institute on Drug Abuse, 2009a, b, c).

Cannabis use disorders which include cannabis abuse and dependence are the most common drug use disorders in the US, and their prevalence has been growing over the last decade (Compton et al., 2004). Clinical trials of interventions to treat these disorders usually include several exclusion criteria which may yield highly selective study samples (Denis et al., 2006). Furthermore, few clinical trials on cannabis dependence are available (McRae et al., 2003). Consequently, the generalizability of results from cannabis dependence studies has not received much attention, but remains important in interpreting the results of these studies as well as studies that will be published in the future.

It is not known whether samples of clinical trials for cannabis dependence represent adults with these disorders in the community. The goal of this study was to assess to what extent to which eligibility criteria commonly used in cannabis dependence clinical trials would likely exclude adults with cannabis dependence from the general population. To examine the extent to which general population would be represented in cannabis dependence treatment studies, we apply commonly used clinical trial eligibility criteria to all individuals with a current diagnosis of cannabis dependence in a large national sample. By applying this method to the population of individuals with cannabis dependence regardless of their current treatment status, we seek to assess the possible effects on a large proportion of individuals with the disorder.

2. Methods

2.1. Source of data

Data were drawn from the National Epidemiological Survey on Alcohol and Related Conditions (NESARC). The 2001–2002 NESARC is a nationally representative sample of the adult population of the United States conducted by the National Institute on Alcoholism and Alcohol Abuse (NIAAA) that has been described in detail elsewhere (Grant et al., 2003a; Grant et al., 2003b; Grant et al., 2004a; Grant et al., 2004b). The target population was the civilian noninstitutionalized population, 18 years and older, residing in households and group quarters in the United States. Face-to-face interviews were conducted with 43,093 respondents. The survey response rate was 81%. Blacks, Hispanics, and young adults (ages 18–24 years) were oversampled with data adjusted for oversampling and nonresponse. The weighted data were then adjusted to represent the US civilian population based on the 2000 census. DSM-IV cannabis dependence was assessed with the NIAAA Alcohol Use Disorder and Associated Disabilities Interview Schedule—DSM-IV Version (AUDADIS-IV) (Grant and Hasin, 2001), a fully structured diagnostic interview for non-clinician interviewers. The high reliability and validity of the AUDADIS cannabis dependence diagnosis (κ = 0.70–0.94) has been demonstrated in numerous clinical and general population studies in the U.S. and abroad (Canino et al., 1999; Chatterji et al., 1997; Cottler et al., 1997; Grant, 1995; Hasin et al., 1997; Ustün et al., 1997). Clinical reappraisal studies showed good concordance between DSM-IV AUDADIS diagnoses of cannabis use disorders including cannabis dependence and those of psychiatrists (Canino et al., 1999; Cottler et al., 1997).

2.2. Study Selection

To locate studies, we conducted electronic database literature searches, examined the reference sections of research reports, reviews, book chapters, and consulted authors working in this area. We searched the Cochrane Database and found two reviews, one on psychotherapeutic interventions for cannabis use and dependence in outpatient settings (Denis et al., 2006) and another on cannabis and schizophrenia (Rathbone et al., 2008). We searched the Pubmed database using the following phrase: [canna* OR marijuana* OR marihuana*] AND [dependence] AND [“treatment” OR “intervention”]. Inclusion criteria focused on studies that: 1) used an intervention for cannabis dependence; 2) enrolled participants 18 years or older; and 3) had DSM-IV or ICD-10 definitions of cannabis dependence. Two authors (MO and SSK) independently screened the titles and abstracts of all publications obtained by the search strategy. All potentially eligible studies were assessed independently for inclusion by two authors with disagreements adjudicated by the senior author (CB). We considered all social, psychotherapeutic, and pharmacological interventions for cannabis dependence regardless of model, setting, duration of intervention or country where the study was conducted. We then summarized the most commonly used exclusion criteria reported in these studies (Table 1).

Table 1.

Estimated percentage of adults with current cannabis dependence (past 12 months) in the NESARC excluded from typical clinical trials of treatments for cannabis dependence by traditional eligibility criteria

Efficacy eligibility criteria (past 12 months)	Current (past 12 months) Cannabis Dependence (N=133)
	%	95% CI
Suicidality	15.98	(4.96 – 40.96)
Psychotic Disorder	2.59	(0.26 – 21.4)
Concurrent Treatment (all “professional” treatment)	14.81	(4.80 – 37.48)
Medical Conditions or pregnancy (HTN, Heart disease, Liver disease, pregnancy)	16.64	(4.84 – 43.95)
Using fewer than 5 joints/week in the previous month	37.44	(17.92 – 62.12)
Illicit drug abuse or dependence (except nicotine and caffeine)	38.22	(19.68 – 60.97)
Inability to provide contact (unemployed and unmarried and NOT student)	2.03	(0.20 – 17.73)
Lack of reliable transportation, excessive commuting distance	2.67	(0.16 – 31.97)
Lack of English literacy	0.00	(0.00 – 0.00)
Lack of sufficient education/literacy (less than 6th grade)	0.00	(0.00 – 0.00)
Legal Problems (incarceration, legal problems interfering with treatment)	40.96	(22.92 – 61.80)
Referred/mandated treatment		NA
Overall percentage	80.03	(55.77 – 92.72)

We note that research groups in the US and outside of the US have published treatment studies on individuals with problematic cannabis use (Copeland et al., 2001; Martin and Copeland, 2008; McCambridge et al., 2008). These studies were not considered in our study as their samples included individuals under the age of 18 (Dennis et al., 2004; McCambridge et al., 2008), or did not use DSM-IV diagnoses of cannabis dependence (Copeland et al., 2001). However, most exclusion criteria for these studies resembled those of treatment studies we considered (e.g. literacy, heavy alcohol use and other illicit drug use) (Copeland et al., 2001; Martin and Copeland, 2008).

Although focused on cannabis use and other substance use disorders, other studies have included individuals with schizophrenia, psychotic disorders, and depression (Bonsack et al., 2007; Edwards et al., 2006; Findling et al., 2009; Kay-Lambkin et al., 2009; Rathbone et al., 2008). This important and developing line of research was not considered because the trials did not assess cannabis dependence, focused on other substance use disorders, or included individuals under age 18 year Findling et al., 2009; Kay-Lambkin et al., 2009), though some used exclusion criteria included in our study (e.g. lack of literacy).

2.3 Clinical Trials Exclusion Criteria

Some eligibility criteria for cannabis outcome studies parallel those used in alcohol outcome studies and thus were operationalized following previous conventions (Blanco et al., 2008b). These criteria include concurrent treatment, medical conditions, illicit drug abuse or dependence on other drugs, social instability, lack of reliable transportation, insufficient education/literacy, and legal problems previously described in other areas of substance use such as having legal proceedings or having felony assault history (Humphreys et al., 2005). Concurrent treatment included treatment in an alcohol or drug detoxification rehabilitation program, inpatient ward of psychiatric or general hospital, outpatient clinic, drug or alcohol rehabilitation program, methadone maintenance program, or treatment by a mental health professional during the past 12-months. Medical conditions often excluded from trials included hypertension, heart and liver disease, and pregnancy. Illicit drug abuse or dependence to other drugs was assessed with the modules on substance use disorders (SUD).

To help facilitate tracking participants for follow-up assessment, trials often exclude socially unstable patients. Respondents were considered socially unstable if they were unemployed, not in school, and unmarried at the time of the survey. Subjects were considered to lack reliable transportation if, when asked about reasons for not seeking treatment, they stated that they did not have any way to get clinical research site. Subjects were also often considered to have insufficient English literacy if, when asked about reasons for not seeking treatment, they stated that they did not go because of concerns over English literacy. Subjects were also classified as having an insufficient level of education if they had not completed at least the sixth grade. Legal problems were operationalized as having been arrested, having been held at a police station, or having had any other legal problems because of their drinking, medicine or drug use in the past 12 months.

Subjects were classified as suicidal if they reported having suicidal ideation in the previous 12 months. Several cannabis dependence treatment studies also excluded persons who were assessed as having “severe psychiatric distress.” Since some of the studies define this criterion as having a psychotic disorder, subjects were classified as ineligible if they had been told by a mental health professional or by any physician that they had schizophrenia or other psychotic disorder. An alternative operationalization of “severe psychiatric distress” included meeting criteria for bipolar I, II, or major depressive disorder during the past 12 months. Using at fewer than 5 joints per week in the previous month was assessed with questions on the frequency of use available in the NESARC. Since most of the survey respondents reported using 1 joint per occasion, the criterion was operationalized as reporting using cannabis less than nearly every day in the previous month. Information on whether the respondents were referred or mandated to treatment was not available in the NESARC and could not be operationalized.

To guard against the possibility of variations in the results due to differences in definitions, we tested different algorithms for defining the eligibility criteria. For instance, we also conducted the analyses including 12-step program attendance in “concurrent treatment” and applying “using less than 40 days in the last 90 days” instead of at least 5 joints/week previous month, and “dependence on other drugs or alcohol” instead of “illicit drug abuse or dependence” as described in some large cannabis outcome studies (Stephens et al., 2002; The Marijuana Treatment Project Research Group, 2004).

We also conducted supplementary analyses on the probability of meeting an additional exclusion criterion by individual criteria. We further determined the largest percentages of individuals that were excluded by each particular criterion among those meeting any of the other criteria. Full results of the additional analyses are available on request.

2.4. Analysis plan

Following previously developed methods (Blanco et al., 2008a; Blanco et al., 2008b) we first determined the percent with 95% confidence intervals [95% CI] of survey respondents who would be excluded by individually applying each eligibility criterion. Because individuals might have been excluded by more than one criterion, we also calculated the overall percentage of subjects that would have been excluded by the simultaneous application of all of the measurable criteria. We conducted these analyses for all individuals with a current (past 12 months) DSM-IV diagnosis of cannabis dependence. We used the SUDAAN statistical software package (Research Triangle Institute, 2004) to accommodate the sampling design and weights of the NESARC to calculate percentages and corresponding 95% CI.

3. Results

Of the 133 subjects who met DSM-IV criteria for cannabis dependence, 80.0% would have been excluded by at least one clinical trial criterion (Table 1). The percentage of respondents excluded due to the application of an individual criterion ranged from 0% (English literacy or less than sixth grade education) to 41.0% (legal problems other than illicit drug use). Legal problems other than illicit drug use, other illicit drug use disorders, and using fewer than 5 joints per week in the previous month were the 3 criteria that excluded the highest percentage of individuals. Concurrent treatment excluded approximately 15% of individuals. Medical conditions also excluded a substantial proportion of individuals. When the criteria of “severe psychiatric distress” was broadened to include bipolar I and II, major depressive disorder and psychotic disorder, exclusion due to psychiatric comorbidity increased from 80.0% to 84.6% of subjects.

Application of different operationalizations revealed similar percentages of exclusions. For example, when the criterion “receiving therapy or in a 12-step program” was used instead of “concurrent treatment,” the percentage of excluded individuals increased from 14.8% to 18.1%. Using “dependence on other drugs or alcohol” instead of “illicit drug abuse or dependence” resulted in the exclusion of 57.8% instead of 38.2%.

Most of the respondents met more than one exclusion criterion. While there was substantial overlap between some criteria, others did not overlap. For instance, among individuals with lack of reliable transportation 92.7% reported using fewer than 5 joints per week and among psychotic individuals 77% were suicidal. By contrast, none of the individuals receiving concurrent treatment were unable to provide contact information (supplementary material).

4. Discussion

Consistent with findings in other areas of psychiatric research (Blanco et al., 2008a; Humphreys et al., 2007; Humphreys and Weisner, 2000; Posternak et al., 2002; Zimmerman et al., 2004), a majority of individuals with cannabis dependence in the community would be excluded from participating in clinical treatment trials for their disorder. By using a community, rather than a treatment-seeking sample, our approach enables us to assess the population representation in clinical trials for cannabis dependence and to estimate the impact of alternate sets of eligibility criteria on the percentage of cannabis dependent adults in the community who would be excluded.

Three criteria excluded the greatest percentage of individuals: legal problems other than illicit drug use, other illicit drug use disorders, and using fewer than 5 joints per week in the previous three months. Studies that use “legal problems” as an exclusion criterion (The Marijuana Treatment Project Research Group, 2004) have generally described this exclusion as a status that might have interfered with treatment or affected treatment outcome (e.g., mandated treatment, pending jail sentencing) and that helped to exclude difficult to follow patients. Some studies in other areas of substance use research have explicitly excluded patients with “a felony assault history” (Humphreys et al., 2005) whereas others apply a less stringent criterion (Humphreys et al., 2007; Humphreys et al., 2005; Project MATCH Research Group, 1997). Dissatisfaction with guarantees about the confidentiality of the data may restrain some individuals with legal problems from study participation. This exclusion is particularly notable in terms of generalizabilty of clinical trial results given that in most countries criminal justice represents one of the largest sources of referrals to treatment systems for substance use disorders (Australian Institute of Health and Welfare 2009, October 2009; European Monitoring Centre for Drugs and Drug Addiction, November 2004; Substance Abuse and Mental Health Services Administration Office of Applied Studies, August 13, 2009; United Nations Office on Drugs and Crime, 2006).

Having an additional illicit drug use disorder resulted in the exclusion of a substantial percentage of individuals. Previous clinical and epidemiologic studies have consistently reported high rates of comorbidity among individuals with substance use disorders (Agosti et al., 2002; Stinson et al., 2005). Exclusion of individuals with other substance use disorders may result in the exclusion of the most common presentations of the disorder.

Using fewer than 5 joints per week also excluded a large proportion of individuals with cannabis dependence. Substance use frequency as an exclusion criterion may rely on the assumption that dependence is unlikely in drug use that does not occur on a daily basis, although patients in clinical trials for cocaine dependence may present with patterns of binge use as well as regular use (McRae et al., 2007). Although the risk of cannabis dependence increases markedly with frequency of use (Copeland and Swift, 2009), previous epidemiological surveys have found that using fewer than 5 joints/week is not uncommon among individuals with cannabis dependence (Grant and Pickering, 1998). The daily use criterion may derive from substance use frequency as an outcome variable in substance use research (Babor et al., 1994; Carroll, 1995; McKay et al., 2001).

Although restrictive eligibility criteria may be appropriate in early efficacy trials, greater attention in later stage trials should be given to justifying trade-offs between the application of each exclusion criterion and its impact on sample representation and therefore the broader generalizability of the results. Some exclusion criteria are necessary to avoid the use of interventions that would likely be ineffective in certain populations or to ensure patient safety, such as pregnant and lactating women. At the same time, stringent eligibility criteria may lead to the exclusion of populations with characteristics associated with poor outcome. This in turn may lead to an overestimation of the efficacy of the intervention under study. For instance, because psychiatric comorbidity is associated with premature treatment dropout and worse outcomes in individuals with substance use disorders (Compton et al., 2003; McLellan et al., 1983), systematically excluding a wide range of common comorbidities may lead to lower dropout from clinical trials that what would be observed with more representative patient populations. A recent review of eligibility criteria of randomized controlled trials suggested that justification for exclusions related to comorbid medical conditions and medicine use are often poorly justified (Spall et al., 2007).

Treatment-seeking populations tend to have longer courses of the disorder and higher rates of comorbidity (Brooner et al., 1997; Cohen and Cohen, 1984). Higher perceived or assessed need or other legal imperatives may powerfully influence treatment-seeking behaviors (Kirchner et al., 2000; Weisner and Matzger, 2002; Wu and Ringwalt, 2004). Clinical trials may tend to exclude those with the greatest overall disease burden and therefore greatest treatment need. Furthermore, the present study found that some criteria were highly associated with the presence of other criteria suggesting that the intentional exclusion of individuals with certain characteristics may result in the unintended exclusion of other groups with cannabis dependence. Future studies should empirically examine the impact on outcome of applying specific exclusion criteria, as was recently done in a large effectiveness trial for adult major depression (Wisniewski et al., 2009). This concern has also been a subject of considerable interest in substance use research (Carroll et al., 1999; Humphreys et al., 2005; Velasquez et al., 2000).

The current study has several limitations. First, we adopted specific conventions to translate clinical criteria to the NESARC sample. Different conventions may have yielded different exclusion estimates. However, the percentage of excluded subjects was high across all criteria specifications. Because applied research criteria vary across studies, our criteria specifications may not represent their application in all cannabis dependence outcome studies. However, by applying criteria that closely represent those used in several studies, our results likely offer a reasonable estimate of the most commonly used criteria. Furthermore, the percentage of excluded subjects was high across several specifications of the criteria. Second, not all trials used all of the operationalized criteria in this study, and thus the overall percentage of excluded individuals in our study may not necessarily represent all the individuals excluded from each individual trial. At the same time, some of the exclusion criteria could not be operationalized. Thus, it is also possible that the actual proportion of community dwelling adults with cannabis dependence who would be actually excluded from clinical trials is higher than our estimates. Third, despite the large overall sample size of the NESARC, the proportion of individuals meeting criteria for current cannabis dependence was relatively small resulting in fairly wide confidence intervals. Nonetheless, the lower bounds of the confidence intervals suggest that at least one-half of individuals with cannabis dependence would have been excluded by the criteria. Finally, it is also possible that different results could have been obtained if individuals in treatment had comprised our sample. Given that treatment-seeking might be a function not only of patient characteristics, but also of insurance and other factors which enable access, our approach puts forward findings that might extend across different systems of care.

Traditional efficacy clinical trials involve highly selected patient samples. Yet substance dependence complicated by other problems or illnesses is the rule, rather than the exception. Epidemiological samples can help estimate the prospective representation of samples that participate in clinical trials. Two courses of action should be considered to better understand the effects of treatments for most persons with cannabis dependence. First, eligibility criteria for clinical trials should be more broadly inclusive to increase the resemblance of sociodemographic and clinical characteristics with affected community samples. This will provide greater assurance of generalizability of results of clinical trials of cannabis dependence. Excluding large proportions of target populations results in selection of highly restricted samples, and a focus on “pure” rather than “typical” patients with the disorder. Second, the impact of common exclusion criteria on treatment outcome should to be investigated in a systematic manner. This can only be done if a broader range of adults with these characteristics are included in clinical trials.