Determining the Primary Endpoint for a Stimulant Abuse Trial: Lessons Learned from STRIDE (CTN 0037)

Abstract

Background

No consensus is available for identifying the best primary outcome for substance abuse trials. While abstinence is the most desirable outcome for substance use interventions, a wide variety of other endpoints have been used to evaluate efficacy trials.

Objectives

This report provides a framework for determining an optimal primary endpoint and the relevant measurement approach for substance use disorder treatment trials. The framework was developed based on a trial for stimulant abuse using exercise as an augmentation treatment, delivered within the NIDA Clinical Trials Network. The use of a common primary endpoint across trials will facilitate comparisons of treatment efficacy.

Methods

Primary endpoint options in existing substance abuse studies were evaluated. This evaluation included surveys of the literature for endpoints and measurement approaches, followed by assessment of endpoint choices against study design issues, population characteristics, tests of sensitivity and tests of clinical meaningfulness.

Conclusion

We concluded that the best current choice for a primary endpoint is percent days abstinent, as measured by the Time Line Follow Back (TLFB) interview conducted three times a week with recall aided by a take-home Substance Use Diary. To further improve the accuracy of the self-reported drug use, an algorithm will be applied to reconcile the results from the TLFB with the results of qualitative urine drug screens.

Scientific Significance

There is a need for a standardized endpoint in this field to allow for comparison across treatment studies, and we suggest that the recommended endpoint be considered for use in this field.

Introduction

Stimulant abuse is a chronic, relapsing illness with few highly efficacious treatments. Research in the treatment of substance use disorders (SUDs) suggests a need for innovative treatments as a substantial proportion of individuals engaged in currently available treatments do not achieve abstinence (1). Research devoted to treating individuals with stimulant abuse and dependence has relied on a variety of endpoints and it is challenging to compare trial outcomes across treatment studies. Long-term continuous abstinence is the preferred clinical goal for individuals being treated for stimulant use disorders. However, the lack of efficacious treatments can make long-term continuous abstinence an unrealistic goal for research studies evaluating treatments (2–5). Furthermore, inherent weaknesses of both self-report and biological outcome measures make the evaluation of long-term continuous abstinence difficult (6, 7). Therefore, long-term continuous abstinence has most often not been seen as the best endpoint to select in the context of a clinical trial. Nonetheless, an endpoint serving as a proximate outcome must be selected that is consistent with the modest efficacy of currently available treatments.

Research in this field has typically chosen endpoints that are clinically believed to be meaningful proxy measures or intermediate endpoints toward the goal of long-term continuous abstinence. These endpoints have included a reduction in use, increases in days/weeks of abstinence, total amount of money spent, or continuous abstinence over specified periods of time (8). Likewise, specific measurement approaches to define study endpoints also vary widely, ranging from self-reported rates of abstinence to detailed toxicology data that quantify exact amount of drug use. Occasionally functional measures have been used, as well as measures evaluating the impact of substance use on society. Currently, there is no gold standard primary endpoint or outcome measure in the field for trials of stimulant abuse treatment. Investigators who are developing trials to assess substance use outcomes must consider several important factors when deciding upon a primary endpoint and relevant outcome measure (9).

This report describes the framework used and issues involved in selecting a primary endpoint and primary outcome measure for the CTN 0037 trial, Stimulant Reduction Intervention using Dosed Exercise (STRIDE), a trial to be conducted through the Clinical Trials Network of the National Institute of Drug Abuse. Hughes et al. (10) previously reported the approach they used to help move the field toward a consensus approach to the study of smoking cessation treatments. This report aims to provide a similar contribution for the field of stimulant use. We present the model we used as well as steps undertaken to select our primary endpoint and outcome measure. Lessons learned through the completion of these steps for STRIDE are also reported.

Procedure to Determine an Optimal Endpoint for a Study

The choice of a primary endpoint is driven by the intent of the study, options in the existing literature, and consistency of the choice with features of the study design (9, 11, 12). The first step in selecting a primary study endpoint is to conduct a review of the literature to determine the available endpoints used within the specific population being studied, as well as the endpoints specifically relevant to the intervention. Once the candidate endpoints are determined, the next step is to select the endpoint that best fits the intent of the study. Finally, it must be determined whether the selected endpoint is clinically meaningful. While these procedures are described as if they occur in a linear fashion, this is an iterative process with ongoing reevaluation of critical factors and decisions by experts in the interventions and disorders of interest. Figure 1 presents our process for deciding upon a primary endpoint for the STRIDE trial.

Figure 1. Flow diagram of the steps needed to decide on a primary endpoint for a clinical trial.

For the STRIDE study, the intent of the study is to assess whether or not augmentation of treatment as usual with exercise or health education will improve stimulant use outcomes. Investigators, with the support and guidance of a Protocol Development Team (PDT), reviewed salient literature on treatments for substance use in general, as well as studies that used our specific interventions or that were conducted within our specific population of interest. The team decided upon a primary endpoint of percent days abstinent, as measured by the Time Line Follow Back informed by a daily Substance Use Calendar and urine drug screens collected 3 times per week. The feasibility and applicability of using the TLFB to assess percent days abstinent in a trial of long duration (i.e., 9 months) and in a setting that requires transitioning from residential to outpatient treatment was considered. Consensus was obtained by the PDT regarding the clinical meaningfulness of an endpoint of percent days abstinent. Note that if at any step a problem is encountered (e.g., power analysis reveals that the number of subjects needed to conduct the trial is not feasible, or the effect associated with the selected endpoint is deemed not clinically meaningful), one can return to the previous step(s) to reevaluate the information and select the new endpoint.

Identify the Intent of the Study

STRIDE is a randomized, controlled trial that aims to assess the addition of vigorous-intensity, high-dose exercise or health education to treatment as usual (TAU; i.e., usual care) in improving stimulant use disorder treatment outcomes. Individuals diagnosed with stimulant abuse or dependence (cocaine, methamphetamine, amphetamine, or other stimulant, except caffeine or nicotine) will receive 3 months of acute phase supervised intervention followed by an additional 6 months of supervised intervention once per week. Both groups will receive stimulant abuse TAU, which will begin in a residential setting, followed by community treatment.

A Protocol Development Team was assembled for the trial that was comprised of experts in stimulant abuse treatment and research, exercise and behavioral interventions, and multisite clinical trial research (Appendix 1). In addition to the overall development of the study design and protocol, this team was tasked with identifying the best study endpoint to evaluate whether or not exercise or health education enhances usual care for stimulant use disorders.

There are a number of important outcomes appropriate to consider when investigating SUD treatments (e.g., psychosocial, health-related outcomes) (13–18). As this is an early efficacy trial, the study primary outcome focuses on the more immediate substance use outcomes rather than other, often longer term problem related outcomes with possibly lower effect sizes. Problem related outcomes will also be collected, however.

Review of the Literature for Candidate Endpoints in Use

We reviewed the existing literature to determine if a standard endpoint was available and suitable for this population (stimulant abusers), setting (residential treatment followed by outpatient care) and the interventions being studied (exercise and health education). Given that STRIDE is a study of a novel treatment approach for stimulant abusers with an innovative study design that follows a participant through a treatment process that includes both residential and outpatient care, no standard endpoint was available.

Because there was no clear choice of endpoint based on this search, we then searched for studies of other behavioral interventions that targeted SUDs in general and those that specifically targeted stimulant abuse, as well as studies examining exercise interventions in other relevant clinical populations (e.g., exercise augmentation for smoking cessation).

Again, endpoints in outcomes research in SUDs are variable, and range from abstinence, to reduction in drug use over time, to relapse prevention (8). A meta-analysis of trials examining psychosocial interventions for SUDs found that abstinence, either self-report, based on urine drug screen, or a combination, was indeed a common endpoint in SUD trials, although several different definitions have been used for abstinence, including mean weeks (or days) of abstinence, mean percent of weeks (or days) abstinent, percent of sample abstinent for a particular duration of time, percent of sample achieving post-treatment abstinence, and post-treatment scores on the Addiction Severity Index drug scale. Time to relapse has also been used, with varying measures of relapse, such as time to first use after abstinence, or time to second or third use (1). Finally, time to treatment discontinuation and treatment retention have also been used; these approaches generally assume that attrition indicates resumed substance use or treatment failure.

Focusing specifically on treatment studies for stimulant users, the variability in endpoints was again apparent. Endpoints used included continuous abstinence (11, 12, 19, 20), mean percent or duration of abstinence (21, 22), treatment retention (2, 12, 23, 24), treatment attendance (12), and frequency and quantity of use (2, 3, 12, 23, 25–27). Toxicology outcomes included mean number of negative (or positive) urine screens throughout treatment and mean percent of negative (or positive) urine screens throughout treatment (7).

Finally, we searched for studies of exercise as a treatment for related clinical conditions, which yielded studies primarily focused on smoking cessation. However, no gold standard for a study endpoint emerged from a review of these trials either. These studies used a variety of study endpoints including continuous abstinence, abstinence occurring over different specific periods of time (e.g., 12 weeks, last month etc.) or at a specific point in time (e.g., 7-day point prevalence abstinence), craving and other outcomes (28–30).

Selection of a Candidate Endpoint

Based on our review of the literature, it was clear there were several endpoints we could consider for STRIDE. Table 1 illustrates the endpoints that were considered by the STRIDE Protocol Development Team, and presents the advantages and disadvantages of each.

Table 1.

Endpoints Considered by the STRIDE Protocol Development Team

Endpoint	Description	Advantages	Disadvantages
Number of Days Abstinent	A measure of frequency of use that can be collected by subjective report or objective measurement.	represents a broad picture of use over any period of time and is sensitive to change in frequency of use	ongoing abstinence is the goal of most treatment programs—this measure does not necessarily show periods of abstinence can also miss severity of use—episodic use of greater severity may have more days of abstinence than more frequent use of less severity if using a self-report, has potential biases and distortions of a self-report there is no consensus regarding what decrease in use is clinically important, although effect sizes can be selected that are consistent with effect sizes in studies of other interventions
Number of Consecutive Weeks Abstinent	A measure of number of continuous weeks in which there is no use. Can be collected by subjective report or objective measurement.	consistent with the goal of ongoing abstinence in substance abuse treatment more stringent outcome than number of days abstinent since continuous abstinence is more difficult to achieve than intermittent days of non-use	if using self-report, has potential biases and distortions of a self- report may be less sensitive to change does not report frequency or severity of use in time of non-abstinence there is no consensus about what decrease in use is clinically important, although effect sizes can be selected that are consistent with effect sizes in studies of other interventions
Rate of Relapse	This measures percent of individuals relapsing at any particular time point. Can be measured by self-report or objective measurement.	identifies the efficacy of a treatment at any time point repeated measures can be used to reflect efficacy over time	this does not measure frequency or amount of use, or timing of abstinence provides only a snapshot of a particular time period-point prevalence
Time to Relapse	Number of days (or weeks) from discontinuation of use to first use. Can be measured by self-report or objective measurement.	this is a measure of continuous abstinence, consistent with an abstinence goal this is a measure typically used by people in recovery	given the relapsing nature of substance abuse, this provides only a snapshot of the initial time period before a first relapse
Reduction in Amount of Drug Use	Change in volume of drug use. Could be based on self-report or objective measure such as quantitative urine drug screen and sweat patch.	useful measure of severity is relevant to a goal of use reduction rather than abstinence	does not focus on abstinence or identify change in frequency of use if using self-report, has potential biases and distortions of a self-report quantitative measures can be difficult to interpret since they are affected by both time and amount of drug usage
Reduction in Amount of Money Spent	Change in dollars spent over an identified time period.	useful measure of severity is relevant to a goal of use reduction rather than abstinence	does not focus on abstinence or identify change in frequency of use if using self-report, has potential for biases and distortions of a self-report

Because it is most representative of long-term continuous abstinence, we first considered consecutive days or weeks of abstinence for our primary endpoint. However, given that this study examines a novel augmentation treatment for stimulant use disorders in a field where augmentation strategies offer only modest incremental improvements (e.g., Carroll et al. (31) reported that the effect size for active psychotherapies compared with a control condition was d=0.16), we decided it was more meaningful to choose an endpoint that would also identify reduction in use. For example, if a participant entered the study after typically using stimulants 3–4 days per week, but reduced usage to once per week or once every other week, consecutive days/weeks of abstinence would not adequately capture that reduction. We therefore decided that percent days of abstinence was the most reasonable primary endpoint for STRIDE.

Selection of a Primary Outcome Measure

After identifying a primary endpoint, we next had to establish our definition of an abstinent day. We again began with a search of relevant literature. The most common outcome measure utilized in treatment studies for stimulant abuse and/or dependent participants was urine drug screens, typically collected 1–3 times per week. However, studies varied widely with respect to the use of self-report or objective measures of drug use, with some trials using only one or the other, and other trials using a combination of the two. Because no consistent outcome measure has been used to assess stimulant use, we assessed the strengths and weaknesses of potential outcome measures. The main options considered by the Protocol Development Team (PDT) are detailed in Table 2.

Table 2.

Outcome Measure Options Considered by the STRIDE Protocol Development Team

Measure	Description	Advantages	Disadvantages	Psychometric Properties
Time Line Follow Back (TLFB) (26)	Semi-structured interview based on calendar to assist in retrospective estimates Records daily use	self-report methods are economical, efficient, relatively noninvasive (30) allow for evaluation of substance use over longer time periods (6) than with biological approaches allows for reporting over an extended period of time, since patients frequently miss appointments when biological samples are taken allows for in-depth look at substance use over time, rather than being limited to a dichotomous variable of use vs. no use commonly used in the field and the CTN and allows for comparison between studies can measure abstinence for days, weeks, or any other timeframe	relies on patient report which can include memory errors and bias, with the potential for these errors to increase over longer recall periods possibility for deliberate and denial-based distortions by the participant (6) may be slightly less valid with longer periods of recall (6)	High test-retest reliability (intraclass correlations range from 0.70 to 0.94) (6) High agreement with urine drug screens (Yule’s Y of 87) (6)
Substance Use Diary	Self-report measure Records daily drug use Can be completed by participant at home	could improve the accuracy of data over a TLFB interview approach since the participant completes the information in real time advantages similar to TLFB: economical, efficient, relatively non-invasive, and providing a broader look at the substance abuse picture, compared to using biological methods	drawbacks similar to other self- report methods: participant memory error, bias, or distortion participant may not completely or accurately fill out the calendar without prompting from an interviewer participant may lose the form or fail to bring it in	Not available#
Addiction Severity Index-Lite (ASI-Lite) (32)	Standardized, semi-structured interview Assesses multiple domains that commonly affect substance abusers (32) Can assess lifetime or current use patterns (30 days) 160 questions	used in many CTN studies, making it easier to compare results across studies gives a view of outcome measures in the multiple domains of functioning that can be affected by substance use, in addition to amount of substance use can be administered by clinicians, researchers, or trained technicians	length of time it takes to complete the assessment at baseline subjective measure of drug use which relies on the participant’s report gathers estimates of substance use over time, but does not allow for specifics of quantity of use clinical interpretation of numerical index (subscale) scores not always clear	High intraclass correlations range from 0.79 to 0.85 for drug use (33)
Urine Drug Screen (UDS)	Biological assay Subject s are required to provide urine samples Measurement frequency is commonly 2-3X per week Most commonly reported as a positive or negative drug screen Note: quantitative analyses can be conducted on urine samples	objective measure for assessing drug use and does not rely on participant’s self-report widely used and accepted method of assessment in treatment studies (9) results can be compared across studies (7) well accepted approach for cocaine assessment and best-studied objective measure for diagnosing cocaine use (34)	short detection time periods, high cost, and personal invasiveness (35) only able to detect drug use within a limited amount of time after the use and can only show if drug use occurred, not how much drug was used or what the pattern of use was like variance in detection sensitivity for different drugs: some drugs are detectable for longer periods of time than others, so it can miss some episodes of use sometimes using a large amount of a substance (e.g., marijuana, cocaine) may cause more than one positive screen, even though there was only one use, referred to as “carryover” (7) can be vulnerable to participant distortion because samples can be switched or participants can alter the sample possibility for missing data if the participant does not show up to provide a sample	Not applicable*
Sweat Patches	Biological assay Based on sweat collected from participant’s skin Good correlation between a weekly patch and patches checked 3X per week (0.69) (34) Good correlation between patches and UDS ( 0.73) (34)	can assess substance use over longer periods of time compared to a UDS, and amount of use can be determined can reduce the variability that occurs because of the time difference between the substance use and a UDS may also allow for the same, or in some cases greater, amount of data provided compared to a UDS, with a less invasive and potentially less expensive measure (34)	may be more expensive than UDS may be irritating for participants to wear may fall off or be removed prematurely by participant, thus losing the data there have not been as many studies completed examining sweat patch use, compared to UDS and other measures extremely reliable for substances such as opiates, but are less reliable for stimulants	Not applicable*

^#Psychometric properties for the Substance Use Diary have not been published.

^*Psychometric properties are not relevant for biological measures.

After considering the potential advantages and disadvantages, we decided upon an approach that integrates the strengths of multiple measures and increases the chances of successful data collection over an extended time period. We chose to assess percent days abstinent during the observation period as measured by self-report using a Time Line Follow Back (TLFB) (32). The TLFB interview is conducted in a nonjudgmental manner by trained research staff, with no adverse consequences for disclosure of use. Participants are informed that their disclosed information is not shared with others, such as clinic staff or treatment providers. This is a critical feature of this process, since individuals are likely to be unwilling to disclose substance use in situations where their report of use would have severe repercussions (e.g., a parolee reporting use to their probation officer). The TLFB has been shown to correlate well with objective measures of use such as urine drug screen (33–35). As noted in Table 2, disadvantages of the TLFB include potential inaccuracy due to memory errors and bias and deliberate and denial-based distortions of reported SUD. To further bolster the accuracy of our assessment, we added two additional tools to the data collection process. First, the participants will use a Substance Use Diary to aid with recall during the TLFB assessment. The Substance Use Diary is a calendar that participants are asked to take home to record use in real time, as well as document other events (e.g., birthdays, travel) that can be helpful to trigger recall during the TLFB interview. The diary is intended to be used in between study visits to help participants maintain an accurate accounting of their usage of stimulants and other drugs. It can be particularly helpful in situations where missed visits occur and greater amounts of time must be recalled. A diary is not expected to provide the same comprehensiveness of information as the structured TLFB interview, but the use of a memory aid such as a diary has been shown to improve the accuracy of the TLFB (36). Finally, to further improve the validity of the TLFB, an algorithm that includes both TLFB self-reported use as well as results from the urine drug screens collected 3 times per week will be employed at study end to reconcile discrepancies between negative TLFB reports and positive urine drug screens. The availability of the primary outcome measure is therefore not affected by an absent urine drug screen.

Design and Population Factors that Influence the Measurement of the Primary Endpoint

An additional consideration in choosing an outcome measure was the applicability of the selected outcome measure for the study design and population. In a traditional experimental design, the research question determines the study endpoint and measure, and the study is then designed around that endpoint and measure. However, applied researchers know that clinical research is highly influenced by the context in which the research will be conducted. Elements of the trial design as well as characteristics of the study population must be considered when selecting an endpoint. These design considerations include the types of outcome measures that are possible (e.g., blind vs. unblind, self-report vs. objective), the treatment setting, and the treatment duration.

Choice of the TLFB self-report measure was consistent with several elements of our study design. Our study intervention will be conducted over a 9-month period during which time participants not only transition from residential to outpatient treatment but are expected to come to the study site 3 times a week for 12 weeks and once a week for 24 more weeks. While an objective measure of stimulant use is optimal, sole reliance on UDSs is not realistic. As stimulant abusing patients are known to be inconsistent in their attendance at treatment and study visits (1) a UDS only endpoint would result in considerable missing data. Furthermore, urine drug screen results are affected by both amount and time of use. Thus, given currently available screening technology, reliance on the UDS as an exclusive outcome can be complicated by this issue as well. The TLFB, on the other hand, allows for data to be collected on all days, since the interviewer prompts the participant to recall use for all days since the last study visit, and accuracy of the information is not impacted by amount and timing of use, as with urine drug screens.

Assessing the Clinical Meaningfulness of the Endpoint

Once the choice of outcome measure was considered with respect to design issues, the next step was to determine what magnitude of change was necessary for a clinically meaningful endpoint. The ability of potential endpoints to be readily translated and adapted for use in clinical practice must be evaluated. Continuous abstinence might be considered the ideal endpoint for individuals with stimulant abuse or dependence, but use of this endpoint has not been realistic given the time limits and expected frequency of attended visits in clinical trials. Thus, while continuous abstinence might be relevant, it is not necessarily the best endpoint for a particular study. Other endpoints may be clinically meaningful and have greater sensitivity and represent better endpoint choices.

Identifying the magnitude of change in a study endpoint that would be clinically meaningful can in part be determined by the literature. Often specific “cut-off” points are established in the literature to determine clinical significance. For example, Higgins et al. (11) considered a treatment to be successful if participants achieved at least 9 weeks of continuous abstinence or 92% or greater days of abstinence. Our choice of endpoint (percent days abstinent) will allow for comparison against the suggested criteria set forth by Higgins et al. Furthermore, Miller and Manuel (37) also identify other endpoints of clinical significance, including quantity of use, length of initial abstinence, use of other illicit substances, and dropout from SUD treatment. Our selection of outcome measure will allow for analysis of these secondary endpoints that may be clinically meaningful.

Another guide to the magnitude of differences that can be considered clinically meaningful are those reported in peer-reviewed literature for that field since peer reviewers must consider whether the results of studies significantly contribute to the field. However, these studies do not necessarily reflect differences that would be meaningful to clinical practitioners. Miller and Manuel (37) note that it is important to understand how clinicians view differences between treatments and whether or not a particular effect would be of interest to them or likely to change their practice. Thus, as always done in the CTN, our PDT included clinical treatment providers who provided valuable input in this regard.

Comment

In the SUD literature, there is no gold standard for a primary endpoint for treatment trials. Although continuous abstinence is the ideal outcome for treatment programs, the state of research in SUD in general and stimulant abuse in particular suggests an endpoint that allows for the modest effects of treatment interventions to be detected. This may be even more important with augmentation interventions that are adding only incremental gains.

We conducted a comprehensive evaluation of the options for primary endpoints in studies of treatments for stimulant abuse, including surveys of the literature for endpoints and measurement approaches, evaluation of the choices against study design issues, population characteristics, and tests of sensitivity and clinical meaningfulness. We have concluded that given the state of the field, the best current choice for a primary endpoint is percent days abstinent, given its ability to detect a clinically meaningful improvement in abstinence, an important clinical goal. We chose the TLFB interview with its extensive available retrospective timeframe, as the most comprehensive measure that could be collected over an extended time period and enhanced its reliability with a take-home Substance Use Diary to aid recall. Furthermore, an algorithm will be used at study end to reconcile the results obtained from the TLFB and the objective urine drug screens. Thus, our selection of primary endpoint and outcome measure: 1) captures use information across our entire acute phase; 2) blends advantages of multiple approaches in that it is informed by collection of real-time data from the Substance Use Diary, as well as objective data from urine drug screens; 3) provides us with a clinically meaningful evaluation of the efficacy of exercise as an augmentation to treatment as usual.

Other fields (i.e., smoking cessation) have used a similar approach to help guide the field toward a consensus model of studying treatment interventions (10). Researchers may want to consider using the endpoint we have identified to improve the field’s ability to compare the efficacy of interventions in trials of stimulant abuse. It will also be important for future studies to consider technological advancements in measurement capabilities, such as the use of electronic diaries and interactive voice response systems, as these will likely have a strong influence on the development of available outcomes in the future. As the field progresses in identifying better treatments for substance use, researchers may also use the framework we have described to determine an optimal primary endpoint for future studies.