Design and Methodological Considerations of an Effectiveness Trial of a Computer-assisted Intervention: An Example from the NIDA Clinical Trials Network

Abstract

Computer-assisted interventions hold the promise of minimizing two problems that are ubiquitous in substance abuse treatment: the lack of ready access to treatment and the challenges to providing empirically-supported treatments. Reviews of research on computer-assisted treatments for mental health and substance abuse report promising findings, but study quality and methodological limitations remain an issue. In addition, relatively few computer-assisted treatments have been tested among illicit substance users. This manuscript describes the methodological considerations of a multi-site effectiveness trial conducted within the National Institute on Drug Abuse's (NIDA's) National Drug Abuse Treatment Clinical Trials Network (CTN). The study is evaluating a web-based version of the Community Reinforcement Approach, in addition to prize-based contingency management, among 500 participants enrolled in 10 outpatient substance abuse treatment programs. Several potential effectiveness trial designs were considered and the rationale for the choice of design in this study is described. The study uses a randomized controlled design (with independent treatment arm allocation), intention-to-treat primary outcome analysis, biological markers for the primary outcome of abstinence, long-term follow-up assessments, precise measurement of intervention dose, and a cost-effectiveness analysis. Input from community providers during protocol development highlighted potential concerns and helped to address issues of practicality and feasibility. Collaboration between providers and investigators supports the utility of infrastructures that enhance research partnerships to facilitate effectiveness trials and dissemination of promising, technologically innovative treatments. Outcomes from this study will further the empirical knowledge base on the effectiveness and cost-effectiveness of computer-assisted treatment in clinical treatment settings.

1.0 Introduction

Computer-assisted interventions are promising in minimizing two problems ubiquitous in substance abuse treatment: lack of treatment access and challenges providing empirically-supported treatments (ESTs). Treatment barriers include geographic proximity, insufficient treatment slots, stigma, and limited treatment options in non-specialty settings [1]. Although a number of psychosocial ESTs exist, patients in community-based, outpatient substance abuse treatment are infrequently provided with such interventions as part of standard care [2,3]. ESTs can be challenging to implement and sustain, and often strain available resources.

Approximately 79% of adult Americans report regular internet use [4], with urban (81%) and rural use (67%) both increasing [5,6]. Computer-assisted interventions allow complex treatments to be delivered with high fidelity and low cost, with minimal demands on staff time or training resources. A computerized program may be less threatening and provide greater anonymity [7,8] while permitting flexibility in when and where to access the intervention. Computerized interventions also allow individuals to review repetitive but critical skills and engage in psychoeducational activities longer than would be possible with a therapist alone [9,10].

1.1 Computer-assisted Substance Abuse Treatment Interventions

Systematic and meta-analytic reviews of computer-assisted treatments for mental health and substance use disorders report promising findings, but methodological weaknesses temper enthusiasm [11–15]. Important questions also remain about the best methods of integrating computer-assisted treatments into standard care. In addition, relatively few computer-assisted treatments have been tested among illicit substance users [11,13,16,17].

Several recent randomized trials have demonstrated efficacy in improving drug use outcomes. Computer-delivered versions of the Community Reinforcement Approach plus voucher incentives [18] and Motivational Interviewing plus cognitive behavioral therapy [19] were comparable to therapist-delivered versions of the same treatments in reducing drug use. Carroll et al. [20] demonstrated that treatment-as-usual (TAU) plus computer-delivered cognitive behavioral therapy was superior to TAU alone in producing significantly more negative urine drug screens among outpatients [21]. Ondersma et al. [22] found a single-session computer-delivered motivational intervention reduced drug use among postpartum women. Rigorous studies that address issues critical to community-based implementation are needed.

1.2 Current Study

The National Institute on Drug Abuse's (NIDA) National Drug Abuse Treatment Clinical Trials Network (CTN) offers a unique platform to test innovative interventions in diverse community settings. To address the potential of technological delivery mechanisms to increase dissemination, the Web-delivery of Evidence-Based, Psychosocial Treatment for Substance Use Disorders (WEB-TX) study was launched by the CTN in 2010. The purpose of WEB-TX is to evaluate the effectiveness of including an efficacious, web-based version of the Community Reinforcement Approach (CRA; [23]) plus contingency management, known as the Therapeutic Education System (TES; [18]), in outpatient substance abuse treatment. TES has prior efficacy data supporting CRA [24,25] combined with abstinent-contingent incentives [26–29] and can target a range of substance use disorders. Study procedures will be completed by summer 2012, with primary outcome analysis and publication to follow.

In designing this multi-site effectiveness trial, the protocol team faced a number of choices pertaining to: 1) the experimental design; 2) specification of treatment and control conditions; 3) site selection; 4) primary outcome measure(s) and analysis; 5) cost effectiveness analysis; and 6) data security. This paper describes the rationale for the design and methods of WEB-TX, maximizing empirical rigor while maintaining applicability to community treatment providers.

2.0 Methods

2.1 Experimental Design Options in a Multi-site Effectiveness Trial

Given prior evidence of TES efficacy and the CTN policy of investigator-clinician collaboration, the primary emphasis of this effectiveness trial was on balancing external and internal validity, while addressing questions of greatest relevance to community-based clinical practice. In keeping with this goal, the protocol team was comprised of treatment researchers and leaders from several community-based treatment programs to help ensure that the design, and ultimately the findings of the trial, would be of direct relevance to advancing clinical practice. In a randomized, controlled effectiveness trial, an experimental intervention of interest is compared to a control condition, which is generally some variation on standard care within community settings. The experimental design addresses a question about the extent to which the new intervention improves outcome in community-based treatment.

Table 1 displays the design chosen for WEB-TX, as well as alternative designs considered. The design typology (i.e., Types 1, 2a, 2b, and 3) shown in the table grew out of an effort to classify effectiveness trials conducted in the CTN [30,31]. The design chosen was consistent with a Type 2b design, which addresses the question of whether TES is effective when substituted for a certain amount of TAU. Participants are randomly assigned to receive either TAU or the intervention condition, in which 2 hours per week of TES replaces comparable TAU that they would have otherwise received at the treatment program. Since community-based substance abuse treatment typically includes a number of clinician-led group therapy sessions, participants in the TES condition are likely to spend two fewer hours per week participating in groups.

Table 1.

Alternative experimental designs considered during the protocol development phase of a randomized clinical trial (WEB-TX) testing the effectiveness of a computer-assisted intervention for treatment of substance use disorders (Therapeutic Education System or TES), compared to treatment-as-usual (TAU) in the setting of community-based substance abuse treatment. The Design Type refers to a classification of effectiveness trials derived from designs implemented in the National Drug Abuse Treatment Clinical Trials Network [30,31]. The shaded row (Reduced TAU + TES vs TAU) was selected for the WEB-TX study.

Design	Type	Primary Question	Rationale
[Reduced TAU + TES] vs [TAU]	2b	What is the effect of substituting 2 hours per week of computer delivered TES, for 2 hours of usual, clinician-delivered treatment?	Tests TES as clinician extender Addresses limited funding in the face of increased demand Provides patients an alternative to group attendance
[TES] vs [TAU]	1	What is the effectiveness of TES as a complete alternative to usual, clinician-delivered treatment?	Useful in settings characterized by limited treatment availability or accessibility Introduces regulatory concerns with little to no face to face clinical contact
[TES + TAU] vs [TAU]	2a	What is the effect of adding TES in addition to usual, clinician-delivered treatment?	Adding treatment could improve retention and clinical outcomes Does not reduce clinician burden
[TES + TAU] vs [Control + TAU]	3	What is the effect of adding TES to usual, clinician-delivered treatment, compared to adding a control intervention to usual treatment?	Maximizes internal validity External validity is weaker; does not increase efficiency or reduce treatment burden Controlled efficacy data for TES already exists [18]; classic efficacy trial unnecessary

The Type 2b `substitution' design was chosen because substance abuse treatment programs operate under conditions of limited funding in the face of high demand for services. The treatment program leaders on the protocol team saw the potential for TES as a `clinician-extender', substituting for some of the time and effort of front line clinicians delivering relatively standard aspects of substance abuse treatment (e.g., relapse prevention and alternative coping strategies). Thus, TES could both free clinicians to focus on issues unique to each patient, as well as free clinicians' time, potentially enabling a given program to include more treatment slots and serve more patients. Further, there is a high drop-out rate within community-based substance abuse treatment, a significant barrier to its effectiveness [32]. In addition to relapse, reasons for drop-out may include the burden of attending multiple group sessions per week or discomfort with group treatment. Therefore, the opportunity to attend fewer groups and instead interact with TES might be attractive to patients and reduce attrition.

In the Type 2a design, TES would have been added to TAU, instead of substituted for a portion of TAU. This design was attractive because it may have yielded a larger effect, as patients would have received extra treatment with no usual treatment removed. This design was also considered because in many treatment settings reimbursement is so low that TAU is limited. However, during discussions with treatment providers, it appeared that many community-based outpatient treatment programs were delivering enough treatment (at least 2 group sessions a week, or 6 or more groups per week for intensive outpatient programs) to make the substitution design (Type 2b) most realistic and attractive overall.

The Type 1 design, where TES would entirely replace TAU, would have similarly been attractive in environments where treatment availability is either very limited, or difficult to access (e.g., rural settings), or where stigma and potential discrimination are deterrents to face-to-face treatment with a clinician (e.g., for military personnel). However, providers raised concerns over regulatory challenges and clinician resistance to not providing any face-to-face therapeutic contact. The Type 3 design is similar to an efficacy trial except conducted in a community-based treatment program rather than a tertiary care research clinic. Since efficacy was established for TES [18] and its underlying components, [28,33] this design was less relevant.

2.2 Design Options to Test TES Intervention Components

Another important design decision surrounded the fact that TES consists of two components: 1) the computer-interactive cognitive behavioral intervention (CRA); and 2) prize-based contingency management. The contingency management component, which consists of an intermittent reinforcement approach in which participants earn chances to draw for prizes based on maintaining a target behavior, has previously been tested and found effective when added to community-based substance abuse treatment [34,35]. Thus, the protocol team debated whether it was important to disentangle the effects of the separate TES components compared to the full TES package. Table 2 displays the 2 by 2 factorial (4-arm) design that would fully dismantle the components. A 4-arm trial would have been complicated to implement and would have likely required a doubling of the sample size, substantially increasing the cost of the trial and/or burden to recruiting sites.

Table 2.

Alternative experimental designs considered to address the two components of the computer-assisted intervention (Therapeutic Education System or TES): 1) the computer interactive cognitive behavioral intervention (CRA), and 2) computer-managed contingency management. The setting is community-based substance abuse treatment, with patients receiving treatment-as-usual (TAU). A 4-arm, 2 by 2, factorial design is possible, fully disentangling the respective effects of computer-assisted counseling and prize-based contingency management. Various 3-arm, and 2-arm designs were also considered, each addressing more focused contrasts. The 2-arm design, indicated by the two shaded cells, was ultimately chosen, contrasting the full TES package (TES counseling + Incentives + TAU) versus TAU alone.

	Computer Interactive Counseling
	No	Yes
Contingency Management (CM)
No	TAU	Counseling + TAU
Yes	CM + TAU	Counseling + CM + TAU

Several 3-arm designs were also debated. For example, the design [TES full package – (i.e., computer interactive component + contingency management) + TAU] versus [TES computer interactive component + TAU] versus [TAU], would have addressed what effect incentives add to the computer-assisted therapy alone. Contingency management adds complexity and expense, and the treatment program providers on the protocol team felt it would be valuable to know if they were a necessary component of the treatment. In the study designs that allow contrast between active treatment components, the treatment effect is likely to be very small and require a larger sample to detect. However, even if differences are difficult to detect, there may be clinical utility in demonstrating the relative safety and effectiveness of TAU delivered with and without contingency management; especially important if programs are more willing to adopt the computer interactive component [35,36].

Determining the most appropriate design with multi-component interventions is a choice that investigators conducting effectiveness trials frequently encounter. Effective treatments, particularly behavioral interventions, often consist of combinations of active ingredients. The two shaded cells in Table 2 represent the 2-arm design ultimately chosen, namely to contrast the full TES package + TAU (with TES replacing a comparable portion of TAU each week) with TAU only. TES was developed as a package, modeled on the combined effect of CRA plus contingency management. In addition, it was decided that it was most important to test the contrast most likely to produce the largest effect and most benefit to treatment programs.

2.3 Site Selection

Once the design of the trial was finalized, the WEB-TX study team began a site selection process with the goal of choosing 10 community-based treatment programs that could successfully enroll 50 participants each (N = 500 overall; see section on Sample Size Calculation below). The site selection process introduced limitations to generalizability by the very nature of setting criteria for inclusion; 41 CTN-affiliated treatment programs were initially nominated to participate. However, site selection criteria were kept purposefully broad, as the trial was designed to take `all comers' to outpatient programs with the goal of determining how the intervention worked in a wide range of treatment seekers with various substance abuse problems.

Outpatient and intensive outpatient treatment settings were determined to be most appropriate, as this is the most prevalent type of treatment program and thus would increase the potential for adoption and subsequent impact on the field if positive results are obtained. Further, programs were eligible if TAU curriculum included at least 2 onsite therapeutic sessions per week (group and/or individual) for a minimum of 12 weeks. This was required to ensure that participants randomized to TES were receiving enough TAU to replace at least two sessions a week with TES and to integrate TES into existing treatment offerings. Allowing the full spectrum of TAU as practiced at different outpatient programs in different regions in the country enhanced external validity; although outcomes from the trial may not be generalizable to very low intensity programs. Interventions like TES could also be valuable in rural settings or underserved geographic areas where treatment availability is limited. In this case, a Type 2a design would be optimal to examine the effect of adding TES to existing TAU.

Opioid treatment programs were excluded from consideration. There are inherent differences in treatment approaches and client population and retention between programs that primarily provide opioid replacement therapies and those that provide outpatient or intensive outpatient treatment. Most relevant to this study, opioid treatment programs tend to have a more homogeneous population and less variability in terms of patient retention which is a co-primary outcome. Also excluded were sites that, at the time of site selection, used any form of contingency management or motivational incentives as a part of standard care. Sites were also considered based on geographic location, again to have the greatest representation across sites that participate in the CTN.

Sites were also screened for patient characteristics, including gender, ethnicity and type of substances treated. Across the 10 selected and participating sites, female patients comprised 10% to 67% (avg. = 34.7%) of program populations and the average percentage of Caucasian clients was 62.1% (range = 20% to 93%). Sites were excluded when the patient population was comprised of a vast majority of `alcohol only' clients and given preference with higher percentage of primary stimulant users, wanting to ensure diverse representation of substances of abuse. Participating sites reported an average of 36.7% (range = 20% to 60%) of patients with a primary alcohol problem and an average of 27.5% (range = 8% to 60%) with a primary stimulant problem. With the primary outcome of retention in mind, the percentage of patients who were mandated to treatment was also considered, as well as the average 90-day retention rates. If many patients were mandated to treatment, retention outcomes could be artificially inflated. If sites have overwhelmingly high patient retention, it would be difficult to show differences in retention as an outcome of the study (e.g., Petry et al. [34]). Participating sites had 50% or fewer patients who were mandated to treatment (avg. = 24.3%), and variability in the percentage of patients who remained in treatment for 90 days (range = 15% – 85%).

Selected sites had broad variability across attributes of interest, increasing the representativeness of the sample, a desirable attribute of effectiveness trials. This may mean, however, that not all sites are best positioned to test all hypotheses. For example, several sites in this sample reported high 90-day retention rates, making them less ideal to testing the retention hypothesis; but these sites had other attributes that made them compelling candidates for the trial.

An important aspect of effectiveness trials is to enable clinicians to use the intervention as they would in a non-research setting. In WEB-TX, clinicians meet with TES participants individually at least every other week to ensure regular face-to-face contact, providing an opportunity to `check-in' and assess TES participation. All clinicians received human subjects' protection training and protocol-specific training to familiarize them with TES. Further, reflecting standard program procedures, a program supervisor provides supervision to clinical staff at monthly intervals around the TES check-in procedures and assesses any challenges or concerns. Thus, TES is partially integrated into TAU, with clinicians making recommendations and inquiring about TES progress.

The decision not to have program staff deliver all aspects of the intervention was based on several issues, including: minimizing program and staff burden as a result of more extensive training, supervision, and workload; and remaining sensitive to participant confidentiality around biological measurement outcomes to minimize the influence of study participation on services they would normally receive. It is also likely that the logistical aspects of TES (e.g., setting up usernames/passwords, assisting with initial logon and access, and being available to answer technical questions) could be overseen by program support staff, rather than clinicians. Thus, there was an attempt to involve program staff in clinical components of the intervention which fit within TAU (i.e., treatment plan reduction decisions and conversations with clients about TES), but did not require the burden of implementing the intervention wholesale.

2.4 Participant Eligibility

Participant eligibility criteria (see Table 3) were kept broad to increase the external validity of outcomes. The primary criteria were substance use in the 30 days prior to baseline assessment (or 60 days if the participant was exiting a controlled environment) and randomization within the first 30 days of the outpatient treatment episode. Protocol procedures are designed, however, to recruit and enroll as early as possible in the treatment episode, ideally within the first week of treatment. Participants with primary alcohol use disorders are eligible if there is recent illicit substance use. This poses challenges to the objective verification of alcohol abstinence for contingency management procedures, as breath alcohol tests detect only recent use. The decision to broaden eligibility at the risk of strict adherence to contingency management tenets was based partially on procedures used in prior motivational incentive studies which produced positive outcomes (e.g., [34]).

Table 3.

Participant Eligibility Criteria

Inclusion Criteria

Male and female patients (≥ 18 years) accepted for outpatient, substance abuse treatment at a participating study site. Self-report any substance use problem, including alcohol. (1) Report use of a drug of abuse within 30 days prior to screening or (2) have exited a controlled environment (e.g., detoxification unit, hospital, or correctional facility) within 30 days of screening and report use of a drug of abuse within 60 days prior to screening. Within the first 30 days of initiating treatment at a participating study site. Self-report a planned treatment episode of at least 90 days (the planned active treatment phase in this trial).

Exclusion Criteria
Participating in opioid treatment programs and/or receiving opioid replacement medication. Plan to move out of the area within the next 90 days. Insufficient ability to provide informed consent to participate. Insufficient ability to use English to participate in the consent process, the intervention or study assessments.

2.5 Randomization

Randomization is conducted centrally by the CTN Data and Statistics Center (DSC2), using randomization schedules based on randomly permuted blocks stratifying by three factors to ensure relative equality of assignment. Randomization strata are defined by (1) clinical site, (2) participant's primary substance of abuse (dichotomized as stimulant vs. non-stimulant), and (3) whether or not the participant is abstinent at the time of randomization based on urine drug and alcohol breathalyzer tests. The treatment model of CRA plus prize-based contingency management, upon which TES is based, has been tested most for efficacy for cocaine dependence, hence the importance of balancing on the substance of abuse in the randomization. Abstinence at the time of treatment entry is common among substance dependent patients, and has been found to be a strong predictor of outcome [37,38], suggesting the importance of balancing on baseline abstinence in the randomization and including baseline abstinence as a covariate in the primary outcome analysis [39].

2.6 Study Procedures and Treatment Delivery

After completion of a baseline assessment and confirmation of eligibility, participants are randomly assigned to (1) TAU or (2) Reduced TAU + TES, for 12 weeks of treatment. During the active study treatment, participants in both conditions are asked to attend twice-weekly research visits to collect self-report and biological measures of substance use. At monthly intervals (week 4, 8, and 12) participants are asked to complete several additional assessments, as well 3- and 6-month post treatment follow up interviews. Both groups are compensated equally for research visit attendance and submission of biological samples.

2.6.1 TAU Condition

Individuals in the TAU condition receive standard level of care at their outpatient treatment program. Participating sites offer a range of outpatient services and from 2 to 12 hours of weekly treatment in their outpatient and intensive outpatient programs. Research staff document TAU on a weekly basis, collecting data on number of days of treatment, number of group and individual sessions, and total hours spent receiving treatment.

2.6.2 Reduced TAU + TES Condition

Individuals assigned to Reduced TAU + TES receive standard treatment at the participating treatment site, but replace approximately two hours of therapeutic activity each week with the interactive computer-assisted TES modules. TES consists of 32 `core' and 30 `optional' modules; each module addresses a specific topic, is self-directed, and utilizes informational technology to facilitate mastery of content through individually-paced testing [18]. Core modules consist of cognitive behavioral and relapse prevention skills-building and HIV prevention. Optional modules include social and relationship skill-building, employment and vocational topics, and more specific modules on HIV, hepatitis and sexually transmitted infection prevention. To better reflect participation requirements in routine outpatient treatment, and minimize burden, participants are asked to complete TES in conjunction with their two research visits per week and standard TAU days. Research staff collaborates with program clinicians to determine which components of a participant's TAU should be reduced to accommodate TES. For example, an issue-specific group that is clinically indicated for a particular participant (e.g., anger management, posttraumatic stress disorder) should not be replaced; instead, a more general group might be selected (e.g., psychoeducation, relapse prevention).

Each study site is equipped with a private work space as well as computer and internet access. A unique feature of internet-supported interventions is that participants can complete TES modules on-site or off-site at a time and place convenient to them. TES participants are given the URL to the TES website, an individualized account username, and a password which is valid through the 12-week treatment phase. Research staff is able to monitor all TES activity through an administrative `back-end system'.

A second component of TES is contingency management incentives that are focused on two clinically relevant treatment outcomes: abstinence and treatment participation. Incentives take the form of earned “draws” which can be redeemed within the TES program, similar to prior studies using abstinent-based contingency procedures [34]. Draws are awarded to participants for negative urine or alcohol breathalyzer screens for their primary substance of abuse and increase each week the participant remains abstinent. Participants earn two bonus draws each time samples are negative for all substances. One draw is also earned for each TES module completed, up to four per week (however participants may complete as many modules as they like). Providing incentives for participation (i.e., TES module completion), as well as abstinence, addresses the problem of high drop-out rates frequently observed in outpatient substance abuse treatment. Approximately half of the draws result in a congratulatory “good job”; the other half reward the participant with either small ($1), large ($20), or jumbo ($80–100) prizes in decreasing probability. Incentives are tailored to a site's particular setting and client population and participants have input to the prizes that will be available in order to enhance prize salience. A TES participant who remained abstinent from all substances and completed all required modules over the 12-week treatment phase could earn 252 draws and an average of $600 in prizes.

2.7 Specification and Analysis of the Primary Outcomes Measures

This study considers co-primary outcomes: one capturing abstinence during treatment, and the other capturing retention in treatment. Both outcomes were relevant to outpatient treatment providers and appropriate target behaviors of the intervention.

2.7.1 Abstinence Outcome

The primary abstinence outcome is self-reported substance use/heavy drinking collected using the TimeLine Follow-Back (TLFB; [40]) and confirmed with a urine drug screen (UDS). The TLFB is completed once per week during the 12-week active treatment phase and urine samples are collected twice a week. Because UDS are collected twice per week, abstinence over the treatment period is measured by half-weeks instead of full seven-day weeks, with the end of a half-week defined by when the UDS was performed. Thus a fully compliant participant completing all UDSs would contribute 24 half-weeks towards the abstinence outcome. Participants are considered abstinent from all substances in a given half-week if their urinalysis at the end of the half-week and their self-report indicate no drug or heavy alcohol use. A half-week with either missing self-report or a missing UDS is treated as missing for the purposes of analysis, with two exceptions: 1) if self-report is missing for any day in the half-week, but the UDS is positive, the participant is considered non-abstinent; and 2) if the UDS is missing in a given half-week but the self-report is positive, the participant is non-abstinent. Utilizing self-report and biological confirmation for primary substance does have inherent limitations, as when the primary substance is marijuana (longer detection time) or another substance with varying detection times.

Because abstinence is measured repeatedly at half-week intervals throughout the 12 weeks of treatment, the primary analysis of this outcome measure will be longitudinal. The analytic approach will be to fit a logistic regression for the odds of abstinence at each half-week using generalized estimating equations [41] to adjust for the correlation of half-weeks within each participant. This approach can also account for the correlation within study site. The longitudinal model will include a linear time by treatment interaction during the first eight weeks (16 half-weeks) of the active treatment phase, since the effect of contingency management has been shown to change over time [42,43]. A constant treatment effect is then assumed throughout the last four weeks (8 half-weeks) of the treatment phase, the time period by which the effect of TES will be formally tested for the primary analysis. Use of this piece-wise longitudinal model also allows all participants who have completed at least one half-week to contribute to the analyses. Since this model makes several assumptions, sensitivity analyses will assess the effect of violating any of the assumptions on the results. Splines and other methodologies can be used to determine whether the piece-wise linear model is reasonable. This analysis plan also assumes that the data are missing at random (i.e., the probability of attending a study visit is not a function of whether the participant was abstinent in the preceding half-week). Due to the nature of substance use, it is fairly likely that this assumption will be violated. While there is no formal test of the missing at random assumption, there are statistical methodologies available to assess the sensitivity of study results to this assumption, such as pattern mixture models [44]. We will also consider several methods of imputation of missing abstinence data. Single imputation methods will be applied ranging from the most conservative (assuming all missing values are non-abstinent) to the most generous (assuming all missing values are abstinent). Multiple imputation methods can also be used where the selection model predicting the likelihood of abstinence during a half-week is based on abstinence during previous half-weeks and additional covariates.

An alternative method for handling the primary abstinence outcome – summing weeks of abstinence – was also considered but rejected based on the following reasons. First, summing weeks would involve imputation of multiple missing UDS weeks, so when the sum of abstinent weeks is computed, imputed data are combined with observed data. Second, summing imputed data can create a bias that will differ across participants depending on the amount of missing data. Finally, summing abstinent weeks also assumes no temporal effect of treatment. Since participants are completing multiple TES modules per week, there may be a cumulative effect; therefore, a longitudinal approach is more appropriate.

2.7.2 Retention Outcome

The primary outcome of retention is defined as the number of weeks from randomization to the last face-to-face visit at the treatment program. Contingency management have been shown to reduce dropout in outpatient, community-based treatment [34]. It was also hypothesized that TES would be attractive to participants, further helping to reduce dropout. For participants who complete treatment, retention time will equal 12 weeks. For participants who do not complete their substance use treatment, retention time is calculated based on their last treatment visit and will range from 0 to 11 weeks. Thus, retention data will be available for all participants. A stratified proportional hazards model will be used to assess this treatment effect between the Reduced TAU + TES and TAU groups adjusting for potential confounders and two of the stratification factors involved in randomization: negative/positive UDS at baseline and stimulant/non-stimulant as primary substance of use. More complex methods, such as copula modeling, may also be implemented to adjust for the correlation of participants within a site.

2.7.3 Sample Size Calculations

Since the study considers two co-primary endpoints simultaneously, the primary analyses will account for multiplicity of comparisons to protect the type I error using the method of [45]. The overall null hypothesis of no treatment effect on abstinence or retention can be rejected if either of the following two conditions holds: both outcomes are significant at the α = 0.05 level or only one outcome is significant at the α = 0.025 level. Sample size computations are based on the more conservative (i.e., requiring somewhat larger sample size) Bonferroni adjustment approach, in which both individual hypotheses need to be rejected at α = 0.025 level. For the abstinence outcome, there is more than 80% power to detect an odds ratio of 1.5 when considering two-sided α = 0.025, and the study is sized at 500 participants. For the retention outcome, assuming 50% retention at 12 weeks in Reduced TAU + TES group and 35% (based on data from Petry et al., [34]) in the TAU group, there is 90% power to reject the hypothesis with a total sample size of 500.

2.8 Cost Effectiveness Analysis

A critical component of this effectiveness trial is the integration of a comprehensive economic evaluation to inform the potential adoption of this computer-assisted intervention. What makes this study unique is the potential for the costs of computer-assistance to be offset by reductions in TAU. It is possible that from the perspective of the treatment program, this tool would have cost advantages if it allows evidence-based psychosocial treatment to be provided via the web-based system, thus allowing clinicians to reduce their contact time with some patients and enabling treatment of a larger number of patients. Further, if TES is shown to markedly improve treatment outcomes relative to TAU, it may be viewed as cost effective from the perspective of a substance abuse treatment program even if there are some added costs of the intervention.

Prior research on cost-effectiveness of computer-assisted interventions has been sparse and limited in scope [46,47]. These studies are based on single-site clinical trials, do not take a payer or societal perspective, and do not consider measures of effectiveness beyond the narrow clinical outcome. The implementation of this effectiveness trial within the CTN offers a unique opportunity to assess cost-effectiveness in a multi-site environment. Broadening the economic assessment to consider a comprehensive set of outcomes from the perspective of key stakeholders offers the opportunity to inform future implementation decisions and research.

The economic analyses are also clinically important. Even if the computerized intervention is shown to be effective in terms of the primary outcomes, it may not be adopted in community-based treatment programs unless it is shown to be cost-effective due to the considerable financial constraints in such treatment settings. In particular, the upfront investment in training time, computer hardware and space allocation along with the uncertainty regarding ongoing expenses may become a barrier to an adoption decision unless there is sufficient evidence that this outlay will provide sufficient value. In effectiveness trials, aspects of an intervention may be managed by research staff. In WEB-TX, this includes entering biological screening data, managing prizes, and answering questions related to module completion. It is important that these tasks also be considered as potential clinic costs in the economic analysis. Therefore, the primary economic outcome is the cost-effectiveness of clinic-specific costs per increased abstinence time.

However, the limitation of costs expressed in terms of abstinence time is the lack of broad agreement as to what level of additional costs a clinic is willing to incur to achieve a given increase in the length of abstinence time. Therefore, the economic analysis includes a secondary measure that expresses costs in terms of quality adjusted life years (QALYs). The advantage of QALYs as an effectiveness measure is the ability to compare value in substance abuse treatment to acceptable thresholds of value in other health care settings. This translation of cost-effectiveness to the broader purchasing of health care services may provide the type of economic evidence that may be useful to negotiating reimbursement with third party payers. Because of the usefulness of this evaluation to perspectives beyond the clinic, this evaluation is designed so that costs can also be estimated from a payer perspective (e.g., facility costs associated with using TES) and from the societal perspective (e.g., criminal and workforce behavior).

2.9 Security and Confidentiality

Electronic documents and data create challenges for ensuring trustworthiness, integrity, security, and confidentiality of patient-sensitive data. Threats to security and confidentiality need to be addressed by providers and researchers, especially for vulnerable populations engaged in stigmatized, illicit behaviors such as drug use. The TES-associated patient-sensitive data in this study are secure and password-protected, and they meet the same security and confidentiality requirements and data management procedures that have been used in other CTN trials. These data are being housed and stored in the centralized electronic data capture system of the CTN DSC2 along with those of other trials. The CTN DSC2 has a well-established system for data collection and management and for monitoring data entry at each treatment program. Data and application logic are stored in one centralized place making application updates, backups, and data analysis and security easier.

In this study, the web-based TES intervention may be accessed by participants on computers supplied by the research study, as well as off-site at a computer of the participant's choosing. Therefore, additional information about confidentiality risks posed by off-site use was included in the informed consent and reviewed with participants. That is, if participants choose to complete TES modules outside the treatment program, the URL address for the website where TES is housed will remain in the browser's history. This website could then be accessed by someone else, which would direct them to the HealthSim, LLC website (the developer of TES). While there would be no access to the participant's research information, the HealthSim, LLC website contains general information about development of applications for substance abuse prevention and intervention. In addition, when TES is accessed, the Internet Protocol (IP) Address from the computer will be captured in the study database. An IP Address is a numerical label assigned to each computer that accesses the internet and is considered Protected Health Information. This becomes a confidentiality issue when participants access TES offsite from personal computers. In WEB-TX, all IP Addresses captured by the study database will be maintained in that secure database only. IP Addresses will not be included in any study reports or data analysis.

3.0 Conclusions

The WEB-TX study was designed to evaluate the effectiveness of including a web-based version of the Community Reinforcement Approach, plus prize-based contingency management targeting drug abstinence and treatment attendance, TES, as part of community-based, outpatient substance abuse treatment. The design addresses calls for increased scientific rigor for studies of computer-assisted therapies for substance use disorders [12,17]. The study uses a randomized controlled design (with independent treatment arm allocation), pre-study analysis to ensure adequate power, intention-to-treat primary outcome analysis, biological markers for the primary outcome of abstinence, long-term follow-up assessments, and precise measurement of intervention dose. Conducting the study within the NIDA funded CTN allows for collection of data from a large and diverse treatment-seeking sample to facilitate subgroup analyses and the examination of possible treatment moderators (e.g., gender, age, primary substance of abuse). Outcomes from this study will further the empirical knowledge base on the effectiveness and cost-effectiveness of computer-assisted treatment in clinical treatment settings.

Input from community providers during protocol development highlighted potential concerns and helped to address issues of practicality and feasibility. Selected design elements helped allay clinical concerns and created a protocol that was scientifically rigorous, as well as feasible, and acceptable and relevant to treatment providers. For example, introducing the study as a means of testing a `clinician extender' model clarified how a technological innovation could be used to address common treatment challenges while minimizing provider concerns about using the computer to replace clinical expertise. The clinician extender model increases clinician time to work with more clinically severe patients, while leaving the review of repetitive but necessary skills training to TES.

In addition, including all clinicians in the conduct of the trial (i.e., all clinicians have the possibility of being assigned a TES participant) and integrating TES supervision into usual clinical staff meetings increased engagement and better reflected how a computer-assisted intervention would be incorporated into treatment programming and potentially assist with future implementation efforts. Provider input was necessary to determine how clinician/patient interactions related to a computer-assisted intervention would be structured. Integrating a computer-assisted intervention into ongoing treatment was also indicated in a small feasibility study examining TES in outpatient substance abuse treatment [48]. The early success of the current study in garnering community-based treatment program support highlights the need for infrastructures that continue to enhance partnerships between researchers and providers to facilitate effectiveness trials and dissemination of promising, technologically innovative treatments.

Several limitations of the WEB-TX study should be noted. First, access to technology is required for implementation of computer-assisted interventions. As noted earlier, internet access in general is increasing [4,5,6], but there is also preliminary evidence that low-income individuals in substance abuse treatment also report improved computer access and knowledge [49]. Accessibility concerns can also be addressed by allowing clients access to computers in the clinic-setting, as is the case in WEB-TX, and improving the translation of interventions onto mobile, hand-held devices like cell phones.

As with any effectiveness trial, design and methodological decisions may place limits on generalizability. In WEB-TX, research staff has oversight over TES administrative tasks, rather than program staff implementing all aspects of the intervention. Dependent on the outcomes of this trial, future research should examine implementation within treatment systems and across programs to assess how TES may best be integrated into service delivery and by whom (e.g., clinical staff versus support staff). Including participants with primary alcohol use disorders creates challenges to contingency management procedures by being able to detect only recent use. Excluding primary alcohol users, however, would greatly limit the representativeness of the sample. A subsample analysis with this group could be an informative undertaking. Finally, allowing offsite completion of TES raises the concern that modules could be completed by someone other than the participant. Although this poses some threat to the accurate capture of intervention dose, it was felt the greater flexibility outweighed this relatively small risk.

Overall, the design of WEB-TX addresses important issues of external validity. The inclusion of diverse outpatient programs, as long as they provide at least 2 hours of treatment per week, increases generalizability of study outcomes. This level of service allows for TES to replace TAU while reflecting real-world treatment provision. Broad participant eligibility criteria create an `all comers' trial, increasing the relevance of outcomes across diverse outpatient substance abuse treatment populations. The inclusion of a cost-effectiveness analysis addresses a long-standing barrier to the dissemination of ESTs. Although interventions may be found efficacious, without concurrent knowledge of fiscal burden, adoption and implementation of ESTs may never occur. Since standard substance abuse treatment services are reimbursed based on face-to-face encounters, it is not currently clear how programs will pay for computer-assisted interventions like TES. State and local substance abuse treatment stakeholders will need to determine best practices for integration of computer-assisted interventions into health systems, how it would be paid for, and whether or not the new program will be worth the investment under various reimbursement scenarios.