Working Memory Training for Adolescents With Cannabis Use Disorders: A Randomized Controlled Trial

Abstract

Adolescent cannabis use is associated with working memory impairment. The present randomized controlled trial assigned adolescents ages 14 to 21 enrolled in cannabis use treatment to receive either working memory training (experimental group) or a control training (control group) as an adjunctive treatment. Cognitive function, drug use, and other outcomes were assessed before and after training. We observed few differences in cognitive, functional, or self-reported drug use outcomes as a function of training group, although tetrahydrocannabinol (THC) urinalysis results favored the experimental group. These findings are similar to previous studies in substance users, which have shown limited transfer effects for working memory training.

Cannabis use disorders in adolescents represent a significant mental health and treatment burden in the United States. The United States National Survey of Drug Use and Health (NSDUH) estimates that 3% of adolescents ages 12 to 17 and 5% of young adults ages 18 to 25 had a cannabis use disorder in the past year (Substance Abuse and Mental Health Services Administration [SAMHSA], 2016a). Similarly, national reporting data from the Treatment Episode Data Set for 2014 indicate cannabis as the primary substance of abuse for 76% of treatment admissions for patients ages 12 to 17, whereas alcohol and opiates accounted for only 12% and 3% of treatment admissions in this age group, respectively (SAMHSA, 2016b). Furthermore, there were more cannabis treatment admissions for patients ages 12 to 24 than the total cannabis admissions for all adults 25 and older, suggesting that adolescents and young adults bear a disproportionate burden of cannabis use disorders requiring treatment (SAMHSA, 2016b). Behavioral treatments for cannabis use disorders generally result in improved outcomes relative to control conditions (Davis et al., 2015), but rates of abstinence tend to be modest even with effective treatments (e.g., Stanger, Ryan, Scherer, Norton, & Budney, 2015) and treatment effects tend to be smaller with longer periods of follow-up (Bender, Tripodi, Sarteschi, & Vaughn, 2011). Given the prevalence of cannabis use disorders in adolescents and challenges to treatment, there is a need to develop additional cannabis use disorder treatments that will improve long-term outcomes.

One potential obstacle to successful treatment may be cognitive impairment associated with adolescent cannabis use. Working memory is one particularly important domain of cognitive functioning that is cross-sectionally associated with behavioral impairment or differential brain response in adolescent or early-onset cannabis users (Becker, Wagner, Gouzoulis-Mayfrank, Spuentrup, & Daumann, 2010; Fried, Watkinson, & Gray, 2005; Hanson et al., 2010; Harvey, Sellman, Porter, & Frampton, 2007; Jager, Block, Luijten, & Ramsey, 2010; Medina et al., 2007; Padula, Schweinsburg, & Tapert, 2007; Schwartz, Gruenewald, Klitzner, & Fedio, 1989; Schweinsburg, Brown, & Tapert, 2008; Schweinsburg, Nagel, et al., 2008; Schweinsburg et al., 2010; Smith, Longo, Fried, Hogan, & Cameron, 2010; Vo, Schacht, Mintzer, & Fishman, 2014). Working memory, which is the ability to manipulate small amounts of information to serve a current goal, is a major component of broader executive functions that serve to select, initiate, monitor, and modulate other cognitive activities (Baddeley, 1992; D’Esposito, 2007; Jurado & Rosselli, 2007; Repovs & Baddeley, 2006). Deficits in working memory are especially concerning due to working memory’s critical role in daily cognitive and psychosocial functions. Better working memory functioning has predicted greater educational achievement (Gathercole, Pickering, Knight, & Stegmann, 2004), better emotion regulation (Schmeichel, Volokhov, & Demaree, 2008), greater behavioral inhibition (Finn, Justus, Mazas, & Steinmetz, 1999), better decision making in laboratory tasks in which immediate gains are paired with higher future losses (Bechara & Martin, 2004), and increased responsiveness to substance abuse treatment (Moeller et al., 2010; Teichner, Horner, & Harvey, 2001). Thus, cognitive deficits associated with cannabis use may perpetuate ongoing use and contribute to substance use treatment failures, despite evidence that some cannabis-related neurocognitive deficits resolve after longer periods of abstinence (Fried et al., 2005; Hanson et al., 2010; Hooper, Woolley, & De Bellis, 2014; Pardini et al., 2015). Therefore, the development of interventions to improve working memory in adolescents with cannabis use disorders warrants study.

Recent studies of working memory training, which typically consists of computerized exercises requiring participants to hold in mind and manipulate sequences of numbers or shapes across repeated sessions, have shown promising outcomes in adult substance users and children with attention deficit hyperactivity disorder (ADHD). Houben, Wiers, and Jansen (2011) observed improved working memory and reduced alcohol drinking following working memory training in adult problem drinkers. In another study, a cognitive training regime broadly targeting executive function and memory domains (including working memory exercises) in alcohol-dependent inpatients resulted in significant improvement in executive function including working memory, improved psychological well-being, and reduced compulsive craving of alcohol (Rupp, Kemmler, Kurz, Hinterhuber, & Fleischhacker, 2012). A controlled trial with adults in treatment for stimulant use disorders found that working memory training decreased delay discounting (i.e., preference for a smaller sooner reward over a larger later one), but found no effects of training on other measures of cognitive performance, including measures of working memory itself (Bickel, Yi, Landes, Hill, & Baxter, 2011). The effects of working memory training on delay discounting are encouraging for the application to adolescents with cannabis use disorders, because greater delay discounting predicts poorer treatment outcomes for adolescent cannabis users (Stanger et al., 2012). Still, the generalizability of the effect of working memory training on delay discounting requires further study, as data do not consistently indicate an effect. Working memory training did not reduce delay discounting in methadone maintenance patients, despite reduced drug use and some working memory improvement in patients who received working memory training relative to control training (Rass et al., 2015). Recent meta-analyses of working memory training for children with ADHD suggest improvement in executive functioning and ADHD symptoms following working memory training (Cortese et al., 2015; Peijnenborgh, Hurks, Aldenkamp, Vles, & Hendriksen, 2016; Shinaver, Entwistle, & Soderqvist, 2014; Spencer-Smith & Klingberg, 2015; cf. Melby-Lervåg & Hulme, 2013; Redick, Shipstead, Wiemers, Melby-Lervåg, & Hulme, 2015, for review). Improvement in ADHD symptoms (e.g., impulsivity) may be relevant for adolescents with cannabis use disorders, given that ADHD is a prospective predictor of adolescent substance use disorders (Lee, Humphreys, Flory, Liu, & Glass, 2011). As a whole, data examining working memory training in relevant populations suggest that working memory training warrants further study as a potentially useful adjunctive treatment for adolescents with cannabis use disorders.

The prevalence of cannabis use disorders in adolescents and the ongoing difficulty with its treatment justifies the need to explore adjunctive treatments that will improve outcomes. Given the importance of working memory to everyday functioning, documented deficits of working memory within cannabis users, and promising effects of working memory training in other relevant populations, working memory training is a well-justified potential adjunctive treatment to adolescent cannabis use disorder treatment. To date there have been no studies of working memory training in adolescents with cannabis use disorder. We sought to address this gap using a randomized, controlled trial to evaluate the effects of working memory training on cognitive and drug use outcomes in adolescent patients undergoing treatment for cannabis use disorder. In addition to examining the effects of working memory training on cognition and drug use, we also assessed potential effects of training on broader functional outcomes, including measures of delay discounting, mental health, reading, and risk behavior. Accordingly, the present study is a rigorous and in-depth examination of potential effects of working memory training in adolescents seeking treatment for cannabis use disorder.

Method

Participants

Individuals were recruited from the Mountain Manor Treatment Center in Baltimore, Maryland. Eligibility criteria included fluency in English, age 14 to 21 years, enrollment in treatment for substance use disorder (determined by American Society of Addiction Medicine Patient Placement Criteria [ASAM PPC2-R] and Diagnostic and Statistical Manual of Mental Disorders [DSM-IV] criteria), with cannabis being the primary substance of abuse. Ongoing treatment for cannabis use disorder at Mountain Manor Treatment Center includes group treatment sessions multiple times weekly with counselors trained and supervised according to principles of cognitive behavioral therapy, motivational enhancement therapy, and motivational interviewing approaches. In addition to group services, patients may be referred to receive additional mental health services, psychiatric medications, or inpatient treatment through Mountain Manor Treatment Center. Exclusion criteria were untreated Axis I psychiatric disorders (except substance use disorder); use of any substance of abuse other than caffeine, nicotine, or cannabis, four or more times per week; or any condition associated with significant cognitive impairment (e.g., intellectual disability or severe brain injury). Participants could be recruited for the study at any point during their ongoing substance use treatment plan; study assessments were not implemented at specific intervals of ongoing substance use treatment. Participants ages 18 years and older gave oral consent. For those participants under 18 years, a legal guardian gave oral consent and then the participant gave oral assent. All participants received compensation for participation. The Johns Hopkins University School of Medicine Institutional Review Board approved all study procedures. One hundred participants consented and were assessed for eligibility, 87 participants completed the pre-assessment and were randomized, and 37 completed the study (see Figure 1). Figure 1 shows that attrition in the study was most commonly due to failure to adhere to ongoing substance use treatment rather than failure to engage with the study procedures specifically.

Figure 1.

Participant flow diagram.

General procedures

A demographics questionnaire and medical chart review were used to assess participant characteristics at screening (e.g., age, gender, race, ethnicity, education, living situation, employment, parent employment and education, history of head trauma, and court-ordered treatment status). Consent and screening sessions were conducted individually. After consent and screening, qualified participants completed pre-training assessment (described later). Following pre-training assessment, participants were randomly assigned to either experimental or active control computerized training conditions using a minimization procedure (Pocock, 1983; Scott, McPherson, Ramsay, & Campbell, 2002) to balance the groups on the following factors measured at consent or at pre-assessment: age (14–17 versus 18–21 years), gender, self-report of ADHD symptoms (<6 versus ≥6 items endorsed for either ADHD Inattentive or ADHD Hyperactive questions on the Global Appraisal of Individual Needs–Initial Lite questionnaire [GAIN-I Lite; Chestnut Health Systems, 2010]), and baseline working memory as determined by the Wechsler Adult Intelligence Scale (WAIS-IV; Wechsler, 2008) or the Wechsler Intelligence Scale for Children (WISC-IV; Wechsler, 2003) (<18 versus ≥18 sum of scaled working memory scores). A research assistant who did not administer assessments enrolled participants and implemented the randomization procedure. Participants completed training sessions and pre- and post-training assessment sessions (described later) during nightly group substance use treatment counseling at Mountain Manor Treatment Center. The session was rescheduled if the participant showed signs of intoxication or reported excessive fatigue.

Pre- and post-training assessments

Participants completed assessment sessions individually before and after the training sessions. Both the pre-training and post-training assessment sessions lasted approximately 2.5 hours. During assessment sessions, a research assistant blind to training condition administered a battery of cognitive tasks and substance use and mental health assessments. Assessments consisted of both paper and computer-administered assessments, described later. Primary working memory outcomes were defined as outcomes from the WISC-IV/WAIS-IV and the visuospatial working memory task, but we assessed additional cognitive outcomes (e.g., inhibitory control, sustained attention), substance use (i.e., self-report, urinalysis), and secondary functional outcomes (e.g., self-reported risk behaviors). Secondary functional outcomes were included to determine whether potential working memory improvements may result in improved functional outcomes in domains indirectly related to working memory but nonetheless important for psychosocial adjustment, such as self-reported risk behavior and reading.

Cognitive outcomes

WISC-IV/WAIS-IV

Working memory index and working memory index percentile scores were calculated based on the participant’s performance on the WISC-IV (Wechsler, 2003), if age 16 or younger, or the WAIS-IV (Wechsler, 2008), if age 17 or older. Raw score for digit span and letter-number sequencing/arithmetic tasks was scaled based on age norms and summed to determine the working memory index and working memory index percentile. Outcomes were summarized using measures that were comparable across the WAIS-IV and WISC-IV: working memory index, working memory index percentile scores, digit span forward (raw score), digit span backward (raw score), and digit span scaled (score scaled according to age norms).

Visuospatial working memory task

For the visuospatial working memory task (Rapport et al., 2008), participants viewed a computer screen with nine squares arranged in an offset 3 × 3 grid pattern where three, four, or six dots were serially presented. One red dot appeared within each trial and the remaining dots were black. The location of the red dot was counterbalanced across trials to appear in each of the nine squares. The red dot was not presented as the first or last stimulus in the sequence to minimize primacy or recency effects. Each trial block (three, four, or six dots) had 24 trials. Participants were asked to replicate the sequence of dots, but to enter the position of the red dot last. The outcome was proportion of correctly replicated sequences across the three trial blocks (three, four, or six dots).

Paced serial addition test (PSAT)

We assessed performance on an adapted, computerized version of the Paced Auditory Serial Addition Test (PASAT). The PASAT is used to assess multiple dimensions of cognitive function related to attention, including sustained attention, divided attention, working memory, and speed of information processing (Tombaugh, 2006). The PASAT was adapted such that the mode of presentation was visual rather than auditory (thus referred to hereafter as PSAT rather than PASAT), which may decrease task difficulty (e.g., Tombaugh, Rees, Baird, & Kost, 2004). The adapted PSAT task visually presented a series of single-digit numbers one at a time (350-ms stimulus presentation) in four consecutive blocks: the practice block consisted of 31 stimuli with an inter-stimulus interval (ISI) of 3.2 seconds, and the three test blocks consisted of 61 stimuli each and increased in speed with the ISI decreasing from 2.8 seconds to 2.4 seconds to 2.0 seconds. Participants were instructed to add the first two numbers and click on the correct answer. The task then required each participant to add every subsequent number to the number preceding it and click on the correct sum. Outcomes were proportion of correct trials, mean reaction time on correct trials, and proportion of dyads. Dyads were defined as the number of correct answers preceded by a correct answer; fewer dyads signified an increased use of an alternative strategy for the task, whereby participants add two numbers, skip one, add two numbers, skip one, and so forth (Tombaugh et al., 2004).

Continuous performance test

A modified version of the Continuous Performance Test (e.g., Rosvold, Mirsky, Sarason, Bransome, & Beck, 1956) measured sustained attention and response inhibition via computer. Participants were shown a series of fixation points followed by letters appearing on the computer screen one at a time. Participants pressed the space bar every time they saw a letter except X. There were 15 practice trials followed by 300 task trials (30 X trials and 270 trials presenting letters other than X). Each letter was presented for 80 ms with an ISI of 250 ± 50 ms. Outcomes were proportion of correct non-X trials (hit rate), proportion of incorrect X trials (false alarm), sensitivity in distinguishing between non-X letters and X (d′), response bias (C), and reaction time for correct trials (ms).

Stroop task

Four blocks of the Stroop color-word interference task were administered via computer: a classic color-word Stroop, cannabis Stroop, neutral word Stroop, and letter string Stroop. The classic color-word interference Stroop block presented color words (yellow, red, green, blue; e.g., Stroop, 1935) in incongruent colors, the cannabis block presented words related to cannabis (e.g., “cannabis,” “dealer,” “dope,” “ganja”; similar to Field, 2005), the neutral word block presented words representing household items (e.g., cabinet, sofa, oven, lamp; similar to Marhe, Luijten, van de Wetering, Smits, & Franken, 2013), and the string block presented strings of the letter “X” of varying length (e.g., XXX, XXXX, XXXXX, XXXXXX; similar to Marhe et al., 2013). Participants were asked to ignore the meaning of the word and press the color key that matches the color in which the word or string was printed, working as quickly and accurately as possible. The four blocks were presented in a random order with 20, 500-ms trials randomized within each block and a 30-second break between blocks. The outcomes for each block were proportion correct responses and reaction time for correct trials. The color-word interference block was compared to the string block (where the string block should have no interference from word content and thus quicker reaction times) to assess any improvement of inhibition in color-word interference as a function of training condition. In previous work, cannabis-dependent individuals have demonstrated slower reaction times to identify the color of cannabis words relative to neutral words (Field, 2005). Thus, performance on the cannabis block may be compared to the neutral word block to examine any changes in potential attentional bias for cannabis-related words.

Delay discounting

The Quick Discounting Operant Task (QDOT; Johnson, 2012) was used to assess inter-temporal choice between a small immediate reward and a larger delayed reward via computer. Participants made a series of discrete choices between a small immediate reward (e.g., get 40 cents now) and a larger later reward (e.g., wait five seconds to get 80 cents) across five delays (i.e., 5, 10, 20, 40, and 80 seconds). The smaller reward was adjusted across four trials per delay to approximate the point at which the smaller immediate reward is subjectively equal to 80 cents after a delay (i.e., indifference point) using a previously described algorithm (Du, Green, & Myerson, 2002; Johnson, 2012; Kowal, Yi, Erisman, & Bickel, 2007). On each trial, a coin dispenser delivered the amount of money chosen either immediately or following the indicated delay. A waiting period at the end of the task ensured that total task time was not dependent on participants’ choices. Delay discounting was summarized using area under the curve (AUC; Myerson, Green, & Warusawitharana, 2001) of the indifference points, in which a lower AUC corresponds to greater delay discounting (i.e., relatively greater preference for smaller, sooner rewards).

Substance use outcomes

Self-reported substance use

Experimenters documented participants’ self-reported use of cannabis, tobacco, and alcohol on each day during the past 30 days in the community (excluding time spent in juvenile detention, jail, or inpatient treatment) using timeline follow-back (TLFB) procedure (Sobell & Sobell, 1992). Cannabis use was summarized as days of cannabis use and the number of joints/blunts smoked, tobacco use as days of tobacco use and the number of tobacco cigarettes smoked, and alcohol use as days of alcohol use and the number of drinks consumed (where one drink was equivalent to 12 ounces of beer, 4 ounces of wine, or 1.5 ounces of spirits).

Urinalysis

Once per week throughout training, the study team recorded results of urinalysis (i.e., tetrahydrocannabinol [THC] positive or negative) conducted as part of the treatment program at Mountain Manor Treatment Center during the cognitive training sessions. Thus, research assistants who conducted pre- and post-assessments were blind to urinalysis results.

Secondary functional outcomes

Risk behavior and functional impairment

Part of the GAIN-I Lite (Chestnut Health Systems, 2010) was used to assess psychological and behavioral functional outcomes over the past 30 days (regardless of time spent in juvenile detention, jail, or inpatient treatment). Primary outcomes included the number of days bothered and impaired by psychological problems, the number of days with behavioral control problems, the number of days involved in criminal activity besides drug use, instances of unprotected sex (i.e., the number of times the participant reported having any kind of vaginal, oral, or anal sex and did not use a condom; participants who did not have sex are included as zero instances of unprotected sex), and proportion of protected sex (i.e., the proportion of times the participant reported having sex and using a condom; participants who did not have sex were excluded).

Difficulties in emotion regulation

Self-reported difficulties in emotion regulation were assessed using the Difficulties in Emotion Regulation Scale (DERS; Gratz & Roemer, 2004), with higher scores (overall range from 36 to 180) indicating greater difficulties.

Reading

Reading comprehension and fluency (i.e., a summary measure of accuracy and rate) were assessed at pre- and post-assessment using the Gray Oral Reading Test–Fifth Edition (GORT-5; Wiederholt & Bryant, 2012).

Therapeutic alliance

Therapeutic alliance between the participant and his or her assigned substance abuse counselor at Mountain Manor Treatment Center was assessed at pre- and post-assessment using the therapist version of the Working Alliance Inventory-Short Revised (WAI-SR; Hatcher & Gillaspy, 2006; Horvath & Greenberg, 1989), which was completed by the counselor.

Training sessions

Training sessions consisted of up to 25 30-minute sessions one to four times per week and were completed at Mountain Manor Treatment Center in an experimental session room separate from other therapeutic rooms (e.g., group therapy rooms). Multiple participants (one to four) completed training sessions simultaneously at separate computer workstations wearing headphones. Research assistants monitored participants during training to keep them on task and prevent distraction. Staff sought to provide equivalent encouragement (e.g., “Good focus”) and monitoring for all participants during training regardless of experimental condition. Participants must have completed at least 10 training sessions to be eligible to complete post-assessment. Although the aim was to complete at least 25 training sessions for all participants, the frequency and number of sessions was dependent in part on the participant’s treatment plan and schedule at Mountain Manor Treatment Center. A true intent-to-treat approach would include all participants in post-assessment regardless of the number of sessions completed. However, the best preliminary test for intervention efficacy would be to ensure cognitive training was of sufficient intensity to benefit outcomes. Thus, the required minimum of 10 sessions to complete the post-assessment was considered a reasonable compromise between testing an intervention of sufficient strength (i.e., less than 10 sessions would not be expected to benefit study outcomes) and including as many randomized participants in post-assessment as possible, given difficulties with adherence to ongoing substance use disorder treatment.

All participants were compensated on an intermittent schedule of reinforcement for attending training sessions in which they could earn draws from a bowl of tokens, similar to other interventions in substance use populations (e.g., Petry & Martin, 2002; Petry, Martin, Cooney, & Kranzler, 2000). Participants earned draws based on attendance for sessions, with the number of per-session draws increasing across sessions up to five draws. If a participant attended three consecutive sessions, three bonus draws were added to the total number of draws. Thus, there was a maximum of eight draws per session. The number of draws was reset to one if the participant missed at least two sessions in one week. The number of draws was not contingent on drug abstinence. Tokens could be exchanged for a reward with no monetary value (i.e., “Good job!”), small amounts of cash (i.e., $1 to $5), or gift cards (i.e., $5 to $80 retail gift cards). Prizes of greater value were less likely to be drawn.

Adaptive working memory training program and non-adaptive control group

The training program (Cogmed RM; Cogmed Inc.; www.cogmed.com) consisted of 12 exercises (eight per session) that require the maintenance and manipulation of sequences of verbal and/or visuospatial information in working memory. Three exercises were common across all sessions, whereas five varied across sessions. Training in the experimental condition involved an adaptive procedure, such that difficulty (i.e., number of stimuli, or level) increased as proficiency was achieved, whereas training in the control condition consisted of a static procedure in which the number of stimuli was always three. Experimental participants also received performance feedback tracking their progress. Both experimental and control participants received equal levels of intermittent feedback from monitoring research assistants based on their effort and focus throughout the session. Research assistants also monitored sessions to ensure continuous participant engagement with the computer program in both conditions. The use of a non-adaptive control condition held constant aspects of the cognitive training that may have been incidentally beneficial (e.g., interaction with computers/study staff, time at the treatment center, urine testing) and allowed for the strongest test of working memory challenge as the crucial aspect of training. This approach has been used successfully in previous studies (e.g., Bigorra, Garolera, Guijarro, & Hervas, 2016). For the experimental condition, performance on training sessions was indexed using measures of improvement for the three training exercises completed at every session.

Data analysis

Data were analyzed with IBM SPSS Statistics (IBM Corporation, Armonk, NY), GraphPad Prism (GraphPad Software, Inc., La Jolla, CA), and Stata (StataCorp LP, College Station, TX).

We conducted independent-samples t-test and chi-square analyses comparing the experimental and control groups at pre-assessment. We screened for systematic differences in these variables between completers and non-completers using independent-samples t-tests.

In the experimental group, learning on the working memory training program was indexed using a longitudinal linear model fit of the highest number of items achieved (e.g., number of items in a sequence to be reproduced by the participant) at each session, with session number as the covariate, and adjusted for highest level attained at baseline for each exercise. The performance variable learning slope was calculated as the average of the learning slopes from the three training exercises completed at every session.

For all outcome variables except urinalysis, a mixed-effects regression model with a random intercept and terms for Group (where β Group represents differences between experimental group and control group at pre-assessment), Time (pre- versus post-training; where β Time represents change over time for the control group), and Group by Time interaction (where β Group by Time represents the difference in change over time between experimental and control groups) was used to determine any effects of training as a function of group. The PSAT, Stroop task, and visuospatial working memory outcomes were analyzed using the same mixed-effects regression models with an additional term and interactions for trial block (PSAT = 3.2 s, 2.8 s, 2.4 s, 2.0 s; Stroop: classic color-word interference, string, cannabis word, neutral word; visuo-spatial working memory: three, four, six dots). To assess color-word interference in the Stroop task, we conducted a repeated-measures ANOVA with within-subjects factors of trial block (i.e., color-word interference block versus string block) and time (pre- versus post-training), and between-subjects factor of group (experimental versus control). We assessed attentional bias on the Stroop using a separate repeated-measures ANOVA with within-subjects factors of trial block (i.e., cannabis word block versus neutral word block) and time, and between-subjects factor of group.

One participant in the experimental group was excluded from the evaluation of total cannabis use because at pre-assessment he could not provide an estimate of how much cannabis he used for several days on the TLFB, but could recall using cannabis on those days (thus he is included in the analysis for days of cannabis use). One participant in the experimental group was excluded from the data analyses for the Stroop task because of color blindness.

We assessed change in odds of THC-positive (binary: THC positive versus THC negative) urine samples over time using a GEE model with an exchangeable correlation matrix and terms for predictors of Group (experimental versus control), Days (days of successive urine samples), and a Group by Days interaction that controlled for baseline urinalysis status.

Results

Participant characteristics

Thirty-seven participants completed the study, N = 20 in the control group and N = 17 in the experimental group. Participant demographic information for completers is displayed in Table 1 for both groups. There were no significant group differences in age, sex, race, ethnicity, working memory index, or ADHD symptoms at baseline. We further compared groups according to years of education, years of parental education, history of head trauma, sleep problems, time to fall asleep, average sleep duration, and whether participants were court-ordered to attend treatment and observed no significant differences (all ps ≥ .17) except mother’s years of education, which was significantly greater in the experimental group (t(28) = 2.50, p = .02). We compared completers (i.e., completed post-assessment) to non-completers (where data were available for non-completers) using the same demographic variables examined across groups and there were no significant differences at the time of consent or pre-assessment (all ps ≥ .07).

Table 1.

Basic Demographic Information for Experimental (N = 17) and control (N = 20) groups.

	Control		Experimental		Total
Variable	M (SD)	N (%)	M (SD)	N (%)	M (SD)	N (%)
Age in years	16.0 (1.6)		16.4 (1.7)		16.2 (1.6)
Years of education	9.4 (1.6)		9.7 (1.6)		9.5 (1.6)
Years of parental education
Mothera	11.7 (2.3)		14.3 (3.3)		13.0 (3.1)
Fatherb	11.6 (3.7)		12.1 (2.4)		11.8 (3.2)
Sex
Male		18 (90.0%)		13 (76.5%)		31 (83.8%)
Female		2 (10.0%)		4 (23.5%)		6 (16.2%)
Race
Black/African-American		11 (55.0%)		13 (76.5%)		24 (64.9%)
White		6 (30.0%)		2 (11.8%)		8 (21.6%)
More than one race		3 (15.0%)		2 (11.8%)		2 (13.5%)
Ethnicity
Non-Hispanic		18 (90.0%)		16 (94.1%)		34 (91.9%)
Hispanic		2 (10.0%)		1 (5.9%)		3 (8.1%)
History of head trauma
Yes		9 (45.0%)		7 (41.2%)		16 (43.2%)
No		11 (55.0%)		10 (58.8%)		21 (56.8%)
Court-ordered to attend treatment
Yes		6 (30.0%)		3 (17.6%)		9 (24.3%)
No		14 (70.0%)		14 (82.4%)		28 (75.7%)
Hours of sleep per night	7.1 (1.5)		6.4 (2.0)		6.8 (1.8)
Sleep problems
Yes		9 (45.0%)		9 (52.9%)		18 (48.6%)
No		11 (55.0%)		8 (47.1%)		19 (51.4%)

Note. Demographic information for participants who completed the study in terms of mean (M) and standard deviation (SD) or number of participants (N) and percentage (%) for the control group (N = 20), experimental group (N = 17), and total (N = 37). Bolded values indicate that there was a significant difference (p < .05) between the experimental group and control group.

^aFive participants in the control group and two participants in the experimental group did not know their mothers’ level of education.

^bEight participants in the control group and nine in the experimental group did not know their fathers’ level of education.

Training data

Analysis of experimental group data from the cognitive training program showed a positive learning slope across all three exercises (visual data link: M = .02, SD = .05; input module: M = .03, SD = .05; input module with lid: M = .03, SD = .06; where M = mean slope), which is consistent with an overall average training improvement of between .5 and .75 items in working memory across 25 training sessions. In support of the upper limit of three stimuli per trial in the non-adaptive control condition as an inactive working memory control condition, all participants in the experimental group correctly completed trials with four or more stimuli in at least one of the training exercises on the very first session of training.

Assessment outcomes

Table 2 displays the fitted means and standard errors from the regression model for each group at pre-assessment and post-assessment.

Table 2.

Pre- and Post-Assessment Outcomes (Fitted Means and Standard Error) for Experimental (N = 17) and Control (N = 20) Groups.

	Control		Experimental
Measure	Pre	Post	Pre	Post
WAIS-IV/ WISC-IV
Working memory index	84.05 (2.60)	90.55 (2.60)	89.06 (2.82)	95.88 (2.82)
Working memory index percentile	21.28 (4.79)	31.55 (4.79)	26.65 (5.19)	39.88 (5.19)
Digit span forward (raw score)	5.80 (.24)	6.00 (.24)	6.18 (.26)	6.41 (.26)
Digit span backward (raw score)	3.55 (.23)	4.70 (.23)	3.82 (.25)	4.76 (.25)
Digit span (scaled score)	7.25 (.56)	8.55 (.56)	8.41 (.61)	9.82 (.61)
Visuospatial working memory task
Proportion correct
3 dots	.90 (.05)	.91 (.05)	.87 (.05)	.88 (.05)
4 dots	.78 (.05)	.76 (.05)	.75 (.05)	.76 (.05)
6 dots	.32 (.05)	.34 (.05)	.26 (.05)	.42 (.05)
Paced Serial Addition Test (PSAT)
Proportion Correct
3.2s	.31 (.04)	.40 (.04)	.42 (.04)	.50 (.04)
2.8s	.30 (.04)	.31 (.04)	.40 (.04)	.41 (.04)
2.4s	.27 (.04)	.24 (.04)	.33 (.04)	.34 (.04)
2.0s	.22 (.04)	.18 (.04)	.30 (.04)	.23 (.04)
Proportion dyads
3.2s	.09 (.03)	.19 (.03)	.17 (.03)	.32 (.03)
2.8s	.09 (.03)	.10 (.03)	.15 (.03)	.22 (.03)
2.4s	.07 (.03)	.06 (.03)	.09 (.03)	.16 (.03)
2.0s	.03 (.03)	.04 (.03)	.08 (.03)	.10 (.03)
Reaction time on correct trials
3.2s	1789.04 (54.87)	1663.47 (54.87)	1656.07 (59.52)	1636.03 (59.52)
2.8s	1461.05 (54.87)	1364.30 (54.87)	1443.56 (59.52)	1372.73 (59.52)
2.4s	1365.52 (54.87)	1203.66 (54.87)	1284.94 (59.52)	1212.57 (59.52)
2.0s	1026.54 (54.87)	907.71 (54.87)	1128.72 (59.52)	918.28 (59.52)
Continuous performance test
Hit rate	.95 (.02)	.87 (.02)	.94 (.03)	.91 (.03)
False alarms	.62 (.05)	.70 (.05)	.64 (.05)	.62 (.05)
d′	1.53 (.26)	.80 (.26)*	1.36 (.28)	1.29 (.28)*
Response bias	–1.15 (.08)	–.99 (.08)	–1.10 (.09)	–1.02 (.09)
Reaction time on correct trials	411.00 (22.25)	361.59 (22.25)*	376.96 (24.13)	383.96 (24.13)*
Stroop
Proportion correct
Classic color-word	.99 (.03)	.95 (.03)	.98 (.03)	.94 (.03)
String	.98 (.03)	.94 (.03)	.98 (.03)	.95 (.03)
Cannabis	.99 (.03)	.95 (.03)	.99 (.03)	.93 (.03)
Neutral word	.99 (.03)	.94 (.03)	.99 (.03)	.92 (.03)
Reaction time on correct trials
Classic color-word	1058.16 (41.38)	1051.16 (41.38)	1057.25 (46.26)	1054.08 (46.26)
String	971.25 (41.38)	992.71 (41.38)	986.96 (46.26)	975.23 (46.26)
Cannabis	1118.94 (41.38)	1091.24 (41.38)	1104.37 (46.26)	1020.51 (46.26)
Neutral word	1002.89 (41.38)	1035.68 (41.38)	941.67 (46.26)	1027.08 (46.26)
Quick discounting operant task
Area under the curve	.73 (.05)	.82 (.05)	.74 (.06)	.83 (.06)
Past 30 days timeline follow-back
Cannabis
Days of cannabis use	8.25 (2.13)	5.10 (2.13)	12.53 (2.31)	6.12 (2.31)
Total blunts/joints smoked	45.70 (25.34)	15.68 (25.34)	140.74 (28.31)	31.29 (27.49)
Tobacco
Days of cigarette use	18.40 (3.03)	15.90 (3.03)	19.76 (3.29)	18.06 (3.29)
Total cigarettes smoked	124.01 (27.95)	98.35 (27.95)	82.21 (30.31)	69.53 (30.31)
Alcohol
Days of alcohol use	.45 (.25)	.20 (0.25)	.24 (.28)	.94 (.28)
Total alcoholic drinks consumed	4.30 (1.67)	.91 (1.67)	2.28 (1.81)	3.85 (1.81)
GAIN-I Lite
Days bothered by psychological problems	4.80 (1.78)	3.80 (1.78)	3.12 (1.93)	3.29 (1.93)
Days impaired by psychological problems	2.80 (1.35)	2.00 (1.35)	3.00 (1.46)	1.76 (1.46)
Days with behavioral control problems	9.15 (2.41)	3.85 (2.41)	12.41 (2.62)	11.06 (2.62)
Days of criminal activity (Besides drug use)	3.70 (1.29)	1.25 (1.29)	1.65 (1.40)	2.76 (1.40)
Instances of unprotected sex	.90 (.60)	.40 (.60)	2.06 (.65)	.35 (.65)
Proportion of sex that was protected	.76 (.09)	.70 (.11)	.70 (.11)	.86 (.13)
Difficulties in Emotion Regulation Scale	96.45 (5.02)	87.95 (5.02)*	87.65 (5.44)	89.29 (5.44)*
Gray Oral Reading Test
Fluency	6.95 (.55)	7.05 (.55)	8.00 (.60)	8.41 (.60)
Comprehension	6.65 (.45)	6.75 (.45)	7.53 (.49)	7.94 (.49)
Therapeutic alliance	37.56 (2.09)	40.10 (2.05)	42.24 (2.22)	40.87 (2.28)

Note. Fitted means and standard error (in parentheses) from mixed-effects regression model described in text for training effects on outcomes. Values are listed for pre-training assessment (Pre) and post-training assessment (Post) for both the control group (N = 20) and experimental group (N = 17). One participant in the experimental group was excluded from the evaluation of total cannabis use because at pre-assessment he could not provide an estimate of how much cannabis he used for several days on the timeline follow-back, but could recall using cannabis on those days (thus he is included in the analysis for days of cannabis use). One participant in the experimental group was excluded from the data analyses for the Stroop task because of color blindness. Proportion of protected sex is reported only for those participants who indicated sexual activity during the past 30 days. Bolded values indicate that there was a significant difference (p < .05) between the experimental group and control group at Time 1 for that outcome. An asterisk (*) indicates that there was a significant difference (p < .05) in change over time as a function of group (experimental or control) for that outcome.

WAIS-IV = Wechsler Adult Intelligence Scale IV, WISC-IV = Wechsler Intelligence Scale for Children IV, GAIN-I Lite = Global Appraisal of Individual Needs-Initial Lite.

Cognitive outcomes

Cognitive outcomes include the WISC-IV/WAIS-IV (i.e., working memory index, digit span), the visuospatial working memory task, the PSAT, continuous performance task, Stroop task, and delay discounting. There were no significant differences between groups at pre-assessment or differences in change over time as a function of group for any cognitive outcome except the PSAT and the continuous performance task. During pre-assessment, the experimental group showed significantly slower reaction time for correct trials during the 2.0-s trial block of the PSAT, and significantly greater proportion correct on the 3.2-s trial block. There were no other significant differences in cognitive outcomes between the experimental group and control group at pre-assessment, and any pre-assessment difference is adjusted for within the regression model when examining differences in change over time. On the continuous performance task there was a significant difference in change over time for the experimental and control groups for d′ (β Group by Time = .66, SE =.26, p =.02) and mean reaction time on correct trials (β Group by Time = 56.40, SE = 23.52, p =.02). These significant differences in change over time as a function of group reflect relative stability in the experimental group on these outcomes (see Table 2), but a significant decrease in these outcomes in the control group (d′: β Time = −.73, SE = .18, p < .001; mean reaction time on correct trials: β Time = −49.41, SE = 15.94, p = .002). There were no other significant differences in change over time between the experimental and control groups for cognitive outcomes, but the results of the manipulation check for the Stroop task suggest significant color-word interference and attentional bias for cannabis words (described later).

Color-word interference and attentional bias for cannabis words

In the Stroop task, we observed a significant color-word interference effect such that reaction times on correct trials were significantly slower during the color-word interference block than reaction times on correct trials during the string block (i.e., main effect of trial block; F(1, 34) = 16.60, p < .001). We did not observe a significant main effect of time, group, a time by group interaction, a block by group interaction, or a time by block by group interaction (all ps ≥ .39). We observed significant attentional bias for cannabis-related words such that reaction times on correct trials were significantly slower during the cannabis word block relative to reaction times on the neutral word block (i.e., main effect of trial block; F(1, 34) = 23.15, p < .001). We observed a significant session by block interaction, reflecting that reaction times for the two blocks were more similar at post-training relative to pre-training (see Table 2; F(1, 34) = 9.01, p = .005). We did not observe any other significant main effects or interactions (all ps ≥ .16).

Substance use outcomes

Substance use outcomes are reported using self-reported substance use as assessed by TLFB and as assessed via urinalysis.

Self-reported substance use

The experimental group reported using significantly more cannabis in the 30 days prior to pre-assessment (β Group = 95.04 blunts/joints, SE = 38.00, p = .02) than the control group. Both the experimental and the control groups showed decreased self-reported cannabis use at post-training relative to pre-training, but the difference in slopes between the groups was not significant (β Group by Time = −79.42, SE = 49.20, p = .10) and the groups were statistically different at pre-training. There were no other significant effects for cannabis (i.e., days of use), alcohol, or tobacco (i.e., amount or days of use).

Urinalysis

There was a significant increase in the odds of a THC-positive urine across days for the control group (OR = 1.01, CI = 1.00–1.02, p = .04; CI borders 1.0 due to rounding), but the odds of a THC-positive urine did not increase for the experimental group (OR = 1.00, CI = 0.99–1.02, p = .36). There was a significant difference in the change in odds of a THC-positive urine across days as a function of group, such that the experimental group had significantly less of an increase in odds of a THC-positive urine sample over time than the control group (i.e., Group by Days interaction; OR = 0.99, CI = 0.97–1.00, p = .03). Figure 2 shows the fitted probability for THC-positive urines over time for both groups.

Figure 2.

Fitted probability for THC-positive urines over time for both groups using GEE model described in text.

Secondary functional outcomes

There was a significant difference in change over time for the experimental and control groups for difficulties in emotion regulation (β Group by Time = 10.15, SE = 3.69, p = .006). This difference in change over time reflects a significant decrease in difficulties in emotion regulation in the control group (β Time = −8.50, SE = 2.50, p < .001). There were no other significant effects for secondary functional outcomes, including risk behavior and functional impairment, reading, or therapeutic alliance.

Discussion

There is a need to identify effective treatments to improve outcomes for adolescents with cannabis use disorder. Despite the potential promise of working memory training as an adjunctive therapy in substance use populations and children with ADHD, no previous studies had examined the effectiveness of adjunctive working memory training in adolescents with cannabis use disorders. The present controlled trial examined potential effects of working memory training in adolescents in treatment for cannabis use disorder by randomizing individuals to active working memory training or a control condition. We assessed a broad range of outcomes including those closely related to the memory training intervention (e.g., digit span) as well as those more generally related to drug use (e.g., self-report, urinalysis) and psychosocial outcomes. Aside from a few measures, there were no significant differences in pre- and post-assessment performance as a function of group, suggesting little benefit from working memory training. Still, urinalysis data suggest the experimental group was less likely to test positively for THC over time relative to the control group. After describing key outcomes, we discuss similar studies and possible reasons we were unable to obtain robust effects of working memory training in the present study.

Overall, we did not observe robust differences in outcomes as a result of working memory training. There were no significant differences in change over time for any of the primary working memory outcomes. Regarding secondary cognitive and functional outcomes, the continuous performance task and Difficulties in Emotion Regulation Scale showed significant differences between groups across time, but these differences are not clearly attributable to the training. In the continuous performance task, measuring sustained attention and response inhibition, the control group showed significant decreases in both mean reaction time on correct trials and sensitivity between distinguishing between targets and nontargets (d′). Significantly less sensitivity reflected by a decrease d′ in the control group, but not the experimental group, may be interpreted as a beneficial effect of training such that the non-trained group worsens but the trained group maintains performance over time. However, the decrease in sensitivity was accompanied by a decrease in mean reaction time on correct trials (i.e., a speed-accuracy trade-off), thus the potential role of training on response inhibition is ambiguous. Similarly, the findings regarding emotion regulation do not indicate a clear effect of training. The significant decrease in difficulties in emotion regulation was detected in the control group but not the experimental group. Self-reported cannabis use decreased more in the experimental group relative to the control group, but this difference was not significant and the groups significantly differed prior to training. On the contrary, urinalysis data suggest a possible benefit to training. The odds of a drug-positive urine in the experimental group were roughly stable, but the control group showed an increase in the odds of drug-positive urine over time (see Figure 2). This suggests that drug use in the experimental group remained stable but increased in the control group. This finding is consistent with a study testing working memory training in methadone patients, which observed that the likelihood of a THC-positive urine in the control group increased over time, whereas the likelihood of THC-positive urines decreased in the experimental group (Rass et al., 2015). In spite of the similarity of urinalysis data to the findings of Rass and colleagues (2015), it is possible that in the present study some spurious group differences or differences in change over time may have arisen given the number of outcomes assessed, especially because corrections for multiple comparisons were not undertaken in the present trial. This analytic approach was selected in order to be maximally sensitive to the potential effects of training within the smaller sample of completers. Still, the potential for a spurious urinalysis result should be carefully considered given the pattern of findings. Based on prior work, training effects should be evident from performance on tasks most similar to the trained tasks (e.g., digit span), whereas performance on tasks dissimilar to the trained tasks (e.g., sustained attention on the continuous performance tasks, emotion regulation, and drug use) should be less likely to change as a result of the training (Rass et al., 2015). Therefore, although the urinalysis results suggest a possible benefit to training, the present data do not provide strong evidence that working memory improvement is the mechanism of this benefit.

The present results are not unique in the failure to find significant effects of working memory training on cognitive outcomes, but observing differences suggestive of a change in drug use or other more conceptually distant outcomes. For example, in a study with stimulant users, Bickel and colleagues (2011) reported no improvement in working memory or other cognitive outcomes in stimulant users following training, but did observe decreased delay discounting. Houben and colleagues (2011) found significant improvement using trained tasks in problem drinkers, but did not examine whether the effects of training generalized to non-trained cognitive tasks. In terms of drug use, Houben and colleagues observed significantly greater reductions in drinking in problem drinkers following working memory training in the experimental group relative to the control group. Rass and colleagues (2015) found a significant effect of working memory training in methadone maintenance patients on some working memory tasks that were similar to training (i.e., digit span backward, visuospatial working memory, although no effect on digit span forward or n-back) and benefits of training on drug use, but not on cognitive tasks that were dissimilar to training (e.g., metamemory, psychomotor speed and attention, reasoning, delay discounting). Urinalysis data of the present study are consistent with possible benefits of working memory training on drug use outcomes. On the other hand, the absence of a clear explanatory mechanism for differences in drug-positive urines as a function of training, combined with equivocal generalization of training benefits to dissimilar tasks in previous studies, suggests that we do not fully understand if or how working memory training may affect cognition or drug use in individuals with substance use disorders. That is, if the mechanism of training benefits is improvement in working memory, it should be the case that tasks similar to training show effects that are greater than or comparable to improvements on dissimilar tasks and drug use outcomes, but this has not consistently been observed in studies in substance use populations to date. The identification of mechanisms underlying working memory training benefits, if such benefits exist, will be important for designing interventions and training components that are effective in improving substance use outcomes.

A number of differences between the present study and previous implementations of working memory training may contribute to the lack of robust training effects observed here. Samples in previous studies implementing working memory training in substance use populations, including the present study, have varied according to primary drugs of abuse, age, education, and possibly socioeconomic status or race (although detailed demographic information is not always described; Bickel et al., 2011; Houben et al., 2011; Rass et al., 2015). Such sample and treatment population differences may account for some of the difficulty in detecting benefits of training in the present study relative to previous work. For example, it may be the case that ongoing cannabis use interferes with the benefits of cognitive training to a greater extent than other substances. Future research implementing working memory training in adolescent cannabis users may take additional steps to promote abstinence to maximize the benefits of training, such as providing additional reinforcers contingent on drug abstinence rather than attendance only.

Research observing positive effects of working memory training in children with ADHD may also provide useful comparisons with the present study (Cortese et al., 2015; Peijnenborgh et al., 2016; Shinaver et al., 2014; Spencer-Smith & Klingberg, 2015; cf. Melby-Lervåg & Hulme, 2013; Shipstead, Redick, & Engle, 2012). For example, a recent randomized controlled trial of working memory training in children with ADHD observed significant improvement in working memory, broader executive functioning, and ADHD symptoms both after training and at long-term follow-up (Bigorra et al., 2016). In addition to examining the effects of training in a younger sample (including participants between 7 and 12 years of age), Bigorra and colleagues conducted training sessions for five sessions per week over 35 days (i.e., five weeks) for a total of 25 sessions. Due to difficulties with regular participant attendance to treatment at Mountain Manor Treatment Center, and consequently attendance at training sessions, participants in the present study completed an average of 19.9 training sessions across an average of 75.9 days (i.e., approximately 1.8 sessions per week across approximately 10 weeks).¹ In addition, Bigorra and colleagues excluded participants who had a history of affective or anxiety disorders, learning disorder, and those who had previously received psychological or pharmacological treatment for ADHD, whereas the present study did not exclude participants for these reasons if the affective or anxiety disorders were being appropriately treated. As in the present study, Bigorra and colleagues used a non-adaptive training as the control condition against which improvement in the experimental training group was judged. Still, it could be that the non-adaptive training condition in the present study retained some benefits of the active training, making differences between groups more difficult to detect. Finally, the present study reported a relatively smaller sample size of individuals who completed the study (N = 37) relative to Bigorra and colleagues (N = 55). Thus, the possibility that the present study may have been insufficiently powered to observe smaller magnitude effects on cognitive or substance use outcomes warrants consideration. However, we did observe a significant color-word interference effect and an attentional bias for cannabis-related words on the Stroop task, similar to previous research (Field, 2005; Stroop, 1935). Thus, the present study appears sufficiently powered to detect previously demonstrated effects, although a larger sample size might be required to detect a longitudinal change in working memory with group differences than an overall demonstration of attentional bias.

Despite demonstrating sufficient power to detect anticipated differences in the color-word interference and attentional bias manipulation checks, substantial attrition following randomization remains a limitation of the present study. It should be noted that we found no evidence that completers were significantly different from non-completers in terms of demographic characteristics or baseline working memory; ideally, all randomized participants would have been included in post-assessment regardless of engagement with training using an intent-to-treat approach. The most common reason for attrition in the present study was not non-adherence to the working memory training, per se, but failure to adhere to ongoing substance use treatment, which indirectly led to attrition from the study, which took place at the same location as treatment (see Figure 1). In short, participants who discontinued the working memory training intervention were generally lost to post-assessment because they were lost to ongoing substance use treatment. Furthermore, a non-negligible percentage of those who did not complete the intervention (12% of participants who dropped out of study; 7% of randomized participants) declined further participation with the study but remained in treatment, suggesting repeated memory training may constitute an additional burden. Even so, post-assessment attrition would be of greater concern if we observed positive results of training. That is, positive effects of training observed only in individuals who completed at least 10 training sessions might be explained by a post-assessment sample biased in favor of those with good treatment adherence, but a null result cannot be explained in this way. We observed null results even in those participants who represent the best chance for positive outcomes within the present sample. Thus, it would be unlikely we would observe positive results if our post-assessment analysis also included those with fewer training sessions as in a true intent-to-treat analysis.

Differences in implementation and final sample size could be expressed as shortcomings of the present study, but they may also speak to the feasibility of working memory training as an adjunctive treatment, and the lack of robustness and generalizability of previously observed effects across other clinical populations or outcomes (e.g., Hulme & Melby-Lervåg, 2012; Kable et al., 2017; Melby-Lervåg & Hulme, 2013; Melby-Lervåg, Redick, & Hulme, 2016; Redick et al., 2015; Shipstead et al., 2012). For example, a recent study with more than 100 young adult participants (ages 18 to 35) found no significant effects of 10 weeks of commercial working memory training on neural activity or decision making (Kable et al., 2017). Thus, the present study adds to a growing literature that suggests that despite initially promising research, working memory training effects may be weaker than anticipated. Adolescent substance use treatment populations bring the additional challenges of psychiatric comorbidity, diversity in demographic characteristics, as well as difficulties with retention and attendance. Unless working memory interventions can be implemented in ways that can overcome the aforementioned challenges, such trainings may not be viable for widespread application.