Abstract
Adolescent marijuana use is associated with neurocognitive impairment, but further work is needed to assess the relationship between treatment-associated abstinence and cognitive performance. This secondary analysis, conducted in the context of a marijuana cessation pharmacotherapy trial in adolescents, examined cognitive performance at baseline and at two time points during treatment using the CNS Vital Signs® assessment battery. Abstinence from marijuana, relative to continued use, as assessed via urine cannabinoid testing, was associated with significant improvement in composite memory (p<0.001), verbal memory (the most impacted component of composite memory) (p<0.001), and psychomotor performance (p=0.045) scores. These findings suggests that some domains of cognitive performance improve significantly even in the early stages of treatment-associated abstinence.
Keywords: cognitive performance, cognition, memory, marijuana, youth, adolescent
1.0 Introduction
Marijuana is the most widely used illicit substance among adolescents. In 2013, Monitoring the Future data indicated daily marijuana use in 1.1% of 8th graders, 4.0% of 10th graders, and 6.5% of 12th graders (Johnston, O'Malley, Bachman, & Schulenberg, 2014). There is growing evidence that heavy marijuana use during adolescence, a time of dynamic brain development, may impact cognition (Randolph, et al., 2013). In adults with persistent marijuana use that started during adolescence, Meier, et al. found a decline in intelligence quotient (IQ), with impairments evident in executive functioning and processing speed (Meier, et al., 2012).
Cognitive performance is multifaceted, and results of studies of marijuana's effects on cognitive performance are mixed. There appear to be certain neuropsychological constructs or domains that are influenced by marijuana use. A recent review of the relevant literature by Randolph and colleagues in 2013 concluded attention, processing speed, verbal declarative memory, and cognitive control are affected by heavy marijuana use in adolescents. In contrast, language, visual declarative memory, perceptual reasoning, inhibition, and planning did not appear to be consistently affected by marijuana (Randolph, et al., 2013).
The literature in this area may often appear contradictory. For example, attention is complex and can be divided into subcategories (complex attention, sustained attention, etc.); some studies have very small sample sizes and methodologies as well as populations studied often differ drastically. Additionally, tests used to examine the same domain or construct can be different across studies. Abdullaev and colleagues found that on tests requiring executive attention (attention required when conflicting information is presented), adolescents who used marijuana performed worse than controls (Abdullaev, et al., 2010). Hanson, et al., concluded that while impairments in verbal memory among cannabis users improve to the level of controls within 3 weeks of abstinence, deficits in attention remain within this same time frame (specifically accuracy in tasks that require attention) (Hanson, et al., 2010). Fried and colleagues did not find a significant difference in tests of attention among groups of marijuana users (heavy and light) and controls whereas they did find significant differences in other domains (overall IQ, processing speed, immediate memory, and delayed memory). Interestingly, the negative impact of marijuana on the cognitive domains that were affected resolved at 3 months of sustained abstinence. The sample in this study consisted of individuals exposed to drugs in utero and so must be interpreted with that in mind (Fried, et al., 2005).
The purpose of the current study was to evaluate potential changes in cognitive task performance among adolescents enrolled in a randomized, placebo-controlled trial of N-acetylcysteine (NAC) added to brief weekly cessation counseling and contingency management for marijuana cessation (Gray et al., 2012). To our knowledge, there are no previous studies that examine cognitive performance within the framework of a placebo-controlled pharmacotherapy treatment trial for cannabis dependence in adolescents. We hypothesized that cognitive performance would improve with marijuana cessation, and that longer periods of abstinence would predict greater improvements in cognitive performance. Participants were cannabis-dependent upon study enrollment, allowing for assessment of possible improvements in cognition with abstinence among a group of relatively heavy marijuana users seeking treatment.
2.0 Materials and methods
2.1 Participants
Participants were 78 treatment-seeking adolescents, ages 15-21, who met DSM-IV criteria for cannabis dependence, enrolled in the parent trial, and completed a baseline cognitive task performance battery at the treatment initiation visit and at least one additional time point (4 and/or 8 weeks after treatment initiation). Participants ages 18 and above provided informed consent. For participants under age 18, the legal guardian provided informed consent and participant provided assent. The university institutional review board approved all procedures for the parent study. All study procedures were performed at the Medical University of South Carolina in Charleston, South Carolina.
2.2 Measurements
CNS Vital Signs (cnsvs.com) is a computer-administered battery of performance tests used in the study to assess cognitive performance at baseline, 4 weeks, and 8 weeks. CNS Vital Signs measures certain clinical domains, including composite memory, verbal memory, visual memory, processing speed, executive function, psychomotor speed, reaction time, complex attention and cognitive flexibility. Tests used to calculate domains include the verbal memory test (identifying words previously presented), visual memory test (identifying symbols or shapes previously presented), finger tapping test, the symbol digit coding test, the Stroop test, the shifting attention test, and the continuous performance test (Gualtieri & Johnson, 2006). CNS Vital Signs has been used to examine the effect of substances on cognitive performance (D.W. Loring, et al., 2012).
Abstinence data was obtained from urine cannabinoid tests (UCT) that occurred twice weekly during the study. Marijuana use at each treatment visit was categorized as not abstinent (NA), recently abstinent (RA), or consistently abstinent (CA). Participants were deemed NA if the urine cannabinoid test was positive (i.e., ≥50 ng/mL) at that visit. Participants were deemed RA if their urine cannabinoid test was negative (i.e., <50 ng/mL) at the visit that the cognitive assessment took place but had been positive at least once between the cognitive assessments. Participants were deemed CA if all urine cannabinoid tests were negative since the last cognitive assessment. Abstinence was grouped this way in light of the secondary nature of this analysis. The study was not sufficiently powered to detect smaller differences in performance that may or may not be present between participants abstinent for one week versus those abstinent for two or three weeks. We thus grouped those individuals together.
Further details of the parent trial are discussed elsewhere (Gray et al., 2012; Roten, et al., 2013).
2.3 Data Processing and Statistical Analysis
The study hypothesis was that increasing lengths of abstinence from marijuana would correlate with improved cognitive performance as measured by CNS Vital Signs (cnsvs.com). Validity of responses to the various components of the CNS Vital Signs was assessed through criteria defined by the CNS Vital Signs Interpretation guide (https://www.cnsvs.com/WhitePapers/CNSVS-InterpretationGuide.pdf). Invalid CNSVS scores and measures that were dependent on invalid scores were marked as such and not included in the analysis models. Since the primary aim of the study was to determine the association between abstinence (via UCT) duration and cognitive performance scores, sporadic missing data from the urine cannabinoid test was not noted as a failed screen (as was assumed in the primary study analysis). Individual self-reported use was examined in conjunction with both the UCT from the visit previous and the visit following the missing visit. If it was determined that abstinence was likely maintained, missing data was noted as such. If abstinence could not be confirmed or multiple consecutive UCT visits were missed, it was assumed the participant would have had a positive urine cannabinoid test.
Prior to analysis, standard descriptive statistics were used to summarize the demographic and clinical characteristics of the cohort. Demographic, clinical, and marijuana use characteristics were examined for univariate predictive relationships with cognitive response outcome as well as possible confounding effects with marijuana abstinence. Marijuana use characteristics included craving which was assessed at baseline with the 12-item, short form of the Marijuana Craving Questionnaire (MCQ). This measurement has been shown to be reliable and valid (Heishman et al., 2009). In the primary analysis model, the effect of abstinence from marijuana on cognitive outcome measures was assessed simultaneously at the 4- and 8-week treatment visits using mixed effect regression models. Group level means were constructed using model based estimates and standard errors. Restricted maximum likelihood (REML) methods were used to estimate fixed effects and variance components in the presence of imbalanced data. Initial models contained abstinence duration (NA, RA, CA), visit, the interaction of abstinence and visit number, and baseline measures of cognitive variables. When the interaction of abstinence and visit number is insignificant, a time naïve cluster analysis of group means was performed. Secondarily, it was also of interest to investigate the effect of abstinence over the entire 8-week treatment period on cognitive outcomes. (For this, the NA group meant positive cannabinoid test at week 8, RA meant negative urine cannabinoid test at week 8 but a positive urine cannabinoid test at some point during treatment, and CA meant a negative urine cannabinoid test at week 8 and throughout treatment). As the primary aim of the parent study was to estimate the effects of NAC on abstinence from marijuana use, all models were additionally adjusted for treatment group assignment. The normality and homoscedasticity of the residuals were checked using graphical techniques and when violations of assumptions were found, outcome measures were appropriately transformed.
Model based statistical results are shown as means and associated standard errors unless otherwise noted. All statistical analyses were conducted using SAS version 9.3 (SAS Institute, 2011). Significance for all planned comparisons was set at a 2-sided p-value of 0.05.
3.0 Results
Baseline demographic and clinical characteristics were examined for the cohort as well as across week 4 abstinence categories (Table 1). At baseline, the mean age of the study cohort was 18.8±1.5 years; 52 (66.7%) of the cohort were male and 71 (91.0%) were Caucasian. There were no significant differences in age, gender or race between those who attained abstinence at week 4 and those who were not abstinent. Participants who were enrolled in school were more likely to attain abstinence at week 4 than those not enrolled (p=0.022). None of the measured demographic or clinical characteristics, including marijuana use duration, were significantly associated with composite memory outcomes during the treatment phase of the study. Study analysis models were initially constructed to account for marijuana abstinence grouping, visit number, treatment group, baseline cognitive measure score, and the interaction between abstinence and visit number. Additional models adjusted for marijuana use duration and school enrollment status. For the assessment of normality and homoscedasticity, all models outcomes, with the exception of the complex attention score, performed well under the model assumptions and no transformations were needed. Complex attention scores were transformed using the natural logarithm and performed well after transformation.
Table 1.
Baseline Characteristics | Overall N=78 | Week 4 Abstinence Status |
P Value | ||
---|---|---|---|---|---|
Not Abstinent N=34 | Recently Abstinent N=22 | Sustained Abstinent N=22 | |||
Demographics | |||||
Age (yrs) | 18.8 ± 1.5 | 18.9 ± 1.4 | 19.0 ± 1.6 | 18.5 ± 1.7 | 0.830 |
Male % (n) | 66.7 (52) | 67.7 (23) | 68.2 (15) | 63.6 (14) | 0.938 |
Caucasian % (n) | 91.0 (71) | 97.1 (33) | 81.8 (18) | 90.9 (20) | 0.150 |
Enrolled In School % (n) | 82.1 (64) | 67.7 (23) | 95.5 (21) | 90.9 (20) | 0.022 |
Weight (lbs) | 149.3 ± 26.1 | 146.3 ± 27.8 | 149.8 ± 24.1 | 153.1± 26.1 | 0.389 |
Smokes Cigarettes % (n) | 53.9 (42) | 52.9 (18) | 54.6 (12) | 54.6 (12) | 0.990 |
Years Smoking Cigs | 1.4 ± 2.0 | 1.6 ± 2.2 | 1.5 ± 2.5 | 1.0 ± 1.1 | 0.833 |
NAC Trt Assignment % (n) | 51.3 (40) | 41.2 (14) | 40.9 (13) | 40.9 (13) | 0.292 |
Marijuana Use Characteristics | |||||
Marijuana Use Duration (yrs) | 4.0 ± 1.8 | 4.4 ± 1.7 | 4.1 ± 1.8 | 3.3 ± 1.7 | 0.052 |
Marijuana quit attempts | 1.8 ± 2.2 | 1.8 ± 1.9 | 2.2 ± 3.1 | 1.4 ± 1.3 | 0.748 |
MCQ Total Score | 48.0 ± 12.2 | 47.2 ± 12.0 | 50.5 ± 12.4 | 46.8 ± 12. | 0.489 |
MCQ: Compulsion | 7.6 ± 3.7 | 7.2 ± 3.4 | 8.0 ± 3.8 | 7.8 ± 4.1 | 0.816 |
MCQ: Emotionality | 12.4 ± 4.9 | 12.2 ± 4.9 | 13.9 ± 4.8 | 11.1 ± 4.8 | 0.206 |
MCQ: Expectancy | 14.4 ± 4.2 | 14.0 ± 4.3 | 14.6 ± 4.6 | 14.6 ± 3.7 | 0.803 |
MCQ: Purposefulness | 13.8 ± 3.9 | 14.0 ± 3.7 | 14.0 ± 4.1 | 13.3 ± 3.9 | 0.794 |
Abstinence from marijuana during the treatment phase of the study, as measured by the UCT, was significantly associated with increased composite memory scores (F2,110=7.68; p<0.001). Those who were consistently abstinent (CA) and those who were recently abstinent (RA) had significantly greater composite memory scores than those were not abstinent (diff(d)=7.2±2.1, p<0.001 and d=7.5±2.4, p=0.002, respectively; Figure 1a). The overall relationship between abstinence and composite memory did not differ between the visit at week 4 as compared to week 8 (Abstinence group x visit number interaction: F2,92.9=0.98; p=0.378) and thus, time naïve models will be presented. Composite memory scores are an amalgam of verbal and visual memory scores. Thus, individual scale scores were examined. Similar to the relationship between abstinence and composite memory, there was a significant relationship between abstinence from marijuana and verbal memory scores during the treatment phase of the study (F2,108=18.0; p<0.001). Those who were either consistently or recently abstinent had greater verbal memory scores than those were not abstinent (d=6.8±1.2, p<0.001 and d=6.0±1.4, p<0.001, respectively; Figure 1b). This pattern was not seen in the relationship between abstinence and visual memory scores (F2,114=0.92; p=0.401) with neither the consistently or recently abstinent scores significantly different from those there were not abstinent (d=0.4±1.2, p=0.756 and d=1.9±1.4, p=0.190, respectively; Figure 1c). Participants that were consistently abstinent from treatment initiation to the first cognitive measure treatment saw significant gains in verbal memory (Δ=2.9±1.4, p=0.048) and those that failed to attain any abstinence had significant decreases (Δ=-4.4±1.2, p<0.001). Participants that had recently attained abstinence had moderate, but insignificant increases in verbal memory scores (Δ=0.5±1.5, p=0.756). Interestingly, those with consistent abstinence did not have significantly different composite, verbal or visual memory scores than those that were recently abstinent (all p>0.28).
In addition to increased scores on memory components of the CNS Vital Signs, those with increased periods of abstinence from marijuana had moderate increases in psychomotor speed scores (F2,112=3.19; p=0.045). Subjects with consistent abstinence and those with recent abstinence had increased psychomotor speed scores as compared to those who were not abstinent at the cognitive performance visit (d=8.4±3.8, p=0.030 and d=9.0±4.1, p=0.030, respectively; Figure 2). Participants that attained any abstinence between treatment initiation to the first cognitive measure treatment saw significant gains in psychomotor speed (Δ=5.3±2.3, p=0.022) while those that failed to attain any abstinence had no significant change (Δ=-1.5±2.7, p=0.579). Duration of abstinence from marijuana was not significantly associated with processing speed, executive function, reaction time, complex attention, or cognitive flexibility in this population.
In the secondary analysis of abstinence of the full 8 week treatment period, the association between abstinence and verbal memory persisted (F2,61=7.44; p=0.002) with both CA and RA subjects with greater scores that those that were not abstinent (50.3±1.4 vs. 46.1±1.1, p=0.025 and 52.3±1.1 vs. 46.1±1.1, p<0.001, respectively). Similar to the primary analysis, the lack of association with visual memory remained (F2,59=0.42; p=0.657). Although the relationship between abstinence and psychomotor speed was in the same direction and magnitude, the association failed to maintain significance (F2,61=1.48; p=0.235). Although there were imbalances in baseline levels of both duration of marijuana use and school enrollment status, neither variable was significantly associated with the cognitive outcomes in univariate or multivariate models. Following adjustment, model parameter and associated standard error estimates were largely unchanged (data not shown).
4.0 Discussion
Cognitive performance in certain domains, namely verbal memory and psychomotor speed, as measured by the CNS Vital Signs battery, was significantly better in those with abstinence when compared to those who were not abstinent. Results suggest an improvement in these cognitive performance domains with abstinence from marijuana. The component most impacted by cessation of marijuana in our study appeared to be verbal memory. This is consistent with some of the previous literature. Data on verbal memory is mixed, as noted by Randolph and colleagues (Randolph, et al., 2013). For example, work by Lisdahl Medina, et al., shows a deficit in certain domains (complex attention, sequencing ability, verbal story memory, and psychomotor speed) in adolescents with marijuana use that remains after 4 weeks of abstinence (2007). In 2010, Hanson, et al., found verbal working memory improved to the level of that of the control group after 3 weeks of abstinence (Hanson, et al., 2010). It is important to note that for verbal memory, scores in the CA group and RA group increased (although increases only significant in the CA group) while the NA group scores decreased. This would suggest that in our sample, abstinence improved verbal memory while continued use worsened verbal memory. As further discussed below, pre-morbid (i.e. prior to marijuana use) verbal memory performance is unknown in this sample. Therefore, the decrease noticed in the NA group could represent their first decline in verbal memory and/or the increase could be an increase from their pre-morbid verbal performance scores. While these possibilities seem unlikely, we cannot rule them out without having a control group or cognitive testing that was conducted prior to participants’ initiation of marijuana use.
Slight improvement was also shown in the psychomotor speed domain. However, there were no improvements noted in processing speed or attention, which conflicts with previous literature (Randolph, et al., 2013). Contrary to our secondary hypothesis, there did not, overall, seem to be greater improvements with longer periods of abstinence. It appeared that there was an initial boost in verbal memory performance that was sustained but did not improve further with continued abstinence. One could speculate from this data that not only is marijuana use selectively toxic to certain components of the developing brain in youth, but some of the components are more resilient to marijuana use following abstinence.
These results should be interpreted within the limitations of the study and this secondary analysis. We were not powered to detect differences in cognitive performance at smaller time differences than shown in our results. As previously noted, this was the reason for grouping the results as we did (CA, RA, and NA). Thus, there may be other cognitive effects of abstinence that were not detected in this study due to the limited power within the secondary analysis. Additionally, it is possible that further cognitive improvement occurs with longer periods of cessation, but we were not powered to detect differences between 4 weeks and 8 weeks of abstinence. Ideally, individuals that maintained abstinence could be followed longer to monitor for continued improvements.
Another limitation in our study is the absence of data about cognitive performance prior to using marijuana. We did not have non-marijuana using controls, which would have helped determine whether initial scores and improvements in scores were related to marijuana use and abstinence or other factors, such as practice effects with the cognitive performance battery, increased familiarity with testing conditions, and other non-specific or unidentified factors. In addition to not having cognitive data on the participants prior to marijuana use, we also did not compare the cognitive performance data obtained in the study to controls without marijuana use. This was not the intent of the study, and any such comparison would be difficult to interpret, given the lack of a true baseline (i.e., before marijuana initiation) assessment.
While we felt it important to have an objective measure of abstinence (qualitative UCT) as the independent variable for cognitive performance, there are limitations with using this measure. This dilemma is not unique to our study. Whether the UCT is negative or positive, the exact time of abstinence (or time since last use) is variable and hinders our ability to have a precise time associated with abstinence. However, this limitation applies across all participants and therefore the results remain meaningful.
Finally, an intoxication effect on the cognitive performance of the NA group cannot be completely ruled out. However, participants were assessed by the study medical clinician at the time of cognitive testing for signs of acute intoxication; if acute intoxication was suspected cognitive performance testing was not administered.
5.0 Conclusions
These results suggest that, in adolescents, cognition, especially certain components of cognition such as verbal memory, may improve within weeks of marijuana cessation. While there are limitations to our study, it is notable that objective verbal memory improvement can be seen and measured in a relatively short time after cessation. Understanding the impact of cessation of marijuana on cognitive performance over time requires further data from more longitudinal studies.
Highlights.
Cognitive performance was measured using CNS Vital Signs®
Abstinence was significantly associated with increased composite memory scores
Abstinence was significantly associated with increased verbal memory scores
Abstinence was significantly associated with modest increase in psychomotor speed
No significant differences in cognitive performance between placebo and control
Acknowledgements
This work was funded by the NIDA-AACAP Resident Research Award. The primary trial was funded by the National Institute on Drug Abuse grant R01DA026777 (PI: Gray). Administrative and technical support was made possible by the National Center for Research Resources grant UL 1RR029882. The authors would additionally like to thank Alan Boyd for his assistance with furthering our understanding of the assessments used in this study.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Disclosures:
Dr. Gray has received funding for unrelated research from Merck, Inc., and Supernus Pharmaceuticals.
Dr. Roten and Mr. Baker report no potential conflicts of interest.
References
- Abdullaev Y, Posner MI, Nunnally R, Dishion TJ. Functional MRI evidence for inefficient attentional control in adolescent chronic cannabis abuse. Behavioural Brain Research. 2010;215:45–57. doi: 10.1016/j.bbr.2010.06.023. [DOI] [PubMed] [Google Scholar]
- Fried PA, Watkinson B, Gray R. Neurocognitive consequences of marihuana- a comparison with pre-drug performance. Neurotoxicology and teratology. 2005;27(2):231–9. doi: 10.1016/j.ntt.2004.11.003. [DOI] [PubMed] [Google Scholar]
- Gray KM, Carpenter MJ, Baker NL, DeSantis SM, Kryway E, Hartwell KJ, et al. A double-blind randomized controlled trial of N-acetylcysteine in cannabis-dependent adolescents. The American Journal of Psychiatry. 2012;169:805–812. doi: 10.1176/appi.ajp.2012.12010055. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gualtieri CT, Johnson GL. Reliability and validity of a computerized neurocognitive test battery, CNS Vital Signs. Archives of Clinical Neuropsychology. 2006;21(7):623–43. doi: 10.1016/j.acn.2006.05.007. [DOI] [PubMed] [Google Scholar]
- Heishman SJ, Evans RJ, Singleton EG, Levin KH, Copersino ML, Gorelick DA. Reliability and validity of a short form of the marijuana craving questionnaire. Drug and Alcohol Dependence. 2009;102(1-3):35–40. doi: 10.1016/j.drugalcdep.2008.12.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Johnston LD, O'Malley PM, Miech RA, Bachman JG, Schulenberg JE. Monitoring the Future national results on drug use: 1975-2013: Overview, Key Findings on Adolescent Drug Use. Institute for Social Research, The University of Michigan; Ann Arbor: 2014. [Google Scholar]
- Lisdahl Medina K, Hanson KL, Schweinsburg AD, Cohen-Zion M, Nagel BJ, Tapert SF. Neuropsychological functioning in adolescent marijuana users: Subtle deficits detectable after a month of abstinence. Journal of the International Neuropsychological Society. 13(5):807–820. doi: 10.1017/S1355617707071032. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Meier MH, Caspi A, Ambler A, et al. Persistent cannabis users show neuropsychological decline from childhood to midlife. Proceedings of the National Academy of Sciences. 2012;209(40):E2657–64. doi: 10.1073/pnas.1206820109. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hanson KL, Winward JL, Schweinsburg AD, Medina KL, Brown SA, Tapert SF. Longitudinal study of cognition among adolescent marijuana users over three weeks of abstinence. Addictive Behaviors. 2010;35:970–976. doi: 10.1016/j.addbeh.2010.06.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Loring DW, Marino SE, Parfitt D, Finney GR, Meador KJ. Acute lorazepam effects on neurocognitive performance. Epilepsy & Behavior. 2012;25:329–333. doi: 10.1016/j.yebeh.2012.08.019. [DOI] [PubMed] [Google Scholar]
- Patterson HD, Thompson R. Recovery of inter-block information when block sizes are unequal. Biometrika. 1971;58:545–554. [Google Scholar]
- Randolph K, Turull P, Margolis A, Tau G. Cannabis and Cognitive Systems in Adolescents. Adolescent Psychiatry. 2013;3:135–147. [Google Scholar]
- Roten AT, Baker NL, Gray KM. Marijuana craving trajectories in an adolescent marijuana cessation pharmacotherapy trial. Addictive Behaviors. 2013;38:1788–1791. doi: 10.1016/j.addbeh.2012.11.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- SAS® 9.3. SAS Institute. Inc. CNS Vital Signs, LLC.; Cary, NC: [January, 2014]. https://www.cnsvs.com/WhitePapers/CNSVSInterpretationGuide.pdf. [Google Scholar]