Abstract
Background:
Intravenous ketamine, which displays rapid antidepressant properties, is posited to reverse depression by rapidly enhancing neuroplasticity. We tested whether an automated, computer-based approach could efficiently leverage enhanced neuroplasticity to extend the durability of rapid clinical response.
Methods:
154 adults (age 18–60) with unipolar, treatment-resistant depression were randomized in a double-blind, parallel arm design to receive an active/active treatment combination [ketamine+active Automated Self-Association Training (ASAT);n=53], or one of two control arms that lacked either the active drug (saline+active ASAT;n=51) or the active behavioral (ketamine+sham ASAT;n=50) components. One day after a single infusion of intravenous ketamine (0.5mg/kg over 40minutes) or inert placebo (saline), active ASAT—targeting self-worth through automated, ‘evaluative conditioning’ training delivered by computer—or sham ASAT (consisting of identical computer tasks that included no positive nor self-referential stimuli) was given, delivered twice daily over 4 consecutive days (eight ~20min sessions; 2.5hours total). The Montgomery-Asberg Depression Rating Scale (MADRS) was the pre-specified primary outcome throughout the main (30-day) study period.
Results:
Ketamine rapidly reduced depression scores at 24-hours post-infusion (group*time:β*=−1.30[95% CI: −1.89 to −0.70];t150=−4.29;p<.0001). In intent-to-treat, linear mixed models, ket+ASAT depression scores remained stably low over the 30-day acute phase, relative to saline+ASAT (β*=−0.61[95% CI:−0.95 to −0.28];t148=−3.62;p=.0004). By contrast, depression scores following ket+Sham followed an increasing linear trajectory from 24-hours to 30-days, approaching saline+ASAT levels (group*time relative to saline+ASAT:β*=0.015[95% CI:0.003 to 0.03];t568=2.35;p=.019).
Conclusions:
After priming the brain with ketamine, training positive self-associations could provide an exceedingly efficient, low-cost, portable, non-invasive, and highly dissemination-ready strategy for leveraging and extending ketamine’s rapid antidepressant effects.
Trial Registration:
Clinicaltrials.gov NCT03237286, “Intravenous Ketamine Plus Neurocognitive Training for Depression” (https://clinicaltrials.gov/ct2/show/NCT03237286)
Introduction
Depression is one of the most prevalent and costly mental health conditions1, with a public disease burden of staggering proportions2. While efficacious treatments have been available for decades, remission rates are low, relapse rates are high, and disorder prevalence rates remain notably consistent or increased, with only 12.7% of patients receiving minimally adequate treatment3. In recent years, two potential breakthroughs have ignited hopes for turning a corner in the clinical management of this disabling disorder. First, rapid-acting pharmacological agents—notably, intravenous ketamine4–6—have shown promise in dramatically reducing symptoms within the space of 2–24 hours, overcoming the sluggish response to conventional therapies (e.g., 4–6 week delays) which prolongs suffering, creates non-compliance issues, and fails to address urgent clinical needs (e.g., suicidality and other psychiatric crises). Second, intervention development has increasingly sought to take advantage of technology to increase patient access, reduce cost, and minimize aversive consequences through the use of automated, computer-based procedures7, including mechanistic treatments designed to modulate affective processing patterns directly in order to reduce symptoms8. The current study aimed to synergistically combine the benefits of these two approaches.
The rapidity of ketamine’s effects offers the promise of a breakthrough in the way that depression is managed, but this promise remains unfulfilled to date. Quickly dissipating effects following a single infusion in isolation9, coupled with substantial concerns regarding safety and feasibility of repeated infusions10,11, limit ketamine’s clinical impact. While substantial advancements have recently been made in understanding ketamine’s molecular mechanisms of action12,13 and developing strategies to extend these effects pharmacologically14,15, translation to safe, effective, novel pharmacology can be slow and uncertain. Furthermore, long-term pharmacological intervention is unlikely to be beneficial, feasible, and appealing to every patient16. A large majority of patients (e.g., up to 91% of depressed patients enrolling in drug trials17) consistently express preference for behavioral or combined behavioral/pharmacological treatments18,19, in part reflecting the belief that such treatments will introduce new learning that will solve the core problem and instantiate lasting change19. Antidepressant pharmacological regimens are challenging to maintain in the long term due to patient discontinuation16 and the rarity of follow-up opportunities in community practice20.
Ketamine is posited to reverse the molecular signature of depression via synaptogenic and neuroplasticity effects at the molecular level12,21,22. Based on prior work, we have hypothesized23,24 that these molecular effects may produce a corresponding neurocognitive shift, in which we can extend rapid mood relief during a window of opportunity using behavioral learning-based approaches—for example, by reinforcing adaptive patterns of cognition through automated training. Our study was designed to initiate training during a key clinical window of opportunity—1–4 days following a ketamine infusion, when euthymic mood and enhanced plasticity were expected—in order to consolidate beneficial processing patterns and prolong ketamine’s rapid mood effects.
In the context of depression, clinical effects of automated training paradigms, though showing some potential8,25, are fairly unreliable, particularly in acutely depressed patients26,27—a situation that is likely exacerbated in the context of treatment-resistant forms of depression. We thus hypothesized that, only after first ‘priming’ brain plasticity with ketamine, the introduction and facilitation of new learning via automated methods might provide an efficient path to enduring relief for treatment-resistant patients. Consistent with rising momentum around the creation of synergistic bio-behavioral approaches to psychiatric management (including both ketamine28–30 and other “psychoplastogenic” drugs31), the current study was designed to push the boundaries of how rapid-acting antidepressant drug agents could be used clinically, i.e. as short-term cognitive flexibility enhancers to promote learning antithetical to the depressed state. Ketamine infusion was used as the first step in a novel, efficient treatment algorithm designed to foster relief that would be both rapid and durable, simultaneously taking advantage of technology’s potential for low-cost, portable, safe, dissemination-ready intervention. This approach is in line with ongoing, similar efforts to pharmacologically augment automated cognitive training interventions in diverse populations (e.g., schizophrenia; cognitive aging)32–36, but targets a unique, affectively valenced aspect of cognition—namely, implicit self-worth.
An automated training paradigm specifically targeting implicit self-associations (“Automated Self-Association Training”; ASAT) was developed, rooted in our prior work suggesting optimal synergy might be achieved when targeting this particular form of information processing following ketamine. Rapid shifts in implicit self-associations following ketamine have been replicated in two previous studies37,38, suggesting ketamine creates the requisite malleability in information processing within this domain. To capitalize on this plasticity and to shift self-associations in a positive direction, we thus adapted existing appetitive conditioning paradigms known broadly as “evaluative conditioning.”39,40 Prior neuroimaging work also implicated rapid shifts following ketamine in prefrontal and striatal network activity and connectivity (e.g., increased activation in the caudate and increased striatal-mPFC connectivity during face processing41), which are widely implicated as substrates of appetitive conditioning in humans and rodents42, further suggesting potential mechanistic synergy (at the neurocircuitry level) for our selected approach when initiated following ketamine. Finally, “depressive self-schemas” (negative self-beliefs) are among the most long-standing and well-documented cognitive mechanisms mediating depression43, suggesting strong clinical relevance for this mechanistic target.
In this first-of-its-kind study, our active/active combination of ketamine+ASAT was compared, in a randomized, double-blind, parallel arm design, to control conditions that included each intervention component in the absence of the other (i.e., saline+active ASAT; ketamine+sham ASAT). This enabled a direct test of hypothesized complementary actions between ketamine and ASAT, in an effort to generate antidepressant action that was both rapid and enduring.
Methods
Participants.
154 adult (age 18–60) patients reporting moderate-to-severe levels of depression [Montgomery-Asberg Depression Rating Scale (MADRS;44) score ≥25], lower-than-normative self-reported self-esteem [i.e., scoring outside of 1SD from the normative mean on the Cognitive Triad Inventory45 “self” subscale46,47 and/or the Rosenberg Self-Esteem Scale48], and at least one failed, adequate trial of an FDA-approved antidepressant medication in the current depressive episode (per Antidepressant Treatment Response Questionnaire; ATRQ49), were randomized. All patients met DSM-5 criteria for Major Depressive Disorder (see Supplement). Any existing depression treatment regimens were required to be stably maintained beginning ≥4 weeks prior to screening (which equated to roughly 6 weeks prior to infusion date) and throughout the 30-day trial. See the Supplement for further rationale and details of all inclusion/exclusion criteria, screening assessments, and study ethical oversight. See Figure 1 for CONSORT flowchart reflecting high compliance/retention and Table 1 for descriptive patient characteristics.
Figure 1.
CONSORT Flow Diagram. Patients are labeled as having received their allocated infusion if they initiated the infusion. For one single patient, who was allocated to the ketamine+sham ASAT arm, the infusion was discontinued after administration of approximately two-thirds of the intended dose, at the patient’s request. Patients are labeled as having “received” their allocated ASAT condition (active or sham) if they completed ≥75% of the intended 8 sessions. All but two of the patients labeled as having “received” their ASAT allocation completed the entire set of 8 sessions, per protocol.
Table 1.
Clinical and demographic characteristics of the randomized (intent-to-treat) sample
Full intent-to-treat sample (n=154) | Ketamine+active ASAT sample (n=53) | Ketamine+sham ASAT sample (n=50) | Saline+active ASAT sample (n=51) | |||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Race/ethnicity | ||||||||
Non-Hispanic Caucasian, n (%) | 116 | (75.32%) | 39 | (73.58%) | 37 | (74.00%) | 40 | (78.43%) |
Hispanic/Latino, n (%) | 11 | (7.14%) | 5 | (9.43%) | 2 | (4.00%) | 4 | (7.84%) |
African American, n (%) | 7 | (4.55%) | 3 | (5.66%) | 2 | (4.00%) | 2 | (3.92%) |
Asian, n (%) | 9 | (5.84%) | 4 | (7.55%) | 2 | (4.00%) | 3 | (5.88%) |
More than one race, n (%) | 11 | (7.14%) | 2 | (3.77%) | 5 | (10.00%) | 4 | (7.84%) |
Unknown/incomplete info, n (%) | 3 | (1.95%) | 0 | (0.00%) | 2 | (4.00%) | 1 | (1.96%) |
Sex | ||||||||
Assigned female sex at birth, n (%) | 97 | (63.0%) | 32 | (60.4%) | 32 | (64.0%) | 33 | (64.7%) |
Gender | ||||||||
Cisgender male, n (%) | 55 | (35.71%) | 21 | (39.62%) | 17 | (34.00%) | 17 | (33.33%) |
Cisgender female, n (%) | 90 | (58.44%) | 29 | (54.72%) | 28 | (56.00%) | 33 | (64.71%) |
Transgender: female-to-male, n (%) | 1 | (0.65%) | 1 | (1.89%) | 0 | (0.00%) | 0 | (0.00%) |
Transgender: male-to-female, n (%) | 1 | (0.65%) | 0 | (0.00%) | 0 | (0.00%) | 1 | (1.96%) |
Non-binary or gender fluid, n (%) | 2 | (1.30%) | 1 | (1.89%) | 1 | (2.00%) | 0 | (0.00%) |
Gender undisclosed/unknown, n (%) | 5 | (3.25%) | 1 | (1.89%) | 4 | (8.00%) | 0 | (0.00%) |
Age, mean (SD) | 34.26 | (10.5) | 34.7 | (10.1) | 34.6 | (11.6) | 33.5 | (9.9) |
# of failed adequate ADM trials, mean (SD) | 2.64 | (1.93) | 2.66 | (1.59) | 2.60 | (1.67) | 2.67 | (2.46) |
Taking psychotropic medication, n (%) | 122 | (79.2%) | 40 | (75.5%) | 42 | (84.0%) | 40 | (78.4%) |
Comorbid anxiety disorder, n (%) | 94 | (61.0%) | 30 | (56.6%) | 30 | (60.0%) | 34 | (66.7%) |
MADRS total score (pre-infusion baseline) | 32.78 | (5.3) | 32.28 | (5.7) | 33.48 | (4.9) | 32.61 | (5.1) |
Note: No variables in the table above differed as a function of treatment group according to unpaired t-tests (for continuous variables) or Chi-squared tests (for categorical variables) (p’s>.49). ASAT=Automated Self-Association Training; ADM=antidepressant medication
To achieve the initial recruitment target of n=150 patients, the study enrolled and randomized n=154 patients, continuing enrollment until a final sample was acquired that included the full target of n=150 patients who completed, at a minimum, 1) an infusion day, 2) a 24-hour assessment visit, and 3) ≥75% of intended training (active or sham ASAT) visits, and were thus considered to have received their ASAT allocation (and labeled accordingly in the CONSORT flowchart; Figure 1). All but two participants who surpassed this 75% threshold for “receiving” their ASAT allocation received the full, 8-session dose of intended ASAT visits, per protocol (n=148 full, per-protocol completers who received all allocated ASAT sessions). 96% of randomized participants were thus fully compliant with all intervention sessions, and 97% of all intended ASAT sessions were delivered.
A 3-arm study design was selected to allocate all available resources towards the active/active (ketamine+ASAT) treatment arm relative to two crucial comparator groups, comprising each intervention component in the absence of the other—a conservative test of the active/active bio-behavioral intervention’s combined impact. An inactive/inactive (no-treatment) arm was foregone in order to maximize statistical power for these more conservative and critical comparisons. An a priori power analysis was based on a principle interest in clinically meaningful, moderate (or larger) effects. The target sample sizes were selected to yield 80% power to detect moderate effects based on Infusion Phase comparisons (see ‘Statistical Analysis’ below) of 100 ketamine and 50 saline patients (d≥.49 and r≥.31 using α=.05), and based on comparisons of 50 patients in each of the ketamine+ASAT, ketamine+sham, and saline+ASAT conditions during the ASAT Phase, where effects of d≥.57 would be detectable with 80% power (using α=.05). Power for primary mixed effects analyses comparing treatment groups, which increases with additional repeated measures, was anticipated to be higher than for these simplified, single-point contrasts, making these power calculations conservative.
Assessment Measures and Timeline.
The current, primary clinical report from the trial focuses on the combined intervention’s impact on the MADRS, the study’s pre-specified primary clinical endpoint for the acute (30-day) study period, to assess overall depression severity across the 30-day period when all other psychiatric treatments were held constant. A single experienced, Master’s-level clinical rater (CRS) administered the MADRS. The scale was repeated at screening (used for eligibility determinations only), pre-infusion baseline (infusion −1 hour; the clinical rater then left and was not present during any intervention procedure), infusion +24 hours (prior to the first session of active/sham ASAT), +5, +12, +21, and +30 days. A 1-year exploratory naturalistic follow-up, utilizing remote assessments acquired strictly through a distinct self-report symptom scale (QIDS-SR50), remains ongoing and data are not included in the present report. Neurocognitive/neuroimaging assessments of ketamine’s impact are undergoing analysis and will be included in forthcoming publications, along with secondary outcomes; however, one such cognitive measure (a “target engagement” measure of implicit self-associations) is analyzed in the Supplement. At the final (+30 days) in-person visit, adequacy of both patient and rater blinding, for both infusion and ASAT allocation, was assessed; see Supplement.
Interventions.
A randomization table using permuted blocks of 3 or 6, stratified by 1) biological sex assigned at birth and 2) modest vs. severe treatment resistance (severe = ≥3 FDA-approved antidepressant trials in episode, per ATRQ), was generated using SAS statistical software (SAS; Cary, NC) by an independent statistical consultant at study start and maintained by the study pharmacist, who, at the point of randomization, conveyed the ASAT allocation (but not the infusion allocation) to a single study staff member assigned to that patient. This single staff member was tasked with opening the correct computer program to deliver the ASAT sessions to that patient, but was not involved in, nor present for, any outcome ratings. Blinding of all other study personnel was maintained until after the final patient’s Day 30 assessment. Infusion. Participants were randomized 2:1 to ketamine (0.5mg/kg) or saline (50ml 0.9% Sodium Chloride) given over a 40min infusion, as in prior studiese.g.,51. All infusions were administered by blinded, licensed nurses in a medical hospital setting with linked ACLS-certified team, blinded study physician (RHH) oversight, and safety/adverse event monitoring sustained for 4-hours post-infusion. Adverse events (Table S2—Supplement) were mild-to-moderate, time-limited, and consistent with the safety profile described in numerous previous trials using identical infusion procedures.
Automated Self-Association Training (ASAT).
Drawing on previous work utilizing ‘evaluative conditioning’ to influence participants’ preferences and ‘liking’39,40, active ASAT was designed to leverage Pavlovian conditioning in order to promote positive implicit self-associations and self-worth. Eight 15–20 minute sessions (delivered in a private research office setting, twice daily for 4 consecutive days, spaced by a ≥20min inter-session interval) were initiated 1 day post-infusion (after the 24-hour post-infusion assessment). In each of the 8 sessions, both verbal and pictorial stimulus pairings were presented, both supraliminally (≥250ms) and subliminally (12ms), to practice and reinforce implicit associations between positive traits [e.g., ‘sweet’, ‘attractive’, photos of smiling actors; the unconditioned stimuli (US)] and self-referential stimuli (e.g., ‘I’, participant headshots; the conditioned stimuli (CS)]. Incidental tasks such as a ‘lexical decision’ task (indicating whether targets were real words or random letter strings) and a rapid mouse-tracking task (clicking as fast as possible on the position of stimuli) were used to enhance engagement and to promote semantic and visual processing of stimuli. Similar forms of evaluative conditioning have been found to alter implicit self-esteem40,52,53, mood reactivity to stress52, and pathological behaviors (anxious avoidance54, self-injury25).
Sham ASAT consisted of the exact same computer tasks, but with predominantly neutral rather than positive US and non-self-relevant CS (words related to ‘others’, pictures of gender-matched strangers), designed to eliminate the possibility of unintended self-referential or negative/iatrogenic learning, while providing a credible “brain-training” paradigm that controlled for all non-specific factors (e.g., exposure to lexical and pictorial stimuli with a range of affective valences; incidental task demands; time spent in the research setting) and was effective in facilitating adequate patient blinding (see Table S1). See Supplement for further technical details of both active and sham ASAT procedures.
Statistical Analysis.
Intent-to-treat, linear mixed effects regression models were applied with continuous MADRS scores as the outcome; days since infusion as a random, continuous within-subject effect; and treatment allocation as a fixed, between-subjects effect. Separate models were designed to test two discrete hypothesized linear patterns, based on: 1) the Infusion Phase of intervention (2-way split: ketamine vs. saline), using pre-infusion and 24-hour assessments only (assessments collected prior to onset of active/sham ASAT); and 2) the ASAT Phase of intervention (3-way split: saline+ASAT vs. ketamine+sham vs. ketamine+ASAT), using 24-hour (as a pre-ASAT baseline for these trajectories) through Day 30 assessment points.
In the Infusion Phase, we hypothesized rapid symptom decreases in the ketamine arm (relative to saline). For descriptive and clinical characterization purposes only, to facilitate comparisons with prior studies, dichotomous outcomes were defined for 24-hour responders (≥50% decrease from pre-infusion MADRS baseline) and remitters (MADRS≤955). In the ASAT Phase, the saline+ASAT arm was the reference group for hypothesis tests, and models included pre-infusion MADRS scores as a covariate. We hypothesized a stable (main) effect extending from 24-hours to Day 30 in the ketamine+active ASAT arm (relative to saline+ASAT), with no group*time interaction. By contrast, for the ketamine+sham ASAT group, we hypothesized a linearly increasing trajectory of symptoms from 24-hours to Day 30 in the ketamine+sham arm (i.e., a group*time interaction relative to the reference group), consistent with dissipation of symptom improvements within 1–2 weeks following a single infusion of ketamine (in isolation) reported in prior ketamine trials4–6.
All models included a random intercept and slope for participant to model patient-level trajectories over time and automatically account for missing data, which was minimal (5.1% of all intended observations). For interpretability, continuous variables were standardized. We report standardized coefficients (using the notation ‘β*’) as a measure of effect size reflecting the number of standard deviations in the dependent measure that correspond to a 1-unit change in the independent measure (e.g., group contrast), akin to Cohen’s d, with 95% profile likelihood confidence intervals. Analyses were performed using R version 3.6.
Sensitivity analyses (see Supplement) probed for the robustness of findings when including the following covariates selected a priori: sex, age, treatment-resistance (moderate vs. severe), and use of concomitant psychotropic medications (dichotomized as yes/no).
Results
Depression severity.
Infusion phase.
Consistent with prior work, ketamine rapidly reduced MADRS total depression scores at 24-hours post-infusion (group*time:β*=−1.30[95% CI: −1.89 to −0.70];t150=−4.29;p<.0001;Figure 2). This moderate-to-large linear effect corresponded to a 52% responder and 28% remitter rate in ketamine-treated participants, compared to a 25% responder and 4% remitter rate in saline-treated participants [number-needed-to-treat (NNT) for response=3.7; NNT for remission=4.2].
Figure 2.
Depression severity scores (total Montgomery-Asberg Depression Rating scale scores) as a function of days since infusion and treatment allocation [blue line: ketamine+active Automated Self-Association Training (ASAT), n=53]; orange line: ketamine+sham ASAT, n=50; red line: saline+active ASAT, n=51]. Line plots represent mean values and error bars represent standard error of the mean within each timepoint and treatment group.
ASAT phase.
In intent-to-treat, linear mixed models, ketamine+ASAT depression scores remained stably low over the 30-day acute phase relative to saline+ASAT (β*=−0.61[95% CI: 0.95 to −0.28];t148=−3.62;p=.0004;Figure 2), with no corresponding group*time interaction (β*=0.009[95% CI: −0.004 to 0.021];t568=1.39;p=.164), suggesting durability of effect over the 30-day window. By contrast, depression scores following ketamine+Sham followed an increasing linear trajectory from 24-hours to 30-days, as the post-ketamine effect gradually waned and depression approached saline+ASAT levels (group*time relative to saline+ASAT:β*=0.015[95% CI: 0.003 to 0.03];t568=2.35;p=.019). While the above, intent-to-treat statistical tests of linear trajectories over the full 30-day period were used for statistical hypothesis tests, pairwise post hoc contrasts at day 30 were used to generate effect size point estimates for further description of clinical impact at this specific timepoint. These effect size point estimates suggested a small effect favoring ketamine+ASAT relative to both saline+ASAT (β*=−0.38[95% CI: −0.78 to 0.027]) and ketamine+Sham (β*=−0.31[95% CI: −0.75 to 0.13]).
All findings above were robustly upheld in sensitivity analyses controlling for covariates (see Supplement). No significant interaction (moderating) effects were observed between covariates and treatment allocation for either study phase (Supplement).
Discussion
In this study, Automated Self-Association Training—a novel, low-cost, fully automated, non-invasive, brief (8 sessions total, ≤20min/session), computer-based intervention—extended the rapid antidepressant effect of a single ketamine infusion for at least 30 days. In one of the largest RCT samples to date, we replicated the well-established finding that intravenous ketamine exerts a rapid antidepressant effect among treatment-resistant depressed patients4–6. Ketamine infusion appeared also to open a clinical window of opportunity, enabling the efficient uptake of positive self-representations (which were evident at the level of implicit cognition following ASAT; see Supplement) to protect against the return of depression over the subsequent month (Figure 2). Symptoms in the saline+ASAT group, which received identical ASAT procedures in the absence of pre-treatment with ketamine, were elevated relative to ketamine+ASAT throughout the entire 30-day follow-up. Of note, symptoms following saline+ASAT also remained stably below pre-treatment baseline, which could be attributable to a stand-alone impact of ASAT procedures (which were effective at altering implicit cognition, as noted above); an enduring, non-specific impact of the overarching clinical research context (e.g., repeated depression assessments and staff interactions); or both. A second comparator group, ketamine followed by sham ASAT, followed the expected trajectory when a single ketamine infusion has been given as a stand-alone treatment, i.e., rapid decrease followed by gradual return of depressive symptoms over the ensuing weeks4–6. To our knowledge, this represents the first study to test a bio-behavioral pairing of ketamine with a behavioral intervention that has included both a no-ketamine and a no-behavioral intervention control. Likewise, clinical trials testing a wide range of posited synergistic treatments (e.g., behavioral treatments paired with “psychoplastogens” or other brain-based targeted approaches, e.g. neuromodulation) have very routinely neglected one or the other of these two critical comparators24. Our design affords pivotal, novel evidence for the combined benefit of ketamine’s acute impact and the introduction of protective learning via ASAT, relative to patients’ pre-treatment baseline and to either intervention component when given in isolation—providing a conservative test of efficacy. However, our design cannot provide estimates of the impact of the combined intervention (nor of each intervention component alone) relative to no intervention at all.
While we expect that a range of traditional and non-traditional behavioral treatments might work synergistically with rapid-acting pharmacology, here we focused specifically on our novel ASAT approach, for both practical and scientific reasons. Practically, ASAT is fully automated and portable, and designed to be efficient, fitting well within the brevity (i.e., about 1 week) of ketamine’s posited window-of-opportunity. By contrast, a full course of either traditional or computer-based CBT typically requires 12–16 1-hour sessions delivered weekly, and meta-analyses suggest optimal computer-based CBT requires supplemental therapist coaching56. Here, 30–40 minutes of ASAT for 4 consecutive days yielded effect sizes for ketamine+ASAT (relative to unimodal treatment conditions) that were moderate across the entire follow-up interval, but small in point estimates at the final assessment point (Day 30). Further work will be needed to understand the patient perspective on the cost/benefit ratio of such gains vis-à-vis other, conventional treatment options (e.g., psychotherapy) and to assess adherence in clinical settings. However, we observed exceptional (97%) compliance with ASAT sessions in our research context, which is consistent with the high compliance and high ratings for acceptability and satisfaction observed previously for other fully automated therapies, across both research and clinical contexts7,57–59.
Scientifically, a robust evidence base suggests that implicit and explicit forms of cognition are distinct entities, and that implicit cognition (e.g., implicit self-concept) could be a more robust predictor of future behavior than explicit thought content60. ASAT is designed to efficiently and directly target implicit cognition, yielding an innovative approach with the potential to profoundly shape future behavior and experience. Furthermore, by targeting a unitary and well-defined implicit cognitive mechanism, ASAT limits the influence of heterogeneous and non-specific factors that are likely influential in many traditional behavioral interventions, and for which the learning that occurs within a brief window may be more difficult to predict and constrain (e.g., positive vs. negative therapy session experiences). Broadly, evaluative conditioning techniques have a large and robust literature in healthy controls61; and in clinical samples, have previously shown replicated acute clinical effects even in an at-home (smartphone) delivery modality25. Although in this initial test of ASAT, we administered the training in a research lab environment, the ability to both achieve and maintain gains via portable, highly dissemination-ready techniques is a crucial criterion for overcoming intractable barriers to access to care, particularly for underserved communities62, representing a critical future direction for this work.
Limitations.
Given budgetary constraints, all available resources were allocated towards determining the advantage of the active/active bio-behavioral treatment combination over its component parts delivered in isolation, providing a conservative test of efficacy focused on establishing the necessity of each active component, but negating the ability to determine intervention efficacy in relation to no-treatment. Functional unblinding was evident for the infusion condition (Table S1; see further discussion in Supplement); clinical effects observed for ketamine vs. saline may thus have been amplified by expectancy. Importantly, because adequate ASAT condition blinding was achieved (Table S1), such effects are unlikely to have factored strongly into the subsequent extending effect of ASAT—the primary, novel focus of the current work. While ASAT achieved its hypothesized effect of extending the initial rapid improvements induced by ketamine, it did not produce incremental improvements over and above ketamine’s rapid effect, and response and remission rates leave substantial room for further improvement of efficacy. Our design cannot directly inform key clinically relevant questions regarding dose-response effects, treatment schedule optimization, and the trade-off between treatment efficiency and obtaining (and sustaining) maximal benefit. We erred on the side of front-loading and maximizing ASAT administration during the acute post-ketamine window of opportunity, within the perceived confines of feasibility and patient burden, while preserving spaced practice (i.e., 20min inter-session breaks) in an effort to enhance learning63. While this initial effort focused on quantifying 30-day symptom trajectories following this highly efficient (5-day) intervention package, future work involving repeated ketamine doses, higher overall doses of ASAT, and/or subsequent ASAT booster sessions would expand the clinical relevance of the present work. Our ongoing, exploratory, naturalistic 1-year follow-up could yield further insights regarding durability and time to relapse, but for ethical reasons we did not ask patients to refrain from freely making treatment changes and additions during follow-up. Future work could also assess the feasibility and impact of targeting additional cognitive-affective associations beyond self-worth (e.g., perceptions of others, the future, etc.), which might enhance effect sizes and/or enable a personalized, modular approach that could be applicable transdiagnostically.
Finally, although clinical trial eligibility criteria were designed to emulate “real-world” treatment-seeking depressed samples (e.g., stable medications maintained rather than discontinued; few exclusions based on medical or psychiatric comorbidities), results may not robustly generalize to patients with distinct characteristics, including those with either greater or lesser degrees of treatment resistance; bipolar depression; comorbid moderate-to-severe substance use disorders; and, critically, greater racial and ethnic diversity than what was achieved here.
Conclusions.
The global burden of depression is extremely high, and is expected to continue to increase within the current context of a significant pandemic. There is an urgent need for novel treatment approaches, particularly those that can provide relief efficiently and at scale. If present results can be replicated, this novel, integrative treatment may provide a method to urgently bring relief, and efficiently extend this relief via safe, low-cost, portable techniques. Alongside our fully automated ASAT intervention, ketamine infusion may likewise be more dissemination-ready than many alternatives, with a favorable safety profile for isolated infusions and wide international medical usage in both anesthetic and subanesthetic applications. Additional efforts are warranted to create, and then exploit, rapidly induced neurobiological states (such as enhanced neuroplasticity) as clinical windows of opportunity in which to introduce new, protective learning.
Supplementary Material
Acknowledgement
Supported by National Institute of Mental Health Biobehavioral Research Awards for Innovative New Scientists (BRAINS) R01 grant R01MH113857 (Dr. Price) and by the Clinical and Translational Sciences Institute at the University of Pittsburgh (UL1-TR-001857). We are deeply grateful to the study participants for their time and dedicated collaboration in this work. We also gratefully acknowledge Joseph Franklin, PhD, Chadi Abdallah, MD, PhD, Satish Iyengar, PhD, and Lisa Parker, PhD for their assistance with this work.
Financial Disclosures
Dr. Price is the named inventor on a University of Pittsburgh-owned provisional patent filing related to the combination intervention described in this manuscript. Dr. Wallace has served as a statistical consultant for Health Rhythms, Noctem Health, and Sleep Number Bed. Dr. Mathew is supported through the use of facilities and resources at the Michael E. Debakey VA Medical Center, Houston, Texas, and receives support from The Menninger Clinic. Dr. Mathew has served as a consultant to Alkermes, Allergan, Axsome Therapeutics, BioXcel Therapeutics, Clexio Biosciences, Eleusis, EMA Wellness, Engrail Therapeutics, Greenwich Biosciences, Intra-Cellular Therapies, Janssen, Levo Therapeutics, Neurocrine, Perception Neuroscience, Praxis Precision Medicines, Relmada Therapeutics, Sage Therapeutics, Seelos Therapeutics, and Signant Health. He has received research support from Biohaven Pharmaceuticals, Merck, Sage Therapeutics, and VistaGen Therapeutics. All other authors report no financial conflicts of interest to disclose.
References
- 1.Kessler RC, Chiu WT, Demler O, Merikangas KR, Walters EE: Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):617–27. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Kessler RC: The global burden of anxiety and mood disorders: putting the European Study of the Epidemiology of Mental Disorders (ESEMeD) findings into perspective. J Clin Psychiatry. 2007;68 Suppl 2:10–9. [PMC free article] [PubMed] [Google Scholar]
- 3.Wang PS, Lane M, Olfson M, Pincus HA, Wells KB, Kessler RC: Twelve-month use of mental health services in the United States: results from the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):629–40. [DOI] [PubMed] [Google Scholar]
- 4.Xu Y, Hackett M, Carter G, Loo C, Galvez V, Glozier N, Glue P, Lapidus K, McGirr A, Somogyi AA, Mitchell PB, Rodgers A: Effects of low-dose and very low-dose dose ketamine among patients with major depression: a systematic review and meta-analysis. Int J Neuropsychopharmacol. 2015 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Newport DJ, Carpenter LL, McDonald WM, Potash JB, Tohen M, Nemeroff CB, Treatments APACoRTFoNBa: Ketamine and Other NMDA Antagonists: Early Clinical Trials and Possible Mechanisms in Depression. Am J Psychiatry. 2015;172(10):950–66. [DOI] [PubMed] [Google Scholar]
- 6.Caddy C, Amit BH, McCloud TL, Rendell JM, Furukawa TA, McShane R, Hawton K, Cipriani A: Ketamine and other glutamate receptor modulators for depression in adults. Cochrane Database Syst Rev. 2015;Sep 23(9):CD011612. [DOI] [PubMed] [Google Scholar]
- 7.Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M: Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry. 2013;35(4):332–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Siegle GJ, Price RB, Jones N, Ghinassi F, Painter T, Thase ME: You gotta work at it: Pupillary indices of task focus are prognostic for response to a neurocognitive intervention for depression. Clin Psychol Sci. 2014;2(4):455–71. [Google Scholar]
- 9.Walter M, Li S, Demenescu LR: Multistage drug effects of ketamine in the treatment of major depression. Eur Arch Psychiatry Clin Neurosci. 2014;264 Suppl 1:S55–65. [DOI] [PubMed] [Google Scholar]
- 10.Schatzberg AF: A word to the wise about ketamine. Am J Psychiatry. 2014;171(3):262–4. [DOI] [PubMed] [Google Scholar]
- 11.Price RB: From Mice to Men: Can Ketamine Enhance Resilience to Stress? Biol Psychiatry. 2016;79(9):e57–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Zanos P, Moaddel R, Morris PJ, Georgiou P, Fischell J, Elmer GI, Alkondon M, Yuan P, Pribut HJ, Singh NS, Dossou KS, Fang Y, Huang XP, Mayo CL, Wainer IW, Albuquerque EX, Thompson SM, Thomas CJ, Zarate CA Jr., Gould TD: NMDAR inhibition-independent antidepressant actions of ketamine metabolites. Nature. 2016;533(7604):481–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Malinow R: Depression: Ketamine steps out of the darkness. Nature. 2016;533(7604):477–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Kantrowitz JT, Halberstam B, Gangwisch J: Single-dose ketamine followed by daily D-Cycloserine in treatment-resistant bipolar depression. J Clin Psychiatry. 2015;76(6):737–8. [DOI] [PubMed] [Google Scholar]
- 15.Abdallah C, Averill L, Gueorguieva R, et al. : Modulation of the antidepressant effects of ketamine by the mTORC1 inhibitor rapamycin. Neuropsychopharmacology 2020; 45:990–997. doi: 10.1038/s41386-020-0644-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Bockting CL, ten Doesschate MC, Spijker J, Spinhoven P, Koeter MW, Schene AH, group Ds: Continuation and maintenance use of antidepressants in recurrent depression. Psychother Psychosom. 2008;77(1):17–26. [DOI] [PubMed] [Google Scholar]
- 17.Steidtmann D, Manber R, Arnow BA, Klein DN, Markowitz JC, Rothbaum BO, Thase ME, Kocsis JH: Patient treatment preference as a predictor of response and attrition in treatment for chronic depression. Depress Anxiety. 2012;29(10):896–905. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Huijbers MJ, Spinhoven P, van Schaik DJ, Nolen WA, Speckens AE: Patients with a preference for medication do equally well in mindfulness-based cognitive therapy for recurrent depression as those preferring mindfulness. J Affect Disord. 2016;195:32–9. [DOI] [PubMed] [Google Scholar]
- 19.van Schaik DJ, Klijn AF, van Hout HP, van Marwijk HW, Beekman AT, de Haan M, van Dyck R: Patients’ preferences in the treatment of depressive disorder in primary care. Gen Hosp Psychiatry. 2004;26(3):184–9. [DOI] [PubMed] [Google Scholar]
- 20.Simon GE, Von Korff M, Rutter CM, Peterson Da: Treatment process and outcomes for managed care patients receiving new antidepressant prescriptions from psychiatrists and primary care physicians. Arch Gen Psychiatry. 2001;58:395–401. [DOI] [PubMed] [Google Scholar]
- 21.Abdallah CG, Sanacora G, Duman RS, Krystal JH: Ketamine and rapid-acting antidepressants: a window into a new neurobiology for mood disorder therapeutics. Annu Rev Med. 2015;66:509–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Duman RS, Aghajanian GK, Sanacora G, Krystal JH: Synaptic plasticity and depression: new insights from stress and rapid-acting antidepressants. Nat Med. 2016;22(3):238–49. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Price RB, Duman R: Neuroplasticity in cognitive and psychological mechanisms of depression: An integrative model. Mol Psychiatry. 2020;25(3):530–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Wilkinson ST, Holtzheimer PE, Gao S, Kirwin DS, Price RB: Leveraging Neuroplasticity to Enhance Adaptive Learning: The Potential for Synergistic Somatic-Behavioral Treatment Combinations to Improve Clinical Outcomes in Depression. 2019;85(6):454–65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Franklin JC, Fox KR, Franklin CR, Kleiman EM, Ribeiro JD, Jaroszewski AC, Hooley JM, Nock MK: A brief mobile app reduces nonsuicidal and suicidal self-injury: Evidence from three randomized controlled trials. J Consult Clin Psychol. 2016;84(6):544–57. [DOI] [PubMed] [Google Scholar]
- 26.Mogoase C, David D, Koster EHW: Clinical Efficacy of Attentional Bias Modification Procedures: An Updated Meta-analysis. J Clin Psychol. 2014;70:1133–57. [DOI] [PubMed] [Google Scholar]
- 27.Motter JN, Pimontel MA, Rindskopf D, Devanand DP, Doraiswamy PM, Sneed JR: Computerized cognitive training and functional recovery in major depressive disorder: A meta-analysis. J Affect Disord. 2016;189:184–91. [DOI] [PubMed] [Google Scholar]
- 28.Dakwar E, Levin FR, Hart CL, Basaraba CN, Choi CJ, Pavlicova M, Nunes EV: A single ketamine infusion combined with motivational enhancement therapy for alcohol use disorder: A randomized, midazolam-controlled pilot trial. Am J Psychiatry. 2020;177:125–33. [DOI] [PubMed] [Google Scholar]
- 29.Dakwar E, Nunes EV, Hart CL, Foltin RW, Mathew SJ, Carpenter KM, Choi CJJ, Basaraba CN, Pavlicova M, Levin FR: A Single Ketamine Infusion Combined With Mindfulness-Based Behavioral Modification to Treat Cocaine Dependence: A Randomized Clinical Trial. Am J Psychiatry. 2019:appiajp201918101123. [DOI] [PubMed] [Google Scholar]
- 30.Wilkinson ST, Rhee TG, Joormann J, Webler R, Ortiz Lopez M, Kitay B, Fasula M, Elder C, Fenton L, Sanacora G: Cognitive Behavioral Therapy to Sustain the Antidepressant Effects of Ketamine in Treatment-Resistant Depression: A Randomized Clinical Trial. Psychother Psychosom. 2021;90(5):318–27. [DOI] [PubMed] [Google Scholar]
- 31.Vargas MV, Meyer R, Avanes AA, Rus M, Olson DE: Psychedelics and Other Psychoplastogens for Treating Mental Illness. Front Psychiatry. 2021;12:727117. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Harvey PD, Sand M: Pharmacological Augmentation of Psychosocial and Remediation Training Efforts in Schizophrenia. Front Psychiatry. 2017;8:177. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Swerdlow NR, Bhakta SG, Talledo J, Kotz J, Roberts BZ, Clifford RE, Thomas ML, Joshi YB, Molina JL, Light GA: Memantine effects on auditory discrimination and training in schizophrenia patients. Neuropsychopharmacology. 2020;45(13):2180–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Swerdlow NR, Tarasenko M, Bhakta SG, Talledo J, Alvarez AI, Hughes EL, Rana B, Vinogradov S, Light GA: Amphetamine Enhances Gains in Auditory Discrimination Training in Adult Schizophrenia Patients. Schizophr Bull. 2017;43(4):872–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Lenze EJ, Stevens A, Waring JD, Pham VT, Haddad R, Shimony J, Miller JP, Bowie CR: Augmenting Computerized Cognitive Training With Vortioxetine for Age-Related Cognitive Decline: A Randomized Controlled Trial. Am J Psychiatry. 2020;177(6):548–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.McClure MM, Graff F, Triebwasser J, Perez-Rodriguez M, Rosell DR, Koenigsberg H, Hazlett EA, Siever LJ, Harvey PD, New AS: Guanfacine Augmentation of a Combined Intervention of Computerized Cognitive Remediation Therapy and Social Skills Training for Schizotypal Personality Disorder. Am J Psychiatry. 2019;176(4):307–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Price RB, Iosifescu DV, Murrough JW, Chang LC, Al Jurdi RK, Iqbal SZ, Soleimani L, Charney DS, Foulkes AL, Mathew SJ: Effects of ketamine on explicit and implicit suicidal cognition: a randomized controlled trial in treatment-resistant depression. Depress Anxiety. 2014;31(4):335–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Price RB, Nock MK, Charney DS, Mathew SJ: Effects of intravenous ketamine on explicit and implicit measures of suicidality in treatment-resistant depression. Biol Psychiatry. 2009;66(5):522–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.De Houwer J, Thomas S, Baeyens F: Associative learning of likes and dislikes: a review of 25 years of research on human evaluative conditioning. Psychol Bull. 2001;127(6):853–69. [DOI] [PubMed] [Google Scholar]
- 40.Martijn MV, Roefs A, Huijding J, Jansen A: Increasing body satisfaction of body concerned women through evaluative conditioning using social stimuli. Health Psychol. 2010;29(5):514–20. [DOI] [PubMed] [Google Scholar]
- 41.Murrough JW, Collins KA, Fields J, DeWilde KE, Phillips ML, Mathew SJ, Wong E, Tang CY, Charney DS, Iosifescu DV: Regulation of neural responses to emotion perception by ketamine in individuals with treatment-resistant major depressive disorder. Transl Psychiatry. 2015;5:e509. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Martin-Soelch C, Linthicum J, Ernst M: Appetitive conditioning: neural bases and implications for psychopathology. Neurosci Biobehav Rev. 2007;31(3):426–40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Dozois D, Beck A: Cognitive schemas, beliefs and assumptions. In: Dobson K, Dozois D, editors. Risk factors in depression. Oxford, England, Elsevier/Academic Press, 2008, p. 121–43 [Google Scholar]
- 44.Montgomery SA, Asberg M: A new depression scale designed to be sensitive to change. Br J Psychiatry. 1979;134:382–9. [DOI] [PubMed] [Google Scholar]
- 45.Beckham EE, Leber WR, Watkins JT, Boyer JL, Cook JB: Development of an instrument to measure Beck’s cognitive triad: the Cognitive Triad Inventory. J Consult Clin Psychol. 1986;54(4):566–7. [DOI] [PubMed] [Google Scholar]
- 46.Anderson KW, Skidmore JR: Empirical analysis of factors in depressive cognition: the cognitive triad inventory. J Clin Psychol. 1995;51(5):603–9. [DOI] [PubMed] [Google Scholar]
- 47.Possel P: Cognitive Triad Inventory (CTI): psychometric properties and factor structure of the German translation. J Behav Ther Exp Psychiatry. 2009;40(2):240–7. [DOI] [PubMed] [Google Scholar]
- 48.Schmitt DP, Allik J: Simultaneous administration of the Rosenberg Self-Esteem Scale in 53 nations: exploring the universal and culture-specific features of global self-esteem. J Pers Soc Psychol. 2005;89(4):623–42. [DOI] [PubMed] [Google Scholar]
- 49.Chandler GM, Iosifescu DV, Pollack MH, Targum SD, Fava M: RESEARCH: Validation of the Massachusetts General Hospital Antidepressant Treatment History Questionnaire (ATRQ). CNS Neurosci Ther. 2010;16(5):322–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Rush AJ, Trivedi MH, Ibrahim HM, Carmody TJ, Arnow B, Klein DN, Markowitz JC, Ninan PT, Kornstein S, Manber R, Thase ME, Kocsis JH, Keller MB: The 16-Item Quick Inventory of Depressive Symptomatology (QIDS), Clinician Rating (QIDS-C) and Self-Report (QIDS-SR): A Psychometric evaluation in patients with chronic major depression. Biol Psychiatry. 2003;54:573–83. [DOI] [PubMed] [Google Scholar]
- 51.Murrough JW, Iosifescu DV, Chang LC, Al Jurdi RK, Green CM, Perez AM, Iqbal S, Pillemer S, Foulkes A, Shah A, Charney DS, Mathew SJ: Antidepressant efficacy of ketamine in treatment-resistant major depression: A Two-site randomized controlled trial. Am J Psychiatry. 2013;170(10):1134–42. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Dijksterhuis A: I like myself but I don’t know why: enhancing implicit self-esteem by subliminal evaluative conditioning. J Pers Soc Psychol. 2004;86(2):345–55. [DOI] [PubMed] [Google Scholar]
- 53.Baccus JR, Baldwin MW, Packer DJ: Increasing implicit self-esteem through classical conditioning. Psychol Sci. 2004;15(7):498–502. [DOI] [PubMed] [Google Scholar]
- 54.Clerkin EM, Teachman BA: Training implicit social anxiety associations: an experimental intervention. J Anxiety Disord. 2010;24(3):300–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Hawley CJ, Gale TM, Sivakumaran T, Hertfordshire Neuroscience Research g: Defining remission by cut off score on the MADRS: selecting the optimal value. J Affect Disord. 2002;72(2):177–84. [DOI] [PubMed] [Google Scholar]
- 56.Andersson G, Cuijpers P: Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther. 2009;38(4):196–205. [DOI] [PubMed] [Google Scholar]
- 57.Price RB, Wallace M, Kuckertz JM, Amir N, Graur S, Cummings L, Popa P, Carlbring P, Bar-Haim Y: Pooled patient-level metaanalysis of children and adults completing a computer-based anxiety intervention targeting attentional bias. Clin Psychol Rev. 2016;50:37–49. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Vallury KD, Jones M, Oosterbroek C: Computerized Cognitive Behavior Therapy for Anxiety and Depression in Rural Areas: A Systematic Review. J Med Internet Res. 2015;17(6):e139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Lau JYF, Sharma NP, Bennett E, Dhakal S, Vaswani A, Pandey R, Niraula S, Kumari V: Acceptability of a brief training programme targeting attention and interpretation biases for threat in youth with a history of maltreatment. Behav Cogn Psychother. 2020;48(3):370–5. [DOI] [PubMed] [Google Scholar]
- 60.Greenwald AG, Poehlman TA, Uhlmann EL, Banaji MR: Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. J Pers Soc Psychol. 2009;97(1):17–41. [DOI] [PubMed] [Google Scholar]
- 61.Hofmann W, De Houwer J, Perugini M, Baeyens F, Crombez G: Evaluative conditioning in humans: a meta-analysis. Psychol Bull. 2010;136(3):390–421. [DOI] [PubMed] [Google Scholar]
- 62.Atdjian S, Vega WA: Disparities in mental health treatment in U.S. racial and ethnic minority groups: implications for psychiatrists. Psychiatr Serv. 2005;56(12):1600–2. [DOI] [PubMed] [Google Scholar]
- 63.Cepeda NJ, Pashler H, Vul E, Wixted JT, Rohrer D: Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychol Bull. 2006;132(3):354–80. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.