Electroencephalographic Biomarkers for Treatment Response Prediction in Major Depressive Illness: A Meta-Analysis

Abstract

Objective:

Reducing unsuccessful treatment trials could improve depression treatment. Quantitative analysis of the electro-encephalogram (QEEG) might predict treatment response, and is being commercially marketed for this purpose. The authors sought to (1) quantify the reliability of QEEG for response prediction in depressive illness and (2) identify methodological limitations of the available evidence.

Method:

The authors performed a meta-analysis of diagnostic accuracy for QEEG in depressive illness, based on articles published between January 2000 and November 2017. The review included all articles that used QEEG to predict response during a major depressive episode, regardless of patient population, treatment, or QEEG marker. The primary meta-analytic outcome was the accuracy for predicting response to depression treatment, expressed as sensitivity, specificity, and the logarithm of the diagnostic odds ratio (DOR). Raters also judged each article on indicators of good research practice.

Results:

In 76 articles reporting 81 biomarkers, the meta-analytic estimates showed sensitivity 0.72 (0.67–0.76), specificity 0.68 (0.63–0.73), log(DOR) 1.89 (1.56–2.21), and area under the receiver-operator curve 0.76 (0.71–0.80). No specific QEEG biomarker or specific treatment showed greater predictive power than the all-studies estimate in a meta-regression. Funnel plot analysis suggested substantial publication bias (arcsine asymmetry test, t=6.33, p=2.64e-8). Most studies did not use ideal practices.

Conclusions:

QEEG does not appear clinically reliable for predicting depression treatment response due to under-reporting of negative results, a lack of out-of-sample validation, and insufficient direct replication of prior findings. Until these limitations are remedied, QEEG is not recommended for guiding psychiatric treatment selection.

Introduction

Major depressive illness remains a leading worldwide contributor to disability, despite the growing availability of medications and psychotherapies (1). The persistent morbidity is partly due to the difficulty of treatment selection. An adequate “dose” of cognitive-behavioral therapy for depression is 10–12 weeks (2). An antidepressant or augmenting medication trial requires at least 4 weeks at an adequate dose (2). Patients may spend months to years searching through options before responding (3). Knowing sooner whether a treatment will be effective could increase the speed and possibly the rate of overall treatment response. The high potential value of treatment-prediction biomarkers has spurred extensive research. Unfortunately, it has also encouraged commercial ventures that market predictive tests to both patients and physicians, marketing that often is not supported by evidence of clinical efficacy (4). Inappropriate use of invalid “predictive” tests could easily increase health care costs without benefiting patients (5).

Predictive biomarker research emphasizes pre-treatment and “treatment emergent” biomarkers. Treatment-emergent markers are physiologic changes that precede and predict the response to effective treatment. They may represent physiologic processes mediating the clinical response, whereas pre-treatment markers may represent moderating factors. If at 1–2 weeks into a treatment trial, we could confidently predict its (non)efficacy, we could move much more quickly through clinical decision trees. For novel therapies such as brain stimulation, treatment emergent markers could also guide “closed loop” treatment, where an aspect of the stimulation is titrated in direct response to the physiologic marker (6,7).

Electroencephalography (EEG) is a promising source of psychiatric biomarkers. Unlike serum chemistry or genetic variation, EEG directly measures brain activity. EEG is potentially more cost-effective than neuro-imaging techniques, such as functional magnetic resonance imaging (fMRI) or nuclear medicine computed tomography (PET/SPECT), that have also been proposed as biomarkers (8–10). EEG recordings can be more feasibly implemented in a wide variety of clinical settings. Finally, EEG has essentially no safety concerns, whereas PET involves radiation and MRI cannot be used in the presence of metal foreign bodies.

Psychiatric biomarker studies have emphasized “quantitative EEG”, or QEEG (see Box 1). Baseline and treatment-emergent biomarkers, as qualitatively reviewed by recent authors (11–13), include simple measures such as loudness dependence of auditory evoked potentials (LDAEP) (14–22), oscillatory power in the theta and alpha (see Box 1) ranges (14,23–39), and the distribution of those low-frequency oscillations over the scalp (35,37,40–45). With the increasing power of modern computers, biomarkers involving multiple mathematical transformations of the EEG signal became available. These include a metric named “cordance” (23,26,46–57) and a proprietary formulation termed the “Antidepressant Treatment Response” (ATR) index (57–61). Each is based on both serendipitous observations and physiologic hypotheses of depressive illness (11,12). LDAEP is believed to measure serotonergic function, oscillations are linked to top-down executive functions (62,63), and cordance may reflect cerebral perfusion changes related to fMRI signals. ATR and related multi-variate markers (64–66) merge these lines of thought to increase predictive power. Recent studies (including CAN-BIND, iSPOT-D, and EMBARC) have sought to create large multi-center datasets that might allow more robust biomarker identification (40,67–72).

Box 1 Basics of EEG Terminology and Biomarkers.

• Montage: placement of individual sensors (electrodes) on a patient’s scalp. The most common is the “International 10–20 System”, but many alternatives exist, particularly as the number of sensors increases above 64.

• Quantitative EEG (QEEG): analysis of EEG through standardized and reproducible mathematical algorithms, as opposed to the visual inspection more common in neurologic diagnosis.

• Alpha, Theta, Beta, Gamma: patterns of rhythmic (sine-wave-like) electrical activity believed to be important for cognition and brain network coordination. Each occurs at a specific frequency (cycles per second, or Hz): 5–8 Hz for theta, 8–15 Hz for alpha, 15–30 Hz for beta, and above 30 Hz for gamma. The definitions are not exact and the boundaries of each “band” vary between authors.

• Evoked Potential: the average brain response to a repeated stimulus, e.g., a pure tone played 100 times. Averaging across the individual presentations (trials) removes background noise, identifying the common/repeatable component.

• Source Localization: applying mathematical transformations that estimate which brain regions likely gave rise to the electrical activity recorded at the scalp. This “inverse problem” has infinite solutions, and many algorithms have been proposed to narrow this to a single “best” answer.

• Cordance: A measure combining multiple mathematical transforms of EEG power across electrodes, often in the prefrontal cortex. Theorized to measure activity related to cerebral perfusion.

Despite the rich literature, the value of QEEG as a treatment response predictor in depressive illness remains unclear. This is in part because there has been no recent meta-analysis aimed at the general psychiatrist or primary care practitioner. The last formal American Psychiatric Association position statement on EEG was in 1991 (73), at which time personal computers were thousands of times less capable than a modern desktop. A 1997 American Academy of Neurology report (74) focused on QEEG in epilepsy and traumatic brain injury. The most recent report, from the American Neuropsychiatric Association, was similarly cognition-oriented (75). Most importantly, all of these are over a decade old. Recent reviews delved into the neurobiology, but did not quantitatively assess QEEG’s predictive power (11–13). The closest was a 2011 meta-analysis that combined imaging and EEG to assess the role of the rostral cingulate cortex in MDD (76). To fill this gap in clinical guidance, we performed a meta-analysis of QEEG as a predictor of treatment response in depression. We cast a broad net, considering all articles on adults with any type of major depressive episode, receiving any intervention, and with any study design or outcome scale. This approach broadly evaluated QEEG’s utility, without being constrained to specific theories of depression or specific markers. We complemented that coarse-grained approach with a meta-regression investigating specific biomarkers, to ensure that inconsistent results across the entire QEEG field would not mask a single effective marker.

Methods

Our review focused on two primary questions:

• 1)What is the overall evidence base for QEEG techniques in predicting (non)response in the treatment of depressive episodes?
• 2)Given recent concerns about reliability in neuro-imaging (10,77), how well did published studies implement practices that support reproducibility and reliability?

We searched PubMed for articles related to EEG, major depression, and response prediction (Supplementary Material). We considered articles published in any indexed year. From these, we kept all that reported prediction of treatment response, to any treatment, in any type of depressive illness, using any EEG metric. Our prospective hypothesis was that EEG cannot reliably predict treatment response. We chose broad inclusion criteria to maximize the chance of a signal detection that falsified our hypothesis. That is, we sought to determine whether there is sufficient evidence to recommend the routine use of any QEEG approach to shape psychiatric treatment. This is an important clinical question, given the commercial availability and promotion of psychiatric QEEG (e.g., http://www.myndanalytics.com/, http://cerescan.com/). We did not include studies that attempted to directly select patients’ medication based on an EEG evaluation, an approach sometimes termed “referenced EEG” or “rEEG” (78). rEEG is not a diagnostic test, and as such does not admit the same form of meta-analysis.

The meta-analysis of diagnostic markers depends on 2×2 tables summarizing correct/incorrect responder/non-responder predictions (79). Two trained raters extracted these from each article, with discrepancies resolved by discussion and final arbitration by ASW. Where necessary, table values were imputed from other data present in the article (Supplementary Material). For articles with more than one marker or treatment (19,29,52,57,60,67,80), we considered them as separate studies. We reasoned that treatments with different mechanisms of action (e.g., rTMS vs. medication) may have different effects on reported biomarkers, even if studied by a single investigator. For studies that reported more than one method of analyzing the same biomarker (23,34,57), we used the predictor with the highest positive predictive value (PPV). This further increased the sensitivity and the chance of a positive meta-analytic result. Articles that did not report sufficient information to reconstruct a 2×2 table (14,15,17,21,25,28,32,33,42,43,81–92) were included in descriptive and study-quality reporting, but not the main meta-analysis.

For quality reporting, we focused on whether the study used analytic methods that yield more reliable conclusions. Chief among these is independent sample verification or “cross validation” – reporting the algorithm’s predictive performance on a sample of patients separate from those originally used to develop it. Cross validation has repeatedly been highlighted as essential to developing a valid biomarker (10,11,61,74,93). Our two other markers of study quality were total sample size and correction for multiple hypothesis testing. Small sample sizes falsely inflate effect sizes (93), while multiple-testing correction is a foundation of good statistical practice.

We conducted univariate and bivariate meta-analyses using R’s “mada” package for analysis and “metafor” for visualizations (94–96). The univariate analysis summarized each study as the natural logarithm of its diagnostic odds ratio (DOR), using a random-effects estimator (79). Bivariate analysis used sensitivity and specificity following the approach of Reitsma et al (97). From the bivariate analysis, we derived the area under the summary receiver-operator curve (AUC), and computed an AUC confidence interval by 500 iterations of bootstrap resampling with replacement. For the univariate analysis, we report I² as a measure of study heterogeneity. As secondary analyses, we separated studies by biomarker type (LDAEP, power features, ATR, cordance, and multivariate) and by treatment type (medication, rTMS, or other). These were then entered as predictor variables in bivariate meta-regressions. Finally, to assess the influence of publication bias, we plotted of log(DOR) against its precision, expressed as both the standard error (funnel plot) and the effective sample size (98). We tested funnel plot asymmetry with the arcsine method described in Rücker et al.(99), as implemented in the “meta” package (100). This test has been suggested to be robust in the presence of heterogeneity and is the recommended choice of a recent working group (101). All of the above were pre-planned analyses. Our analysis and reporting conforms to the PRISMA guidelines (102); please see Supplementary Material for the checklist. The Supplementary Material also reports an alternative approach using standardized mean differences.

Results

Descriptive Study Characteristics

We identified 995 articles (Figure 1). 90 of these appeared to discuss response prediction. 76 articles, covering 81 biomarkers, were eligible for descriptive analysis. Of these, 53 articles, discussing 57 biomarkers, included sufficient information for meta-analysis. The majority of articles that did not include sufficient 2×2 table information still reported a statistically significant result (22/24, 91.7%).

Figure 1,

PRISMA diagram for meta-analysis of QEEG biomarkers in depression treatment.

Studies varied in the degree of treatment-resistance, included/excluded diagnoses, details of EEG recording, and specific analytic/statistical approach (Table S1). 70% (57/81) were studies of response to medication, with most of the remaining (14/81, 17%) predicting response to rTMS. Citalopram/escitalopram and venlafaxine were the most commonly studied medications, representing 23% (13/57) and 19% (11/57) of medication studies, respectively. Most reported markers were from resting state EEG (70%, 57/81) and did not source localize the EEG data (79%, 64/81). The most heavily represented biomarkers were low-frequency EEG power (31%, 25/81) and cordance (19%, 15/81).

No study was a pre-planned, independent-sample replication of a prior investigation with identical medication regimens and outcome measures. A few markers, however, were studied repeatedly with similar designs. Three LDAEP studies attempted to predict response to citalopram (19,21,103). They had inconsistent results that appeared to be dependent on source-localization technique. Olbrich and colleagues used a “vigilance” marker that had been validated (using different recording/analysis methods) in smaller prior datasets (67). Cook and colleagues used cordance to predict the response to varying medication protocols, but a series of their studies found better-than-chance prediction using the same equipment, outcome measures, and decision rule (cordance decrease at 1 week of treatment) (26,54,55). Bares and colleagues used different populations (bipolar depression vs. MDD), treatments, and response definitions, but also repeatedly reported successful response prediction with a 1-week cordance decrease (47,50,51). A pair of studies with a relatively large sample size found that the ATR predicted response to different medications (59,60). These studies were based on earlier reports of cordance and power biomarkers by the same researchers using different medication regimens (104). The larger ATR studies reported a slight modification of a previously-unpublished version of ATR (4.1 in the study reports vs. 4.0 in the trial protocol and prior poster presentations). Widge et al. reported that the same version of ATR did not predict response to rTMS (61). Reports from the iSPOT-D study were hypothesis-driven and meant to test biomarkers that had previously been published, although they used a different medication protocol (25,40,69). Finally, theta power source-localized to the anterior cingulate cortex was reported by multiple labs as a predictor of response to different monoaminergic medications (14,29,31). A recent report from the EMBARC study (which did not report information necessary for meta-analysis) also found cingulate theta to predict antidepressant response, although theta changes did not differ between patients receiving sertraline and those receiving placebo (105).

Study Quality

Study sizes were generally small (median n=25). The distribution was tri-modal (Figure S1), with peaks at approximately N=20, N=85, and N=660. The last reflects reports from the recently-concluded iSPOT-D study (40,67,69).

Most studies did not meet the quality metrics. 40 studies reported testing only a single EEG feature or finding no significant results, and thus did not require multiple-comparisons corrections. Of the 36 that tested multiple features, 67% (24/36) did not report use of a statistical correction. Of 71 markers reported to have significant predictive validity, only 6 (8%) were studied with cross-validation or another out-of-sample verification. 3 of these were from the same first author (106–108). One article reported using cross-validation, but did not include cross-validated algorithm performance in its main text or abstract (60).

Overall Efficacy

For all biomarkers taken together, the meta-analysis suggested predictive power above chance (Figure 2). The meta-analytic estimate of sensitivity was 0.72 (0.67–0.76), specificity 0.68 (0.63–0.73), and log(DOR) 1.89 (1.56–2.21). These correspond to an AUC of 0.76 (0.71–0.80). The univariate analysis did not suggest study heterogeneity as a driver of results (I²=0%, Q=55.9, p(Q)=0.48). This implies that in general, QEEG may have predictive power for treatment response in depressive illness. No biomarker or treatment type showed significantly greater predictive power than another. In bi-variate meta-regressions (Tables S3-S4), the Akaike Information Criterion (AIC) increased from its omnibus value of −115.7 to −104.1 for a model split by biomarker type and to −107.7 for a model split by treatment type. Increases in AIC imply that model terms have no true explanatory power (109). This is further supported by most meta-regression model coefficients failing to reach significance. We considered the possibility that these reflect older studies identifying incorrect candidates, with newer studies honing in on true effects. A bivariate meta-regression of diagnostic accuracy against publication year showed no effect (p>0.27, Z-test on regression coefficients).

Figure 2:

Meta-analytic results. Sensitivity (A), specificity (B), and log(DOR) (C) are for prediction of clinical antidepressant response based on QEEG biomarkers are presented as forest plots. Meta-analytic estimates show modest predictive power for clinical response. This is also visible in a summary ROC curve (D), where the area under the curve is estimated as 0.76.

Funnel-plot analysis suggested that QEEG’s apparent predictive power is driven by small studies with strong positive results. The plot was specifically depleted in studies with smaller effect sizes that may not have reached pre-specified significance thresholds (Figure 3A), and there was a tight correlation between effect size and the reciprocal of effective sample size (Figure 3B). The arcsine test for funnel plot asymmetry rejected the null hypothesis (t=6.33, p=4.64e-8).

Figure 3:

Meta-analytic results are influenced by publication bias. (A), the funnel plot of study effect size (log of DOR) vs. standard error of that effect size. Dashed lines represent the meta-analytic estimate and its 95% confidence interval. Small studies with effect sizes between 0 (no effect) and approximately 2 (modest effect), are under-represented. (B), scatter plot of effect size (log of DOR) against the reciprocal of effective sample size, showing that the two are linearly related. The overlaid line is a robust linear regression fit. The association of effect size with study size holds across biomarker types, reflected here by different shapes/markers.

Discussion

QEEG is commercially promoted to psychiatrists and our patients as a “brain map” for customizing patients’ depression treatment. Our findings indicate that QEEG, as studied and published to date, is not well-supported as a predictive biomarker for treatment response in depression. Use of commercial or research-grade QEEG methods in routine clinical practice would not be a wise use of healthcare dollars. This conclusion is likely not surprising to experts in QEEG, who are familiar with the limitations of this literature. It is important, however, for practicing psychiatrists to understand given the availability of QEEG as a diagnostic test. At present, marketed approaches do not represent evidence-based care. This mirrors other biomarker fields such as pharmacogenomics and neuro-imaging, where recent reviews suggest that industry claims substantially exceed the true evidence base (4,110). Like those markers, QEEG may become clinically useful, but only with further and more rigorous study.

We showed that the QEEG literature generally describes tests with reasonable predictive power for antidepressant response (sensitivity 0.72, specificity 0.68). This apparent utility, however, may be an artifact of study design and selective publication. We observed a strong funnel plot asymmetry, indicating that many negative/weak studies are not in the published literature. Of those that were published, many have small sample sizes. Small samples inflate effect sizes, which may give false impressions of efficacy (111). This is doubly true given the wide range of options available to EEG data analysts, which can lead to inadvertent multiple-hypothesis testing (93). We also identified a common methodological deficit in the lack of cross-validation, which could over-estimate predictive capabilities. Taken together, the findings suggest that community standards in this area of psychiatric research do not yet enforce robust/rigorous practices, despite recent calls for improvement (11,77,93). Our results indicate that QEEG is not ready for widespread use. Cordance and cingulate theta power are closest to proof of concept, with studies reporting successful treatment prediction across different medication classes and study designs (14,29,31,47–49,51,105). ATR has been successful across medication classes, but only when tested by its original developers (58,59). A direct and identical replication of at least some of those findings is still necessary. These design and reporting limitations suggest that QEEG has not yet been studied or validated to a level that would make it reliable for regular clinical use.

We designed this meta-analysis for maximum sensitivity, because we sought to demonstrate QEEG’s lack of maturity as a biomarker. This makes our omnibus meta-analytic results overly optimistic, and obscures three further limitations of QEEG as a response predictor. First, we accepted each individual study’s definition of the relevant marker, without enforcing consistent definitions within or between studies. For example, “alpha” EEG has been defined differently for different sets of sensors within the same patient (34) and at different measurement timepoints (60). Enforcing consistent definitions would attenuate the predictive signal, because it reduces “researcher degrees of freedom” (77). On the other hand, an important limitation of our meta-analysis is that it could not identify a narrow biomarker. If QEEG can predict response to a single specific treatment, or in a biologically well-defined sub-population, that finding would be obscured by our omnibus treatment. Marker-specific meta-analysis, as in (76), would be necessary to answer that question.

Second, we did not consider studies as negative if they found significant change in the “wrong” direction. For instance, theta cordance decline during the first week of treatment is believed to predict medication response (26,47,48,51,52,55). Two studies reported instead that a cordance increase predicted treatment response (46,53). LDAEP studies have reported responders to have both higher (17,19,20) and lower (15) loudness-dependence compared to non-responders. This could be explained by differences in collection technique, or in the biological basis of the interventions (e.g., the inconsistent study used noradrenergic medication, whereas LDAEP is felt to assess serotonergic tone). It could also be explained by true effect sizes of zero, and modeling these discrepancies differently would reduce our estimates of QEEG’s efficacy.

Third, and arguably most important, depression itself is heterogeneous (6,112). Defining and subtyping it is one of the major challenges of modern psychiatry, with many proposals for possible endophenotypes (6,9,12,113,114). When we consider that each primary study effectively recorded from many different neurobiological entities, the rationale for QEEG-based prediction is less clear. As an example, a recent attempt to validate an obsessive-compulsive disorder biomarker, using the originating group’s own software, showed a significant signal in the opposite direction from the original study (115). Further, studies often predict of antidepressant response for patients receiving medications with diverse mechanisms of action. Considering that patients who do not respond to one medication class (e.g., serotonergic) often respond to another (e.g., noradrenergic or multi-receptor), it does not make sense for any single EEG measure to predict response to multiple drug types. Similarly, although the goal of many recent studies is to explicitly select medication on the basis of a single EEG recording (40,70,72,78), this may not be possible given the many ways in which neurotransmitter biology could affect the EEG. Reliable electrophysiologic biomarkers may require “purification” of patient samples to those with identifiable circuit or objective behavioral deficits (6,116) or use of medications with simple receptor profiles. It may also be helpful to shift from resting-state markers to activity recorded during standardized tasks (6), as a way of increasing the signal from a target cortical region. Task-related EEG activity has good test-retest reliability, potentially improving its utility as a biomarker (71).

We stress that our meta-analysis means that QEEG as currently known is not ready for routine clinical use. It does not mean QEEG research should be stopped or slowed. Many popular QEEG markers have meaningful biological rationales. LDAEP is strongly linked to serotonergic function in animals and humans (117). Cordance was originally derived from hemodynamic measures (11,54). Neither cordance nor ATR changed substantially in placebo responders, even though both changed in medication responders (26,59,117). The theta and alpha oscillations emphasized in modern QEEG markers are strongly linked to cognition and executive function (62,118). Our results do not imply that QEEG findings are not “real”; they call into question the robustness and reliability of links between symptom checklists and specific aspects of resting-state brain activity. If future studies can be conducted with an emphasis on rigorous methods and reporting, and with specific attempts to replicate prior results, QEEG still has much potential.