Towards robust anxiety biomarkers: A machine learning approach in a large-scale sample

Abstract

Background

The field of psychiatry has long sought biomarkers that can objectively diagnose patients, predict treatment response, or identify individuals at risk of illness onset. However, reliable psychiatric biomarkers have yet to emerge. The recent application of machine learning techniques to develop neuroimaging-based biomarkers has yielded promising preliminary results. However, much of the work in this domain has not met best practice standards from the field of machine learning. This is especially true for studies of anxiety, creating uncertainty about the potential for anxiety biomarker development.

Methods

We applied machine learning tools to predict trait anxiety from neuroimaging measurements in humans. Using publicly available data from the Brain Genomics Superstruct Project, we compared a suite of neuroimaging-based machine learning models predicting anxiety within a discovery sample (n=531, 307 female) via k-fold cross-validation, and we tested the final model (a stacked model incorporating region-to-region functional connectivity, amygdala seed-to-voxel connectivity, and volumetric and cortical thickness data) in a held out, unseen test sample (n=348, 209 female).

Results

Though the best model was able to predict anxiety within the discovery sample (cross-validated R² of .06, permutation test p<.001), the generalization test within the holdout sample failed (R² of −.04, permutation test p>.05).

Conclusions

In this study, we did not find evidence of a generalizable anxiety biomarker. However, we encourage other researchers to investigate this topic, utilizing large samples and proper methodology, in order to clarify the potential of neuroimaging-based anxiety biomarkers.

Introduction

Biomarkers are objective, reproducible biological measures of medical state (1). Biomarkers can perform an invaluable function by informing treatment plans or indicating the presence, prognosis, or risk level of disease. For example, doctors test for elevated cardiac troponin to assess whether a heart attack has occurred (2) and determine treatment course (3). AIDS is defined by CD4 (T cell) count, (4) and CD4 count is used to gauge opportunistic infection risk (5). The identification of psychiatric biomarkers ready for use in clinical practice has been elusive. Recently, there has been a focus on developing psychiatric biomarkers from neuroimaging data. Thousands of studies have aimed to identify brain-based differences between patients with mental illnesses and mentally healthy patients, and authors often speculate that the neural differences identified could form the basis of a biomarker. However, clinically useful neuroimaging-derived biomarkers have not emerged (6–9).

It has been argued that progress in neuroscience, psychology, and psychiatry could be advanced by placing more emphasis on prediction, rather than explanation alone (8–12). Psychiatric neuroscience has tended to prioritize explanation by favoring theory-driven, tightly controlled studies (often with small samples). Understanding the mechanisms of a psychiatric disorder would clearly advance the ability to develop tests and treatments. However, maximizing prediction of a clinical variable is not usually an explicit goal of these studies. A more direct emphasis on prediction of clinical outcomes may speed psychiatric translation and engender reproducible science.

Traditionally, neuroimaging studies aiming to identify biomarkers tested for significant differences in the means of populations with and without the disorder in a given measure(s) like amygdala-PFC connectivity, or BOLD signal in individual voxels during a task. This work has provided a foundational step in highlighting the areas of the brain where the differences between patients and non-patients are most striking. However, a significant difference in mean between populations does not indicate that an individual could be classified with meaningful accuracy on the basis of that variable in isolation. There may be enough overlap between the populations that one cannot predict illness status with sufficient reliability (12–17).

Recently, there has been interest in using machine learning to develop biomarkers, as machine learning provides many tools that can complement traditional statistical approaches. In machine learning, prediction is typically valued over explanation. In this paper we focus on supervised machine learning models. A model, in this context, is a function that transforms input variables, or “features,” as they are referred to in machine learning, into a prediction of a “target variable,” like patient group or symptom score, by learning the relationship between the features and target variable. Machine learning models are typically multivariate—they leverage the combined effects of many variables to predict group membership, potentially allowing for greater predictive power than any individual predictor (18, 19).

Researchers have applied machine learning tools to neuroimaging measurements in order to differentiate patients and controls, with promising but mixed success (15, 20). We identified 23 papers that have used machine learning to predict anxiety status or traits from neuroimaging data (selection criteria in the Supplement, study characteristics in Table S1) (21–42). Most studies classifying patients versus controls reported accuracies of over 80%, and some over 90%. Below we review methodological considerations important to the interpretation of these studies.

Training and testing a model on different participants, in order to account for overfitting, is fundamental to the practice of machine learning (18, 19). Overfitting is the phenomenon that a model may be able to predict the target variable from the training data (data used to learn the parameters of the model) very well, even perfectly, but fail to perform well on novel examples (test data) that were not used to train the model. It is necessary to apply the model to previously unseen test data to evaluate its performance. This can be done with cross-validation, where the data are iteratively split into training and test sets, with training data used to fit the model and test data used to evaluate it. The 23 studies of anxiety neuromarkers reviewed tended to use cross-validation to assess predictive performance. However, to assess the model’s generalizability, it is crucial, especially if multiple models or variants of the analysis are tested with cross-validation, to additionally test the final model on a completely held out dataset, which only two of the reviewed studies did (25, 40). If the researchers use cross-validation multiple times on the same dataset to assess different types of classifiers, different feature types, or different model hyperparameters (without nested cross-validation), and pick the best result to include or emphasize in the manuscript, the cross-validation accuracy is no longer an unbiased estimate of generalization performance. Stated differently, the researcher risks overfitting via the model selection process (9, 15, 16, 43). There are many examples within the broader field of machine learning-based psychiatric neuroimaging where performance on held out datasets was substantially worse than that obtained by internal cross-validation, suggesting that it is dangerous to assume cross-validation accuracy is an unbiased assessment of how the model will perform on new data (25, 44–46). Using cross-validation alone to assess models is risky because cross-validation accuracy is a quite variable estimator of generalization, particularly with small samples (47). So while cross-validation or some initial validation is a necessary first step for any machine learning model, it is strongly recommended to additionally test the model on a dataset that has been completely held out throughout the analysis process (ideally, an external dataset). However, likely due to the high cost of acquiring data, most of the studies reviewed did not perform a holdout test.

Another limitation of the 23 reviewed studies is small sample size. With the exception of 3 studies that used the Human Connectome Project (HCP) sample (48), none of the studies had an n>181, and most had fewer than 100 participants. The machine learning literature emphasizes the importance of a large sample size. The amount of data available to a model often (but not always) has more influence on success than algorithm choice (49). Data are more easily overfit when the sample size is small, and this includes “procedural” overfitting by testing multiple methods with cross-validation. In neuroimaging datasets, the number of features typically vastly outnumbers the number of subjects, which can also make models more prone to overfitting (10, 43, 47). Due to technical and cultural changes within the field of neuroscience (50, 51), there has been a shift towards collection of large, cross-site neuroimaging datasets and concatenation of existing datasets (e.g., 48, 52–55). This commendable effort will likely be crucial in the emergence of biomarkers. Of course, “large” and “small” are relative terms—machine learning applications in natural language processing or image processing often involve millions (or more!) of samples (56), but by neuroimaging standards, an N approaching or exceeding 1000 is considered large (57).

Three studies that explored trait anxiety prediction utilized the HCP sample (40–42), the largest sample yet studied to answer this question. Two of the three reported the ability to reliably predict anxious personality/neuroticism. Though the HCP studies make unique contributions to the literature, they should not be considered independent evidence because they use the same sample. One study (40) took a particularly compelling approach by performing holdout tests (some successful) with 2 independent datasets.

The present study pursued the development of a neuroimaging-based anxiety biomarker with machine learning tools, utilizing a large sample and testing the proposed model on a completely held out dataset. As reviewed above, this question has only been addressed with one large sample previously. Additionally, nearly all prior reports on this topic have relied solely on cross-validation, and it is unclear whether results will generalize to truly unseen samples. We tested whether trait anxiety could be predicted from neuroimaging measurements with a suite of machine learning algorithms, using a large, publicly available sample. We first considered models of whole-brain, region-to-region resting-state functional connectivity data. Subsequently, we explored the utility of adding grey matter volumetric/thickness measurements and region-to-voxel connectivity data as features. We performed all model comparison within a discovery sample, and tested our final model on a held out dataset. A dimensional approach to studying behavioral systems relevant to psychiatric disorders may be fruitful in linking biology and mental illness (58–60); thus we chose to predict variation in anxiety in a non-clinical sample. We note that our model does not produce true “predictions”, in the sense that the target variable was a measure of current anxiety rather than future anxiety. However, we view this study as an important analytical stepping-stone in the development of pragmatic clinical tools—if challenges arise in predicting current anxiety, the same challenges may be present in predicting future anxiety.

Methods and Materials

Dataset

The data are from the Brain Genomics Superstruct Project (GSP), a large-scale, multi-site brain imaging project (55). The publically released GSP dataset consists of resting-state fMRI and structural MRI scans of 1570 participants. Self-report and behavioral data are available for a subset of participants (n=926). The Supplement details motion- and coverage-related exclusions (47 participants). Data collection and sharing were approved by the Partners Health Care Institutional Review Board and the Harvard University Committee on the Use of Human Subjects in Research.

Data were first split into a discovery sample (n=531 after exclusions), and a final model evaluation sample (referred to as the “holdout” sample, n=348 after exclusions). The holdout neuroimaging data were sequestered (not downloaded) until the final model was tested. The two samples did not differ in age, sex, level of education, estimated IQ, anxiety score, number of runs, motion statistics, site of acquisition, and console software (ps>.32).

Target variable

The target variable was a composite anxiety score derived from several questionnaires administered through an online battery (55). We used a composite score with the rationale that it would be more stable and less idiosyncratic than any individual anxiety-related scale. Four anxiety-related questionnaires were collected for all participants: the Spielberger Trait Anxiety Inventory (61), the NEO-neuroticism scale (62), the Behavioral Inhibition Scale (63), and the Harm Avoidance scale from the Temperament and Character Inventory (64). The composite anxiety score was derived (following 65) by z-scoring each of these 4 scales across participants and taking the mean of these 4 z-scores per participant. In computing the composite anxiety scores for the holdout sample, we performed z-score transformations based on the means and standard deviations of the discovery sample. In the Supplement, we report results for individual scales for the best performing model.

Neuroimaging data collection and preprocessing

Imaging sequences are described in the Supplement. FMRI data were preprocessed with FMRIB Software Library (FSL) 5.0.9 (66) and FreeSurfer 6.0 (http://surfer.nmr.mgh.harvard.edu/) using standard methods for resting-state functional connectivity analysis (see Supplement). Structural data was processed, and region-wise volume and cortical thickness measurements were extracted in FreeSurfer (see Supplement).

Functional connectivity measures

In the model building/selection phase, working with the discovery sample, we evaluated 6 methods for parceling the brain into regions from which to derive connectivity measurements. Table S2 lists the 6 parcellations evaluated. The best performing method in the discovery sample was the FreeSurfer segmentation (67, 68).

To construct region-to-region connectivity features, we extracted the mean BOLD timecourse from each region in the parcellation. We computed the Pearson’s correlation in signal between each pair of regions (transformed with Fisher’s r-to-z). These z-values were used as features in the models.

Some of the stacked models utilized voxelwise connectivity maps. Each voxel’s connectivity to a given seed region was used as a feature. These maps were generated with FSL using FreeSurfer-defined seed regions. See the Supplement for additional details on functional connectivity measures.

Modeling/model selection

Modeling was carried out in python with the scikit-learn package (69). We used R², calculated on the test data, as an evaluation metric (see Supplement for discussion of R²).

Within the discovery sample, each model was constructed and evaluated with stratified k-fold cross-validation (k=6). For several of the models tested, we tuned a hyperparameter of the model with nested cross-validation (see Supplement for further description of modeling, cross-validation and hyperparameter tuning). Various model classes were evaluated in the discovery sample, including Ridge Regression, Lasso Regression, Partial Least Squares Regression, Principal Components Regression, Random Forest Regression, Support Vector Regression with a linear or polynomial kernel, Relevance Vector Regression, and the “Connectome-based Predictive Modeling” approach (70–72). We also attempted to specifically replicate methods from prior trait anxiety prediction studies (see Supplement). Table S3 lists models tested and hyperparameters tuned.

Several models evaluated in the discovery sample were stacked models. Model stacking is a method of combining predictions from several models (referred to as base models), by building a model where the predictions of base models serve as features. We combined models that made predictions based on different data sources, such as region-to-region connectivity, structural MRI data, and voxelwise connectivity data (see Supplement and Figure S1 for detailed explanation of model stacking.)

To assess the significance of the R² observed in the best model (the model with the highest R²), we used permutation testing (see Supplement).

To address the possibility that the model could be learning to predict some confound rather than anxiety scores, we performed a control analysis in the discovery sample with the best model, in which we regressed out potential confounds from the features. We also performed an analysis on censored data (see Supplement).

We tested whether the model that had performed best in the discovery sample could predict anxiety in the holdout sample. We retrained this best performing model using the entire discovery sample, and generated predictions of this model for the unseen holdout data. We computed the R² with these holdout predictions, and assessed significance with a permutation test.

Data and code availability

The imaging data are publically available at (http://neuroinformatics.harvard.edu/gsp/). Some data, including anxiety-related questionnaires, require approval for access (instructions: http://neuroinformatics.harvard.edu/gsp/get). Code used to run the analyses is available upon request.

Results

Sample Characteristics

Across the discovery and holdout samples, participants had a mean age of 21.59 (SD 2.87) ranging from 18–35. Age in the public GSP release is binned by 2 years to protect the privacy of participants, so the mean and SD are not exact. The sample was 59% female. To illustrate the range of anxiety-like phenotypes present in the sample, Figure 1 shows a histogram of scores on the Spielberger Trait Anxiety Index. For reference, this figure illustrates the mean Spielberger Trait Anxiety scores of patient samples with anxiety disorders or PTSD in recent studies. It is apparent from the figure that the GSP participants are not from a clinical sample, but some participants do show levels of anxiety comparable to patient populations.

Figure 1. Sample heterogeneity in trait anxiety.

Distribution of Spielberger Trait Anxiety Index (STAI-T) in the complete sample analyzed in this paper (n=879), and in recent clinical studies of anxiety (n=12). Filled bars show the scores in the current sample. Empty bars show mean scores of samples with anxiety disorders or PTSD from recent studies (85–96).

Discovery sample model performance

The model producing the greatest cross-validated R² (within the discovery sample) was a stacked model that incorporated region-to-region connectivity data, volumetric/cortical thickness data, and voxelwise bilateral amygdala connectivity data, model 1 from Table S3. The regions used as seeds for the region-to-region and amygdala voxelwise connectivity features came from the FreeSurfer segmentation. This model resulted in a cross-validated R² of .06. Figure 2A shows the relationship between actual anxiety scores and predicted anxiety scores. This level of performance significantly exceeded chance levels (p<.001, as determined by a permutation test, Figure 2B.) Performance of other models tested in the discovery sample is summarized in Table S3. Anxiety could also be predicted from censored connectivity data, and data that had undergone confound regression (see Supplement).

Figure 2. Model performance in the discovery sample.

A. Actual anxiety scores plotted against predicted anxiety scores, in the discovery sample. Model predictions are from the best performing model, model 1 (see Table S3). B. Empirical null distribution of R² generated in permutation test, in the discovery sample. The dotted black line shows the 95% confidence bound. The solid red line shows the actual R² of the model using the (unscrambled) data.

Holdout sample test

We tested the best model in the holdout sample after training it with the entire discovery sample. The R² in the holdout sample was −.04, which failed to achieve significance with a permutation test (Figure 3).

Figure 3. Model performance in the holdout sample.

A. Actual anxiety scores plotted against predicted anxiety scores, in the holdout sample. B. Empirical null distribution of R² generated in permutation test, in the holdout sample. The dotted black line shows the 95% confidence bound. The solid red line shows the actual R² of the model using the (unscrambled) data.

One possible explanation for the poor performance in the holdout dataset is that by testing so many models within the discovery sample (see Table S3), we may have overfit the model to the discovery sample through the model selection process. To attempt to understand the reason for the generalization failure, we tested this hypothesis by examining whether a model that was tested early in the model comparison process (with an R² of .03 in the discovery sample), a ridge regression from region-to-region connectivity data (model 4 from Table S3), outperformed the final model. This ridge model that had been tested early in the model exploration process did not outperform the final model (R² of −.06 in the holdout sample).

Discussion

In this study, we attempted to predict trait anxiety in a large sample by applying machine learning tools to multimodal neuroimaging data. The best model (determined in the discovery sample) was a stacked model, with three ridge regression base models which used different data sources as features: whole brain region-to-region connectivity data, amygdala seed-to-voxel connectivity data, and grey matter volumetric/thickness data. This model significantly predicted anxiety scores in the discovery sample as assessed by cross-validation, but when tested on a previously unseen holdout sample, it did not successfully predict anxiety scores. Our ability to predict anxiety within the discovery sample is consistent with prior work (21–41). However, when we tested generalizability to a holdout sample, a step most previous studies did not take, the model failed to make accurate out-of-sample predictions. Thus, our findings do not support the hypothesis that anxiety is predictable from neural measurements.

We studied a limited set of brain phenotypes and applied a circumscribed set of approaches. Our study should be considered a proof-of-concept for evaluating relations linking brain functions to behavior rather than decisively addressing the full range of possible associations between neuroimaging measures and anxiety. There are multiple possible explanations for why we were ultimately unable to predict anxiety scores.

One set of possible reasons for our failure to predict anxiety relates to the anxiety phenotype examined. We did not analyze a clinical sample, so perhaps there was not sufficient clinical heterogeneity for the model to learn to make accurate predictions. As shown in Figure 1, there were participants with anxiety scores close to the mean scores of anxiety patients in clinical studies, but the number of participants with scores in this range was relatively small, and perhaps insufficient to train the model. Though the sample used here was large in comparison to those from most fMRI studies, larger samples exist, such as the UK Biobank sample (52), which includes over 10,000 subjects. Prediction might be improved in these larger samples, as the quantity of training data is an important determinant of model performance, and as there might be more subjects in a high anxiety range to inform the model. We encourage other researchers to investigate this question in large open-access samples. However, despite the lack of participants in this high range in the GSP sample, there is still substantial variability, and one would expect that this healthy heterogeneity would be predictable from neural measurements. Additionally, anxiety scores were only assessed once. Multiple assessments might yield a more stable estimate of the phenotype and improve prediction. It has been suggested that the non-biological nature of current diagnostic categories has stymied progress in identifying biological mechanisms of these disorders (6). This argument can be made about continuous variables as well—a measurement may not “carve nature at the joints.” It is likely that our anxiety measure does not reflect a single process; relatedly, two individuals could have the same elevated anxiety score with different underlying brain mechanisms, and this may impair prediction of the score.

Feature-related issues could also have impaired prediction. One limitation to note is that the imaging sequences used lack the spatial and temporal precision of current approaches (data collection began in 2008). It is possible that with more state-of-the-art sequences, prediction would be facilitated. Relatedly, each subject had 6–12 minutes of resting-state data, but recent studies have suggested that substantially improved reliability of connectivity estimates can be obtained with ~15–25 minutes of data (73–75). Others have recently shown that the inclusion of task-based fMRI data can improve connectivity estimates and predictive performance from connectivity data (76, 77). A limitation of the current paper is the unavailability of longer resting-state scans and task-based data in these subjects.

Model selection-related issues could also underlie our failure to predict anxiety in the holdout set. One possibility is that a model exists that could successfully predict anxiety from the measurements obtained, but we did not identify it. However, in the discovery sample, we tested a large range of both linear and non-linear models in combination with different parcellations for extracting connectivity features (see Table S2). Conversely, another concern we had was that we may have tested too many models in the discovery sample, leading to an overfitting of the model selection process to the discovery sample. In other words, a model tested early in the exploration of the discovery sample may have actually outperformed the chosen model when tested on the holdout sample, despite performing worse on the discovery sample. To investigate whether this could be the case, we performed a supplementary test in the holdout sample of a model that had been tested early in the model comparison process. However, this earlier model also failed to accurately predict anxiety scores in the holdout sample. This failure did not support the explanation that we could have obtained better holdout performance had we stopped the model testing in the discovery sample sooner. It does, however, allow for the possibility that anxiety was not predictable from the measurements obtained, but the good performance in the discovery sample was illusory and due to procedural overfitting.

We close by providing suggestions on how to proceed with research on neuroimaging-based psychiatric biomarker development, given our observations in the current study. Previous anxiety biomarker research has tended to use small samples (only one large-scale sample investigated previously (40–42), with mixed results) and evaluate models with cross-validation only. As demonstrated here, it is possible to achieve promising results via internal cross-validation that do not generalize to a held out sample. We therefore recommend that future studies utilize large samples, and test their models on truly unseen holdout data. Heterogeneity in preprocessing and statistical approaches creates problems for interpreting and replicating traditional neuroimaging analyses, but machine learning-based neuroimaging studies arguably suffer from these issues even more. The number of possible models from which to choose is large, methods for assessing generalization vary, and standards of reporting/visualizing feature importance (which also depend on which model is used) are undefined. We therefore recommend further research on methods development that can illuminate best practices (e.g., 78), and that studies attempt to replicate the specific methods of other studies. The continued acquisition of new large samples will also undoubtedly be crucial to biomarker development. One difficulty in this field is that the phenotypes we want to predict may be multidimensional and may not derive from a single biological mechanism (6, 79). We see promise in applying unsupervised machine learning methods to biomarker development, as these methods may circumvent this issue (although see 80). Another class of methods that could help with this issue is multi-output learning, where multiple phenotypes are predicted with the same model (81). This methodology takes advantage of relationships between different target variables (possibly helping to disambiguate cases where subjects have the same anxiety score with different underlying mechanisms) and has been shown to improve upon single-output model predictions (81). Lastly, we note that in our comparisons of models within the discovery sample, a stacked model that combined predictions from multimodal data performed best. Though we interpret this result with caution, as this model did not ultimately successfully predict anxiety, the result is consistent with other neuroimaging biomarker studies suggesting that stacked multimodal models outperform non-stacked models (81, 82).

The potential to develop neuroimaging biomarkers for anxiety is unclear, but some research suggests that success is possible. In this study, we were unable to find evidence of a generalizable anxiety biomarker. Though this research area is proving challenging, some encouraging results have emerged. Outside the field of psychiatry, there have been successful attempts at producing generalizable neuromarkers of psychological states and traits (71, 83, 84). Given the potential of biomarkers to revolutionize psychiatry, it is important to rigorously explore their possible development and application.