Neuroimaging in Psychiatry and Neurodevelopment: why the emperor has no clothes

Abstract

Neuroimaging has been a dominant force in guiding research into psychiatric and neurodevelopmental disorders for decades, yet researchers have been unable to formulate sensitive or specific imaging tests for these conditions. The search for neuroimaging biomarkers has been constrained by limited reproducibility of imaging techniques, limited tools for evaluating neurochemistry, heterogeneity of patient populations not defined by brain-based phenotypes, limited exploration of temporal components of brain function, and relatively few studies evaluating developmental and longitudinal trajectories of brain function. Opportunities for development of clinically impactful imaging metrics include longer duration functional imaging data sets, new engineering approaches to mitigate suboptimal spatiotemporal resolution, improvements in image post-processing and analysis strategies, big data approaches combined with data sharing of multisite imaging samples, and new techniques that allow dynamical exploration of brain function across multiple timescales. Despite narrow clinical impact of neuroimaging methods, there is reason for optimism that imaging will contribute to diagnosis, prognosis, and treatment monitoring for psychiatric and neurodevelopmental disorders in the near future.

The emperor’s Wardrobe

There are no neuroimaging tests in widespread clinical use to diagnose neurodevelopmental or neuropsychiatric disorders. Despite billions of dollars in research spanning decades, there are no sensitive and specific imaging tests that provide diagnostic or prognostic information to guide clinicians and their patients. Neuroimaging has been used to study a variety of mental illnesses such as depression, schizophrenia, bipolar disorder, autism, attention deficit hyperactivity disorder (ADHD), obsessive compulsive disorder, Tourette Syndrome, anxiety disorders, post-traumatic stress disorder, addiction, and a number of less common conditions associated with impairments of brain function. Despite the extraordinary promise of brain imaging in defining pathophysiology and testing hypotheses about these and other conditions, the sobering conclusion is that the role of neuroimaging in 2018 for psychiatry and neurodevelopment is limited to excluding unexpected structural lesions like stroke, tumor, or brain malformations.

This is not for lack of trying. Brain imaging has been a staple of research into mechanisms and treatment efficacy for the last half century, and has helped transform views of mental illness among physicians and the general public. Psychiatric disorders are now seen as conditions affecting the brain, with the promise that treatments affecting brain function can improve patient outcomes. Yet for individual patients, researchers have more work to do before we can manage diagnosis, prognosis, and treatment monitoring. In this review, we consider what factors have contributed towards the limited clinical utility of neuroimaging in neurodevelopment and psychiatry, and offer some pathways towards greater relevance. First, we discuss the state of affairs in diagnostic neuroimaging by considering five mental illnesses and briefly reviewing efforts to use imaging as diagnostic tests, including autism, ADHD, schizophrenia, bipolar disorder, and major depression. Second, we discuss factors related to limited efficacy of clinical neuroimaging tests and sources of variability that may be responsible for the challenges of successful functional neuroimaging. Finally, we suggest promising future directions that may bridge these gaps in our abilities and pathways toward a more robust role for brain imaging in the management of neurodevelopmental and psychiatric disorders.

Autism

While the potential role of neuroimaging involves many objectives, including treatment monitoring, prognostication, and discovery of pathophysiological mechanisms, it is helpful to use diagnostic classification as a benchmark for the ability of neuroimaging tests to identify features related to the condition. Structural brain imaging studies, including gray matter morphometry¹ and diffusion imaging metrics,² have occasionally yielded classification accuracies of 80% or greater within single site data, but most of the literature has focused on functional imaging metrics such as functional MRI connectivity as having the greatest propensity for classification of autism or related symptoms. Studies including resting state fMRI data from a single site with uniform acquisition protocol were able to achieve about 80% classification accuracy,^3–5 including replication in a sample that included unaffected siblings,³ but still underperformed behavioral measures in classification.⁴ Using similar techniques when applied to a multisite sample, the ABIDE data set,^6,7 reduced accuracy to 60–75%, significantly better than chance, but insufficient to serve as a useful clinical test despite the use of sophisticated classification algorithms.^8–12 Ultimately, these approaches have not been successful in overcoming the heterogeneity⁸ of even carefully curated research populations using best practices for ascertainment and diagnosis.

ADHD

Brain imaging research into ADHD has demonstrated a number of structural and functional abnormalities in certain brain regions and networks (e.g. basal ganglia, frontostriatal circuits, and default network), yet not with sensitivity or specificity needed for clinical application of neuroimaging. In a recent international competition to classify ADHD subjects from a multisite resting state fMRI data set (ADHD-200), the highest classification accuracy (62% for 3-group classification) was obtained from among 21 entries for a strategy that did not use any brain imaging data.¹³ Other strategies in the competition, such as a multimodal classifier using regional homogeneity and functional connectivity,¹⁴ were able to achieve 2-group classification accuracy between 60 and 70%. An analysis of the ADHD-200 global competition data led to maximum internal accuracies of 78%¹⁵ and classification methods utilizing MRI and neuroimaging markers have led to accuracies of 76.15%.¹⁶ Although functional imaging classifiers have typically had highest accuracies, at least one structural imaging classifier achieved accuracies of over 90%.¹⁷ However, in a recent review of classification accuracy, it was suggested that a number of classification models were unduly influenced by small sample sizes or circular analyses.¹⁸

Schizophrenia

Schizophrenia has been associated with dysfunctional connectivity across frontotemporal brain regions,¹⁹ in circuits of the cortico-cerebellar-striatal-thalamic loop,²⁰ and between task-positive and task-negative networks.²⁰ Classification of schizophrenia based on gray matter density data from MRI scans provided up to 71.4% accuracy.²¹ One recent classification study using both structural and diffusion tensor imaging reported a 75.05% classification accuracy.²² Using whole-brain resting-state functional connectivity data, an accuracy of 81.3% was reported when differentiating schizophrenic subjects from controls and 75.6% when differentiating schizophrenic subjects with and without auditory verbal hallucinations.²³ Using a hybrid machine learning method of classification with both genetic and brain imaging data, the highest prediction accuracy for diagnosis of schizophrenia approaching 90%.^24,25

Major depression

Major depression has been studied using resting state neuroimaging to show increased connectivity within the anterior default network and between the default and salience network,²⁶ as well as decreased connectivity between frontoparietal attentional regions.²⁷ Research on resting state neuroimaging also shows a correlation between major depression and Dorsal Nexus connectivity.²⁸ Classification accuracies of about 90% were found when applying machine learning techniques to structural neuroimaging.²⁹ In the past 10 years, classification accuracy of major depression using structural or functional neuroimaging data have ranged between 45 and 99%.³⁰ Using a linear support vector machine classifier in whole-brain functional connectivity data, 100% of patients were correctly classified as having major depressive disorder compared to 89.7% healthy controls for an overall accuracy of 94.3%.³¹

Bipolar disorder

Bipolar disorder has been correlated with abnormal hippocampal and cerebral brain volumes,³² with functional connectivity characterized by atypical overconnectivity between default and salience networks.³³ Although a growing literature has evaluated brain volumes, networks, and biomarkers associated with bipolar disorder, there are still limited clinical applications of imaging, in part due to the difficulty of diagnosis and discrimination between other mood disorders.^30,34 When using biomarkers and neuroimaging to discriminate between schizophrenia and bipolar disorder alone, prediction accuracies of 95% for controls, 92% for schizophrenia, and 83% for bipolar subjects were achieved.³⁵ In a large multisite sample of 853 bipolar subjects and 2167 controls, support vector machine learning incorporating cortical thickness, surface area, and subcortical volumes was used to obtain a classification of 65.23% with individual site accuracies ranging between 45.23 and 81.07%.³⁶

Additional challenges

Many of the classification accuracies detailed above are within the range of accuracy used in other medical tests and may seem to approach a level of confidence that could contribute to patient management, but three factors temper enthusiasm. First, depending on the pre-test probabilities of patients imaged, there remains the possibility of substantial overdiagnosis if relying on imaging given accuracies in the range of 80–90%. To the extent that such a test is used for screening given the lack of other objective diagnostic tests for these conditions, the false positive rate would likely be unacceptable. Second, the overwhelming majority of these studies used carefully selected and curated patient samples. Real-world applications are likely to suffer significantly when applied to more complex, less characterized patient cohorts, and these tests would invariably perform worse when applied to variable scanner hardware, inconsistent imaging protocols, and scan conditions that were not tested in experimental protocols. Third, virtually all of the studies above examine a specific group compared to control cohorts. Clinical applications rarely face such a binary choice, but rather are required to distinguish between many potential conditions in a differential diagnosis. The conclusion, at least temporarily, is that while there is clear evidence that neuroimaging techniques are performing well above chance at recognizing disease features, they are not yet operating at a level ready for clinical translation in a complex, real-world environment.

Why is it so hard?

Heterogeneity and individual variability of diagnostic populations

Diagnostic categories in psychiatry and neurodevelopment have made historic changes toward more objective definitions such as with symptom clusters in the diagnostic and statistical manual, but yet fail to align with emerging neurophysiological constructs from clinical neuroscience and genetics.^37,38 This lack of a gold standard for psychiatric and neurodevelopmental diagnoses based on neurobiological mechanisms is unavoidable when many of the relevant basic mechanisms are unknown. But most categorical diagnoses in psychiatry and neurodevelopment exhibit marked heterogeneity, limiting any potential imaging test from sensitive and specific diagnoses that may not align with neurobiology. The challenge for neuroimaging is to not simply to reconstruct existing categories but define pathophysiological mechanisms for the underlying conditions.

Inadequate information about brain function

Particularly for structural imaging metrics, researchers may be faced with the reality that brain structure at currently achievable spatial resolutions may simply not contain enough information about individual differences in brain function to serve as meaningful clinical tests for disorders of brain function. Technological development has shown consistent progress in pushing scan resolution to finer scales, but as imaging approaches mesoscale levels where features such as individual cortical columns are identified, idiosyncrasies associated with individual differences such as cortical folding patterns and anatomical variation loom larger, and the challenge of extracting meaningful differences from uninteresting variation becomes ever more complex as the granularity of our imaging tests improves. It remains unknown at what spatial scales fundamental differences contribute to the pathophysiology of neurodevelopmental and psychiatric conditions. For imaging modalities that do contain information about brain function, an entirely separate set of complexities emerges from the difficulty in achieving reliable measurements.

Limited single-subject reproducibility

A critical requirement for using neuroimaging tests to serve as reliable clinical instruments is that they are reproducible. At a minimum, test–retest reliability is a prerequisite for any meaningful clinical metric. Most structural and diffusion imaging metrics demonstrate excellent reproducibility with intraclass correlation coefficients around 0.9,³⁹ although even simple gray matter volumetric metrics can vary significantly with processing strategy, magnetic field strength, scanner and pulse strength.⁴⁰

Functional MRI measurements, including resting state functional connectivity are more problematic. Early reports suggested that canonical resting state networks showed stabilization of correlation values with acquisition times of as little as 5 min^41,42 and that graph theory metrics may stabilize with as little as 2 min of data.⁴³ Yet these types of coarse summary metrics are unlikely to be sensitive and specific biomarkers for clinical applications. When examining individual connections between regions or networks, subsequent work has shown that the single most important factors contributing to reproducibility are head motion^44,45 and imaging duration, with accuracy and reproducibility of functional connectivity measurements improving with the square root of imaging time.^46–50 This cannot simply be compensated for by faster imaging strategies such as may be obtained through multislice image sequences,⁵¹ as the critical factor appears to be the aggregate time in scanner rather than number of volumes used,⁴⁹ and probably represents an opportunity to sample more comprehensively brain microstates given nonstationarity of brain activity. Up to several hours of total imaging time, which need not be consecutive, continues to show improvements in accuracy, particularly when accuracy of specific connections or voxel level precision is required.⁵² The overwhelming majority of functional connectivity studies have used image durations of between 5 and 10 min, well below durations at which reliable identification of individual subjects is seen in MRI fingerprinting analyses.^48,53

In addition to head motion and imaging time, reproducibility is also limited by drowsiness, with large changes in functional connectivity in stages of sleep that may occur within minutes of scan onset and may not be perceived by either examiner or research subject.^54–56 Resting state acquisitions can be performed differently by subjects, just as any other task, and results are affected by whether subjects’ eyes are open,^57,58 whether stimulation is present,⁵⁹ time of day,^60,61 caffeine consumption,^62,63 and physiological parameters.⁶⁴

There is an extensive literature on the effects of post-processing strategy on results of functional connectivity, e.g. related to the use of nuisance regressors such as the global signal,^65–68 order of processing operations,⁶⁹ and parcellations used,^70–72 none of which are standardized. In combination with continued evolution of pulse sequences, scanner hardware, and software platforms, adoption of consistent acquisition and analysis strategies across sites is a daunting challenge for development of stable clinical neuroimaging tests.

Large data sets such as the Consortium for Reliability and Reproducibility,⁷³ Adolescent Brain Cognitive Development,⁷⁴ and Human Connectome Project⁷⁵ are invaluable resources for establishing constraints on the reliability of functional neuroimaging metrics given test–retest or longitudinal data in large cohorts, both on a single scanner or across multiple sites.

Lack of information about the temporal domain of brain activity

Many of the most commonly used functional neuroimaging techniques, including PET, task fMRI, functional MRI connectivity, and diffusion MRI contain little or no information about the temporal dynamics of brain activity. This may be a critical limitation. Even disorders with profound functional impairment, such as Down Syndrome, show only subtle, quantitative alteration in the architecture of brain networks that may only be identified in population studies.⁷⁶ A rapidly evolving literature is demonstrating that there is unique information available in the timing of functional brain activity that may be more important to functional brain disorders.^77–79

Dynamical connectivity methods have been developed to evaluate how functional connections between brain regions change over time, acknowledging that brain activity is nonstationary and that shifts between cognitive states may correspond to important alterations in connectivity and activation.^77–79 For example, connectivity may be evaluated over shorter time periods using sliding window approaches.^80–83 Other techniques may use component techniques,⁸⁴ evaluation of duration of connectivity,⁸⁵ temporal derivatives,⁸⁶ or other techniques to mitigate problems of reduced accuracy when connectivity measurements are created from short time windows.⁸⁷ Ultimately, most electrophysiological recordings measure brain dynamical activity over seconds or minutes, while even dynamical functional MRI measures typically only extend for minutes. Longer duration recordings that sample more gradual changes between dwell times in brain states over hours, or techniques that bridge the spatiotemporal scales of electrophysiologic and cross-sectional imaging modalities are opportunities for obtaining understudied information that may improve clinical utility of functional brain imaging.

Limited information about neurochemistry

Many of the most transformative therapies within psychiatry over the last several decades have targeted neurotransmitter systems through serotonin, dopamine, endocannabinoid, or opioid receptors. Imaging tools that visualize neurochemistry have lagged behind these therapies, and molecular imaging targets have primarily been developed within PET systems that until recently have been studied in parallel rather than in combination with MRI and electrophysiological systems where most of the research into connectivity and function has proceeded at finer spatiotemporal resolutions. Some key receptors have no available imaging strategy for human in vivo studies, and dynamic imaging of the timing of neurochemistry processes in humans is in its infancy. These limitations remain lost opportunities for understanding critical loci of the neurophysiological mechanisms for psychiatric and neurodevelopmental conditions.

Few studies with longitudinal trajectories

Most studies examining psychiatric and neurodevelopmental conditions use cross-sectional designs, which have significant limitations in providing information about developmental processes.⁸⁸ Because inferences about developmental processes are made by comparing different individuals at different ages in cross-sectional studies, and developmental processes naturally vary in different individuals, longitudinal studies are needed to verify the age-related findings of cross-sectional research.^88,89

From hardware to software: Paths forward

Improving reproducibility

Functional imaging tests can only achieve clinical utility in individual patients if they are reproducible at the single subject level, and investigators should consider including replication as a standard feature of neuroimaging designs, particularly given risk of false-positive results in neuroimaging studies. For functional imaging tests, reliability can be improved with technical improvements to image acquisition. Multiband or multislice imaging protocols^51,90,91 offer two key reliability improvements. First, faster imaging times allow more data points to reconstruct functional activation with improved signal to noise obtained by better temporal fitting of waveforms, whether for task-based imaging or functional connectivity. More importantly, faster imaging greatly mitigates aliasing of head motion, respiratory and cardiac artefacts into frequency ranges of interest (less than 0.1 Hz), and allows for identification and isolation of volumes where head motion occurs.⁹² Time savings can alternately improve either temporal or spatial resolution depending on the desired application.

Image duration is of particular importance to functional imaging paradigms. Individual connections are limited to accuracies greater than 0.2 correlation units if less than 10 min of aggregate image data are used for functional connectivity measurements, much greater than patient–control differences for most applications, and image times of 30–60 min (as long as feasible) are much more likely to result in clinically meaningful single subject results.⁴⁹ This is perhaps even more critical for dynamical connectivity and analysis of temporal aspects of brain function. Leveraging the temporal aspects of brain function is likely to have particularly high yield for disorders of brain function (software rather than hardware), and many applications may require sampling brain states that require sufficient imaging duration to acquire.

Continued improvements to artefact correction and optimization of post-processing strategies will continue to be nontrivial, important details that may avoid pitfalls to robust reproducibility and interpretability of results. Ultimately, clinical translation will be focused on simplification and standardization of imaging measures rather than progressive complexity that makes multisite testing and implementation more difficult. The experience of the Quantitative Imaging Biomarkers Alliance initiative has demonstrated that barriers to standardization needed for clinical applications are more problematic as the complexity of the processing requirements and specificity for particular hardware and software implementations increase.⁹³

Data sharing and aggregation of multisite data sets will continue to play a critical role in facilitating big data approaches to isolating the biological signals of interest for psychiatric and neurodevelopmental applications, as well as demonstrating specificity of potential biomarkers for particular acquisition and analysis strategies. Broadening access to multidisciplinary scientific teams beyond the sites where data were collected has been an important advance in fostering creative new approaches to analyzing complex imaging data.

New engineering approaches

Engineering new approaches for image acquisition will continue to play a vital role in developing clinically useful imaging tests for disorders of brain function. Imaging technologies that can expand imaging into more ecological environments, particularly for longer temporal duration imaging would be of great utility. A “Holter monitor” for the brain that could be sent home with a patient and collect data for epochs of many hours or days may have the potential for radical new insights into brain function. Fortunately, possibilities for such devices exist and are in research and development phases, including long-term EEG⁹⁴ and room-temperature portable MEG devices that may be wearable, communicate wirelessly with data storage units, and interface with deep phenotyping hardware such as actigraphy or monitoring of environmental stimulation.⁹⁵

Protocols employing repeated scans, longitudinal or serial imaging, and timing of imaging relative to symptoms or critical clinical periods will also improve data acquisition to include high-yield epochs more relevant for diagnosis, prognosis, and treatment monitoring. Advances in imaging of neurochemistry may be particularly meaningful, especially if techniques for dynamic, in vivo, human imaging of neurotransmitter systems allow access to signaling pathways particularly relevant to disorders of brain function.

Deep learning and dynamic simulations

Successful clinical application of neuroimaging in psychiatry and neurodevelopment may require identifying many features, each of which may lack sensitivity and specificity by itself for clinical application. This situation is well suited to deep learning techniques that may facilitate identification of clinical outcomes from complex data that are not visible by inspection of images alone. Recent advances in machine learning applications are likely to accelerate recognition of subtle but informative patterns in functional imaging data.

In silico approaches to neuroimaging, using computational models to predict brain function based on an individual’s connectome or other imaging data,^47,96,97 are promising strategies to simulate effects of interventions and extend precision medicine to predict how those interventions may affect brain network function in specific patients, reducing the need for trial and error in selecting treatments.

Dimensional analysis of brain function

A consequence of seeking for brain-based phenotypes for disorders of brain function may require realignment of traditional categories. This approach has been championed under the National Institute of Mental Health’s Research Domain Criteria initiative.³⁷ Instead of seeking to find imaging tests that recapitulate diagnoses such as schizophrenia, major depression, and bipolar disorder, it may instead be more productive to identify novel categories more relevant to underlying neurobiology, many of which may be shared across diagnoses. Effectively, imaging may be seeking less specific, but more grounded phenotypes. Just as an internist may find useful general inflammatory markers such as C-reactive protein that are not associated with specific diagnoses, so psychiatric neuroimagers may need to identify intermediate, cross-disorder markers associated with patterns of brain functional disorganization. Features that may be shared across many disorders, such as atypical overconnectivity between default and attention control networks, atypical negative connectivity, regional patterns of dysconnectivity, or abnormal slowing of brain state transitions, or something entirely different, may contribute towards a library of brain-based traits that can inform diagnosis and prognosis of multiple diagnoses.

Imaging effects of medication and altered physiology

An opportunity for near-term impact on clinical imaging of brain function would be to characterize more specifically the changes in brain physiology associated with commonly used psychiatric medications or therapies. Relatively little information is available to document mechanisms of action of even the most commonly used classes of drugs beyond basic receptor targets. How do SSRI medications affect brain network interactions? How do antipsychotic medications influence cortical neurophysiology? Looking at whole-brain or downstream effects of useful pharmaceutical agents on multiple neural systems may be a productive avenue for identifying targets for constraining development of novel therapies.

Practical considerations for clinical imaging applications

An important barrier for translation of brain functional imaging into the clinic continues to be pragmatic and cost concerns. Medical imaging continues to be a significant driver of the cost of medical care, and expansion of indications for new, sophisticated technologies must take into account sustainability and value within the larger health care system.⁹⁸ In response to these pressures, there has been an impetus within the MRI community for development of rapid MRI protocols (https://www.ismrm.org/workshops/2018/HighValue/), development of low-cost but high-performance MRI systems,⁹⁹ and workflow innovations to improve productivity and lower cost of imaging through machine learning applications.¹⁰⁰ All of these developments exist in tension with the need for longer acquisitions, higher resolution, and postprocessing heavy applications for functional imaging discussed above. Once sensitive and specific applications are developed, however, it is hoped that identifying the core information needed for diagnosis will translate to cheaper or faster implementations.

Implementation of advanced MRI techniques in rural settings or locations where supporting computational infrastructure is not available is another important limitation. Efforts at cross-site reproducibility, automation of post-processing pipelines accessible to clinical workflow timing, and quantitative thresholds for diagnosis that can be ported throughout the healthcare system must be developed in parallel with algorithms and acquisition strategies for clinical functional brain imaging. One potential early application that may be automated for widespread use could be resting state brain mapping for presurgical localization of eloquent brain regions, mitigating the need for complicated hardware and software solutions.¹⁰¹ For psychiatric and neurodevelopmental applications, a possible early application of functional imaging could be image guidance for deep brain stimulation or neuromodulation of depression or obsessive–compulsive disorder.¹⁰²

Conclusions

The Emperor’s wardrobe is bare, but the tailors are actively engaged, and there is great reason for optimism that neuroimaging will continue to make inroads toward direct clinical impact in patient management in the near future. Imaging techniques already show demonstrable effects across multiple disorders in population studies, and advances in technology suggest that these findings may be applicable to individuals with continued development. While we do not have answers for specific tests that may break through into clinical use, we suggest strategies for emphasis in neuroimaging that may push the boundaries of the adjacent possible: improving reproducibility, engineering advances, doubling down on neurochemistry, emphasizing the time domain of brain function, harnessing the power of machine learning, developing a library of dimensional neurobiological metrics, and revisiting specific effects of currently effective medications using state-of-the-art imaging techniques.