Precision diagnostics based on machine learning-derived imaging signatures

Abstract

The complexity of modern multi-parametric MRI has increasingly challenged conventional interpretations of such images. Machine learning has emerged as a powerful approach to integrating diverse and complex imaging data into signatures of diagnostic and predictive value. It has also allowed us to progress from group comparisons to imaging biomarkers that offer value on an individual basis. We review several directions of research around this topic, emphasizing the use of machine learning in personalized predictions of clinical outcome, in breaking down broad umbrella diagnostic categories into more detailed and precise subtypes, and in non-invasively estimating cancer molecular characteristics. These methods and studies contribute to the field of precision medicine, by introducing more specific diagnostic and predictive biomarkers of clinical outcome, therefore pointing to better matching of treatments to patients.

1. Introduction

Medical imaging has witnessed a remarkable growth over the past 3 decades, which is unlikely to slow down in the near future. This growth has enabled the medical and biological communities to better understand normal and abnormal structure and function, and to develop a broad spectrum of imaging-based diagnostic and prognostic biomarkers. For example, in clinical neuroscience imaging has significantly contributed to better understanding processes of structural and functional maturation of the brain during its development, to identifying structural and functional deviations from normality that are associated with diseases such as schizophrenia, autism, bipolar disorder, multiple sclerosis, amongst others, and to better understanding the process of brain aging, its heterogeneity, and the multiple factors that contribute to or are associated with it. In cancer research, imaging has discovered complex phenotypes that are associated with clinical outcome, response to treatment, and molecular characteristics of the underlying disease.

Many of these developments have taken place in just 20 years, and have propelled imaging from the rudimentary role of identifying basic structures and lesions, to identifying complex, subtle, spatially distributed and multi-parametric phenotypes. These phenotypes are most often no longer visually appreciable, and they require advanced analytic methods to be detected and measured. Even more challenging is their detection at the individual person level. Although it is important to know that, for example, metabolic activity in the posterior cingulate is somewhat reduced at preclinical stages of Alzheimer’s Disease, while at the same time subtle hippocampal loss begins to occur, it is far more challenging to establish an imaging signature of preclinical Alzheimer’s Disease (AD) that can be detected with high specificity and sensitivity in single individuals. Machine learning has played a transformative role in this respect during the past 15 years, in that it has been able to harness the power of complex multi-parametric models that are sufficiently distinct to identify disease processes and to predict outcome on an individual basis. The current paper presents certain aspects and case examples of imaging-based precision diagnostics and predictive modeling.

The first three sections of our paper focus on methodological issues that aim to address two challenges. The first challenge relates to disease heterogeneity, which is frequently underappreciated by single disease umbrellas. In particular, a machine learning model encodes a pattern that separates or fits the data. In practice, there often are multiple patterns that characterize disease (subtypes), or normal structure and function. While nonlinear models deal with this heterogeneity, they do it in a way that is difficult to interpret. Clinical adoption of these tools might require approaches that capture such heterogeneity in relatively simple and interpretable ways. For example, disease subtypes could be captured by two or more imaging signatures, allowing for clinical refinement of a diagnosis. (An imaging signature herein is a high-dimensional multiparametric model integrating many imaging features into a biomarker of diagnostic or predictive value.) Even for diseases like Alzheimer’s that are thought to have a fairly consistent pattern, multiple studies have elucidated important heterogeneity that calls for multiple imaging signatures, with differential clinical characteristics [1-3], Section 2.1 provides two examples of semi-supervised learning methods aiming to capture heterogeneity. The subcategorization of a patient into one of several subtypes would be easier for clinicians to adopt, in contrast to a complex and difficult to interpret

The second challenge relates to complexity of imaging phenotypes and measures. If we take brain MRI as an example, current commonly used protocols include various structural contrasts, diffusion protocols yielding complex structural connectomic measures, resting state functional magnetic resonance imaging (fMRI), which provides functional connectivity measures, task fMRI which provides a “stress test” of brain function, perfusion imaging which provides blood flow and related measures, as well as various other protocols that provide diverse information about structure and metabolic activity. As these protocols grew to be increasingly complex, and as the volume of imaging data available for training machine learning models grew exponentially, the need for the development and adoption of even more complex machine learning models became apparent. Section 2.2 describes examples of deep learning models aiming to address the field of precision diagnostics.

The remainder of the paper describes several example studies from clinical neuroscience (Section 3) as well as from cancer research (Section 4). In these studies, machine learning tools were used in the context of precision diagnostics, prediction of clinical outcome, and characterization of molecular characteristics.

2. Methodological Developments that support Precision Diagnostics from Imaging Signatures

2.1. Heterogeneity: Is it a single disease signature or multiple subtypes?

In order to derive an imaging signature of disease, the majority of the studies assume that there is a single imaging pattern that distinguishes patients from healthy individuals. In other words, most approaches assume a unifying pathophysiological process that affects patients and perform a monistic analysis to identify it. This assumption is often violated as many brain disorders (e.g., Autism Spectrum Disorder, [4, 5] Schizophrenia [6-8] and Alzheimer’s Disease or AD [1, 2, 9]) are characterized by significant clinical heterogeneity--see Fig. 1a for a graphical illustration of the problem. In such cases, common approaches can only provide a partial description of the disease changes. Accurately characterizing disease effects in the presence of heterogeneity is essential for deepening our understanding as well as for enabling more accurate diagnosis, prognosis and targeted treatment.

Fig. 1.

a) Heterogeneity problem setting: X denotes the reference population, and Y denotes the heterogeneous disease population, b) CHIMERA [14] takes a primarily generative angle, and dissects heterogeneity by transforming X into a distribution X′, consisting of subgroups ${\mathit{X}}_1^{\prime }$ and ${\mathit{X}}_2^{\prime }$, that overlaps with Y under the influence of transformations T₁ and T₂, respectively, c) HYDRA [15] takes a primarily discriminative angle, and dissects heterogeneity by assigning members of the heterogeneous population to the hyperplane (denoted by either {w₂, b₁} or {w₂, b₂} here) that best separates them for the healthy reference population. Note that different colors denote distinct groups and subgroups.

Several studies have used clustering methods to address this challenge (e.g. [2, 10-13]). However, clustering is heavily influenced by sources of confounding variations that are not relevant to a disease pathophysiological process, such as demographics or even technical confounds (e.g. scanner type). Semi-supervised methods offer an alternative approach to this problem. By seeking to cluster disease effects, i.e. differences between patients and controls, rather than patients themselves, these approaches zoom into the heterogeneity of pathological processes, rather than heterogeneity in general.

In the remainder of this section, we briefly discuss two complementary approaches, the first one being primarily generative, and the second being primarily discriminative. Before detailing the approaches, let us provide some common notation and set the problem. Let us assume that our dataset contains M normal control samples X = {x₁, … x_M} that define our reference population, and N patient samples Y = {y₁, …, y_N}. Let us also assume that the samples are described by a D-dimensional set of imaging features ${\mathit{x}}_i\in {\mathbb{R}}^D$ and ${\mathit{y}}_i\in {\mathbb{R}}^D$. The goal is to assign a subgroup label l_i to each patient sample y_i

CHIMERA

The first method, termed CHIMERA¹ [14], aims to cluster heterogeneous disease effects via distribution matching of imaging patterns. CHIMERA models every subject as a point in a high dimensional feature space. The two populations are therefore represented as distinct point distributions. In such a setting, a straightforward way to dissect disease heterogeneity would be to use a clustering approach to partition the patient distribution. However, such an approach would be driven by the similarities between different individuals. As such, it would be sensitive to strong covariations in the data, producing clusters that partition the normal variability in the data due to age, sex, and other confounding factors that don’t necessarily align with the disease effect.

To tackle this challenge, CHIMERA takes into account label information and relates the two distributions in a generative framework. It assumes that the patient distribution is generated from the healthy control distribution under the effect of an unknown transformation T, which reflects the disease effect. This transformation may be estimated by solving a matching problem, where one seeks to align the two point distributions. The main underlying assumption here is that if patients had been spared from the disease, then the patient and healthy control point distributions would overlap since each patient’s brain would look like the brain of a healthy subject of similar age and gender. Therefore, any differences reflect pathological processes. Distinct pathological processes may be captured by introducing multiple distinct transformations in the model (see Fig. 1b for a graphical illustration).

In this setting, the transformation T is defined as a convex combination of K linear transformations that map the D-dimensional imaging features of a healthy sample ${\mathit{x}}_i\in {\mathbb{R}}^D$ to the patient distribution as ${\mathit{x}}_i^{\prime }=\mathit{T}({\mathit{x}}_i)={\sum }_{k=1}^K{\zeta }_k{\mathit{T}}_k({\mathit{x}}_i)$, where the coefficients ζ_κ can be considered as the probability that the healthy sample x_i follows the transformation T_k. The distinct transformations are estimated by solving the distribution matching, which is conducted as a variant of the coherent point drift algorithm [16]. In this setting, each transformed healthy sample point is considered as a centroid of a spherical Gaussian cluster, and patient points are treated as independent and identically distributed data generated by a Gaussian Mixture Model (GMM) with equal weights for each cluster. The data likelihood of this mixture model quantifies the similarity between patient points and transformed healthy points in the imaging domain, while also taking into account covariate information (e.g., age and gender) aiming to match healthy points to patient points of similar demographic distribution. The optimal transformations are estimated by optimizing the data likelihood during the distribution matching, while also penalizing large transformations through the introduction of a regularization term whose goal is to improve the stability of the clustering results. The resulting energy objective is optimized using an Expectation-Maximization approach[17]. Having estimated the probability for a healthy sample to undergo a transformation T_k, then it is straightforward to estimate the likelihood that a patient sample y_i may be generated by the same transformation, and a patient can be assigned the label corresponding to the largest likelihood.

This method was initially validated using simulated data and was shown to perform better than k-means clustering [18] and hierarchical Ward clustering [19] in the presence of heterogeneity [14], CHIMERA was latter applied to study the neuroanatomical heterogeneity of AD [1] as well as that of Schizophrenia [20]. Some results along these lines are presented in Section 3.1, below.

HYDRA

The second method, termed HYDRA² [15], aims to reveal heterogeneity through discriminative analysis. HYDRA also models subjects as points in a high dimensional space, but contrary to the previous method, it takes a purely discriminative approach. HYDRA is based on the Support Vector Machine (SVM) classification framework, and aims to capitalize on the modeling capacity of linear SVMs in high dimensional spaces, which is high enough to discriminate between two homogeneous classes. However, linear SVMs may be challenged in the presence of heterogeneity (e.g., one class is generated by a multimodal distribution). While they may be still able to distinguish between two classes, the resulting classification margin may be small, impacting their generalization power. In such a setting, non-linear classifiers, such as Gaussian or polynomial kernel SVMs, may allow for larger margins, but at the cost of limited interpretability when aiming to characterize heterogeneity.

HYDRA aims to address the aforementioned limitations by extending linear maximum margin classifiers to the non-linear case in a piecewise fashion. By combining multiple hyperplanes, a convex polytope is formed that separates the two groups. The convex polytope encloses the heathy samples, while allowing for disease subtyping by associating patient samples to distinct faces of the polytope (see Fig. 1c for a graphical illustration). Our assumption here is that each face of the convex polytope captures a distinct multivariate pattern of difference between the two groups, and hence a distinct disease process.

The convex polytope is estimated by solving simultaneously a binary classification problem and a clustering one. The convex polytope can be estimated by solving the following optimization problem:

$$\underset{\begin{array}{c}\hfill \{{\mathit{w}}_j,b_j{\}}_{j=1}^K\hfill \\ \hfill \{s_{i,j}{\}}_{i,j}^{N,K}\hfill \end{array}}{\min }\sum _{j=1}^K\frac{\Vert {\mathit{w}}_j{\Vert }_2^2}{2}+\sum _{j=1}^K\sum _{i=1}^M\frac{1}{K}\max \{0,1-{\mathit{w}}_j^T{\mathit{x}}_i-b_j\}+C\sum _{j=1}^K\sum _{i=1}^N{s}_{i,j}\max \{0,1+{\mathit{w}}_j^T{\mathit{y}}_i+b_j\},$$

where ${\mathit{w}}_j\in {\mathbb{R}}^D$ and b_j define one of the K hyperplanes that correspond to the faces of the polytope, and s_i,j is a binary variable.The above optimization problem bears similarities with the objective function of standard linear SVMs, but also significant differences. Similarly to standard SVMs, the first term encourages maximum average margin across all Kfaces of the convex polytope. However, contrary to standard SVMs, positive and negative samples are treated differently. The second term ensures that all healthy samples are placed in the same side of the half-space by all K linear hyperplanes, effectively enclosing them within the convex polytope with slack. Lastly, the third term ensures that a subgroup of the patient samples lies outside one of the faces of the polytope with slack. The indicator variable s_i,j assigns patient samples to faces of the polytope, thus deciding which patient samples lie outside each face of the polytope. In other words, disease samples are labeled according to the face of the polytope they are assigned to. The above optimization problem is solved in an iterative fashion by alternating between clustering (i.e., assigning patient samples to faces of the polytope) and estimating the hyperplanes that maximize the overall margin. This coupling between clustering and classification allows for segregating patients based on disease effects rather than global anatomy.

HYDRA was extensively validated using simulated data, and was shown to perform better than k-means clustering [18] in the presence of heterogeneity [15]. It was additionally tested using genetic and imaging data obtained from the ADNI study. In both cases, HYDRA identified reproducible disease subgroups that were characterized by distinct genetic and neuroanatomical profiles, as well as demographic, cognitive and cerebrospinal fluid (CSF) characteristics.

Section 2.2. Emerging Deep learning methods in brain image analysis

Finding optimal sets of features.

Deep learning techniques have been widely adopted in medical imaging analysis [21], following their success in natural image analysis [22]. Particularly, convolutional neural networks (CNNs) are widely used in medical image analysis for learning imaging features, complementing hand-crafted features they might be not optimal and less discriminative for many tasks. As an example, we take the prediction of cognitive decline and AD progression from MRI. Recent studies have demonstrated that CNNs could be trained to learn discriminative hippocampal information from structural MRI data for distinguishing AD patients from cognitively normal controls or classifying individual subjects into multiple categories such as cognitively normal, stable mild cognitive impairment (MCI), progressive MCI, and AD dementia [23-25]. Particularly, a deep ordinal ranking model has demonstrated better classification performance than multi-category classification models for classifying individual subjects into categories of cognitively normal, stable MCI, progressive MCI, and AD dementia by taking into account the inherent ordinal severity of brain degeneration associated with normal aging, MCI, and AD [23], and a deep learning time-to-event prediction model built on CNN hippocampal MRI features has demonstrated improved performance for predicting individual subjects’ progression to AD dementia, compared with prediction models built on hippocampal volume, shape, and texture MRI features [24, 25]. Such methods provide a cost effective and accurate means for prognosis and potentially to facilitate enrollment in clinical trials with individuals likely to progress to AD dementia within a specific temporal period.

Processing of temporal data.

MRI datasets are increasingly becoming longitudinal, or have a temporal component (e.g. fMRI), and hence might benefit for specialized deep learning models. Recurrent neural networks (RNNs) are deep learning models particularly suitable for sequence modeling [26]. RNNs with long short-term memory (LSTM) [27] have achieved remarkable advances in brain decoding of fMRI data [28]. For the brain decoding of fMRI data, multivariate pattern analysis (MVPA) is the most successful technique [29, 30]. Most MPVA methods build brain decoding models on voxel-wise fMRI measures in the whole brain or selected brain regions of individual time points or within temporal windows with a fixed length using conventional pattern classification techniques, such as SVM and logistic regression. These methods do not take into consideration the temporal dependency that is inherently present in sequential fMRI data. Voxel-wise measures are subject to physiological and instrumental noise and brain decoding models built upon all voxels in the brain suffer from the “curse of dimensionality” (aka overfitting). Such limitations can be overcome by building brain decoding models on individualized intrinsic functional networks (FNs) using LSTM RNNs. Particularly, FNs provided a robust and generally applicable means to extract functional signatures for the brain decoding due to their good correspondence to the task activations [31]. Using the FNs, fMRI data could be represented by a low-dimension feature vector, which could alleviate the curse of dimensionality, be general to different brain decoding tasks, and provide better interpretability by identifying the most sensitive FNs related to the brain decoding tasks using sensitivity analysis. It has been demonstrated that brain decoding models built on FNs using LSTM RNNs accurately decode fine-grained working memory tasks and motor tasks in real time [28]. LSTM RNNs in conjunction with autoencoders have also demonstrated promising performance in detection of temporal transition of functional states in both resting state and task fMRI data [32].

The individualized FNs also facilitate computation of whole brain voxelwise functional connectivity measures, providing fine-grained functional connectivity information of the brain [33]. It has been demonstrated that age prediction models built on the whole brain voxelwise functional connectivity measures had better age prediction performance than those built on coarse-grained functional connectivity measures between FNs, regardless of pattern recognition methods including sparse regularized least-squares regression and CNNs [33]. The individualized FNs could also be computed at multiple spatial scales with a hierarchical organization [34]. Compared to FNs identified at different scales independently or in a greedy agglomerative way, the multi-scale hierarchical individualized FNs better capture informative intrinsic functional networks and improve the prediction performance of task activations evoked by different tasks.

CNNs for segmentation.

A large effort has been devoted to the development of deep learning based medical image segmentation methods, promoted by publicly available data and medical image segmentation challenges, such as “Multimodal Brain Tumor Segmentation Challenge (BraTS)” [35-37]. Most deep learning based medical image segmentation methods are built upon CNNs to segment images in a voxelwise classification setting based on image patches or implemented as fully convolutional networks (FCNs) [38]. These methods essentially classify individual pixels/voxels separately based on their context information, lacking explicit constraints to encourage spatial and appearance consistency in the segmentation results. Markov random fields, particularly conditional random fields (CRFs), could be used as a post-processing procedure to model dependencies of pixelwise/voxelwise segmentation results and enhance spatial and appearance consistency of the segmentation results. Particularly, improved brain tumor segmentation performance has been achieved by an end-to-end deep learning based brain tumor segmentation method with integrated FCNs and CRFs [39]. In particular, CRFs are implemented as recurrent neural networks (RNNs) [40] so that both FCNs and CRFs can be trained with back-propagation algorithms. The FCNs and CRFs are fine-tuned based on CNNs trained on image patches that are sampled from training tumor images to have balanced segmentation labels. Validation results based on BraTS 2013, 2015, and 2016 data sets have demonstrated that the end-to-end brain tumor segmentation method with integrated FCNs and CRFs could achieve promising segmentation performance.

CNNs for registration.

Deformable image registration, i.e. the process of transforming one scan to another, is a fundamental component in many machine learning pipelines, since it maps data from multiple individuals to a standardized reference system, in which a machine learning model can be applied [41]. Image registration is typically formulated as an optimization problem to seek a spatial transformation that establishes correspondence between a pair of images. Since conventional image registration methods solve the image registration problem using iterative optimization algorithms, they are time-consuming. Such a limitation could be overcome by deep learning based image registration methods built upon FCNs [42, 43]. Specifically, FCNs are used to encode spatial transformations between pairs of images to be registered, and optimal transformations for image registration are estimated by maximizing an image-wise similarity metric between fixed and deformed moving images, similar to conventional image registration algorithms. Since the FCN based image registration simultaneously optimizes and learns spatial transformations for the image registration, it can be directly used to register a pair of images, and the registration of a set of images is also a training procedure for FCNs so that the learned FCNs can be directly adopted to register new images by feedforward computation of the learned FCNs without any optimization. This FCN based image registration algorithm is essentially applicable to any image registration problem that can be solved using conventional image registration algorithms in that the FNCs are simply used to encode the spatial transformations that facilitate voxel-to-voxel correspondence between images to be registered. Successful strategies developed for the conventional image registration algorithms could be directly adopted in the FCN based image registration, such as inverse consistent image registration and diffeomorphic image registration. Moreover, the FCN based image registration can be implemented in a multi-resolution image registration framework to jointly optimize and learn spatial transformations at multiple spatial resolutions with deep self-supervision through typical feedforward and backpropagation computation [42, 43].

3. Applications in clinical neuroscience

This section focuses on application of machine learning methods in neuroimaging studies related to healthy and diseased brain development and aging. These studies reveal that imaging patterns of brain age through the lifespan can be established on an individual person basis, thereby identifying individuals who are potentially on an accelerated brain aging path. Moreover, imaging patterns of diseases can also be identified on an individual patient basis, thereby offering imaging biomarkers of diseases. Finally, these methods can define imaging subtypes, thereby contributing to precision diagnostics and individualized prognosis.

3.1. Brain aging and neurodegenerative diseases.

With the increase in population age, the prevalence of age-associated diseases and conditions is becoming higher. Brain aging has been linked to multifaceted structural and functional changes. Dementia, cognitive decline, vascular related changes in the brain as well as genetics are causes of deviation in individual’s brain aging trajectories [44, 45]. Neuroimaging coupled with machine learning methods has helped in building imaging markers to assess identifying subjects who are at higher risk of vulnerability for age-related changes, or who are on an accelerated brain aging trajectory by virtue of having brain age higher than their chronological age. Learning based brain age signatures are therefore becoming important biomarkers in clinical research [46-49]. In addition to producing a comprehensive overall brain age index, machine learning methods have demonstrated ability for single subject classification of specific age-related diseases such as Alzheimer’s disease, and hence have helped build more specific neuroimaging markers of AD and other diseases. In a recent report that reviewed 409 studies on AD classification using neuroimaging modalities, structural MRI showed to be the most utilized modality [50].

The availability of individualized indices of brain aging and of AD-like brain change have contributed to our understanding of the relationship between accelerated brain aging and dementia. Importantly, they have allowed us to obtain biomarkers that reflect an individual’s brain health, rather than deriving group statistics. Habes et al. used machine learning based approach in a large sample from the population-based Study of Health in Pomerania (SHIP) (age range 20-90 years, n=2705) to derive an imaging index of brain aging (SPARE-BA) [47]. The authors also derived an index trained with the ADNI cohort to capture patterns associated with Alzheimer's disease (SPARE-AD). Both indices were calculated independently in SHIP. Regression analysis showed that SPARE-BA was associated with factors that were different from those associated with SPARE-AD. Furthermore, individuals with accelerated aging displayed atrophy patterns that were partially overlapping with, but notably deviating from those typically found in AD. Finally, differential association with AD genetics further supported the hypothesis that distinct mechanisms might underlie lifetime brain aging and late-life neurodegeneration.

Advances in the field enabled integration of individualized indices of functional and structural brain aging together to identify pathophysiological process that affects accelerated brain aging. In a recent study from the Baltimore Longitudinal Study of Aging brain age was predicted using structural (SPARE-BA) and functional (fSPARE-BA) MRI. Advanced (or “accelerated”) brain agers were defined as being older than their chorological age, in at least one modality (structure or function), for further heterogeneity analysis. Using a mixture of expert methods, the study discovered five distinct subtypes of accelerated brain aging. One group (or accelerated brain aging subtypes) showed extensive structural, functional loss and high white matter hyperintensity load resembling AD-like changes. Other group showed mid brain and posterior medial atrophy and decreased insula connectivity, showing similar changes seen in patients with Parkinson’s disease. Another group had focal hippocampal atrophy, low white matter hyperintensity burden, and higher medial-temporal connectivity, supporting the hypothesis of high brain reserve counterbalancing brain changes in subjects with early stages of AD. Overall the study demonstrated that distinct patterns could be seen in accelerated aging, which could be attributed to early stages of vascular disease or neurodegenerative disorders [51].

To further dissect the heterogeneity patterns of neurodegeneration in AD, as well as its prodromal stage of MCI, Dong, et al. applied the CHIMERA and HYDRA methods described earlier, and discovered 4 distinct subtypes of AD and MCI, having different characteristics both in terms of amyloid and tau markers, but also in terms of cognitive decline and clinical progression [1], Fig. 3 summarizes some of those findings.

Fig. 3.

Heterogeneity of patterns of brain atrophy revealed by the semi-supervised learning methods CHIMERA and HYDRA. The majority (87%) of MCI and AD individuals differed from cognitively normal older adults in 3 different patterns (marked by numbers 2,3, and 4), whereas the remaining 13% displayed no anatomical differences from the control group. Interestingly, classification of MCI individuals into one of these 4 subtypes revealed differences in CSF biomarkers, cognitive decline and clinical progression to AD. This approach can therefore offer a more precise diagnosis of MCI patients, which has prognostic value.

In summary, machine learning has enabled the construction of brain aging and disease-specific imaging signatures, which allow us to parse individuals’ trajectories of normal and abnormal brain aging and hence develop predictive markers that can ultimately indicate an individual’s risk of clinical progression at early stages.

3.2. Applications in brain development and neuropsychiatric diseases

As described in Section 3.1, patterns of brain aging have captured processes of typical brain aging as well as deviations from it. Similar approaches have been applied more generally to broader age ranges, including brain development [46, 52, 53]. In the context of precision diagnostics, these indices relate an individual’s brain structure or function relative to normative curves, and hence they allow us to understand how deviations of an individual from these curves may relate to cognitive and clinical variables. We briefly describe two applications that portray the use of supervised learning for pattern regression and classification in the context of brain development during adolescence, as well as schizophrenia.

Imaging Signature of Brain Development

Studies during the past decade have shown that the brain develops in a highly coordinated way, which can be captured by machine learning indices on an individual basis. The brain development index (BDI) is an example of such an index, which offered a multi-modal MRI based summary index of brain maturation [53]. The index was calculated using structural and diffusion tensor imaging (DTI) scans of 621 healthy children and adolescents of age 8 to 22 studied as part of the Philadelphia Neurodevelopmental Cohort (PNC). The input data included gray matter and white matter tissue density maps in a common atlas space calculated from the T1-weighted scans of the subjects, and their functional anisotropy and apparent diffusion coefficient maps derived from the DTI scans.

A linear epsilon-insensitive support vector regression (ε-SVR) method was used for training the model [54], The BDI model found a consistent developmental trajectory within the PNC sample. It obtained a cross-validated correlation coefficient of r = 0.89 between the calculated BDI and the chronological age of the subjects, with mean absolute error of 1.22 years. Significantly, deviations from the age trajectory were found to be associated with the cognitive processing speed, showing potential for application of the BDI model in screening for individuals who might not be following typical brain development.

Structural imaging signature of schizophrenia

Several MRI studies have also provided evidence of patterns of gray matter deficits in schizophrenia [55], and have shown the feasibility of machine learning approaches for automatic classification of individual patients [56]. However, previous analyses were often limited to single site and/or relatively small samples.

In Rozycki, et. al., 2018, a multivariate pattern classification approach using an extensive set of features derived from structural MRI, coupled with SVM, was used to compute a neuroanatomical signature of patients with schizophrenia using MRI data pooled from 5 sites (941 adult participants, including 440 patients with schizophrenia), in order to provide robust, multi-site classification of schizophrenia [57]. A linear SVM was trained using a combination of voxelwise tissue density map values, volumes of anatomical regions, and regional shape, intensity and texture features. Across-sites harmonization for intracranial volume, site, age, and sex was done prior to model training using multilinear regression, estimating the parameters of the model from the control subjects only. Classification accuracy was evaluated for cross-validated classification using pooled data, as well as for leave-site-out prediction.

On pooled data, the classification model obtained 76% accuracy overall (AUC=0.84), with performance consistently better than single site classification for each site, indicating the importance and promise of using larger samples obtained by pooling multiple datasets. Also, leave-site-out validation obtained classification accuracy between 72 to 77 %, suggesting a robust generalizability across sites and patient cohorts.

4. Applications in Cancer research

In the field of cancer imaging, machine learning has witnessed a rapidly growing interest and use in a variety of problems, aiming to transition the field from rough diagnostic measures such as tumor volume, enhancement and perfusion, into more refined imaging phenotypes that relate to clinical variables such as response to treatment and clinical outcome (Radiomics), as well as underlying molecular characteristics of the cancer (Radiogenomics) [58]. These emerging fields convert clinical imaging data into a high dimensional comprehensive mineable feature space using a large number of automatically extracted data-characterization algorithms [59, 60], leading to patient-specific, rich and non-invasive biomarkers of cancer. Additionally, radiomic and radiogenomic methods allow for sampling the heterogeneity over the entire tumor, rather than of a single tissue specimen [61]. In the remainder of this section we discuss radiomic and radiogenomic methods in the context of brain and breast cancer.

4.1. Brain Cancer Radiomics

Radiomics [62] in neuro-oncology seeks to improve the understanding of the biology and treatment in brain tumors by extracting rich quantitative features relying on multi-parametric signals, texture parameters, shape properties, spatial patterns derived from atlas registration, and biophysical models of tumor growth from clinical imaging data. MRI plays an indispensable role in the radiomic analysis of brain tumors for various important reasons: i) MRI has an excellent capacity for the detection of soft-tissue contrast by revealing superior anatomic information; ii) multi-parametric MRI (mpMRI) non-invasively and non-destructively provides comprehensive information to characterize the tumor and surrounding abnormal tissues [63-66] repeatedly; and iii) each sequence of MRI reflects different characteristic of tissue, like T1-weighted post-contrast (T1-Gd) sequence highlights regional angiogenesis and integrity of the blood-brain barrier in the tumor. T2-weighted (T2) and T2- fluid attenuated inversion recovery (T2-FLAIR) sequences provide an in vivo assessment of the extracellular fluid in brain parenchyma [67], and indirectly tissue necrosis. Measures derived from DTI provide information regarding the water diffusion process in the brain, affected in part by tumor cells architecture and density [68]. Dynamic susceptibility contrast-enhanced (DSC)-MRI techniques quantify regional microvasculature and hemodynamics [69, 70].

The quantitative values derived from these imaging sequences can then be integrated using machine learning to develop models for tumor diagnosis, patient prognosis [71], relative tumor heterogeneity that can then guide clinical decisions [72], peritumoral heterogeneity/infiltration [73], and assessment of pseudo-progression [74] (Fig. 4). We briefly describe and discuss these applications of radiomics in the context of glioblastoma, the most aggressive brain tumor with very poor prognosis.

Fig. 4.

Illustrative overview of computational mpMRI analysis approach towards creating machine learning predictors of clinical outcome and molecular characteristics in glioblastoma patients.

Breaking down umbrella cancer diagnoses into subtypes

Brain tumors, and especially glioblastomas, exhibit significant molecular, histological, and imaging heterogeneity across and within patients, leading to several diagnostic and therapeutic challenges [75, 76]. The heterogeneous landscape, variable proliferation, different response to the same treatment [77], and resistance to standard treatment regimens suggest that the same treatment cannot be effective for every tumor. Thus, accurate non-invasive characterization of the heterogeneity of brain tumors is critical not only for better understanding of brain tumors, but also for developing personalized therapies to improve patient outcome, and for facilitating targeted enrollment into clinical trials. Radiomic analysis of mpMRI has been used as a powerful diagnostic method for in vivo characterization of diverse aspects of brain tumors and their micro-environment [78, 79].

Several efforts have documented evidence of heterogeneity in glioblastoma [80-82] which is the most lethal brain tumor with a median survival of ~16 months. Zinn et al. captured heterogeneity within the peritumoral edema region using conventional MRIs [83], followed by the use of additional features of other tumor subregions (enhancing tumor and non-enhancing tumor core) [78, 79, 84]. Itakura et al. extracted image features capturing the shape, texture, and edge sharpness of each tumor subregion from mpMRI images of de novo, solitary, unilateral glioblastoma patients [85]. Different phenotypic clusters emerged using consensus clustering on these image features. Each cluster mapped to a unique set of molecular signaling pathways using pathway activity estimates derived from the analysis of The Cancer Genomic Atlas tumor copy number and gene expression data. Later, Rathore et. al. [86] employed mpMRI data to characterize heterogeneity in glioblastoma patients, and applied a high-dimensional unsupervised clustering on a comprehensive set of features, reflecting imaging surrogates of tumor progression, angiogenesis, proliferation, cellularity, and peritumoral infiltration to identify intrinsic imaging phenotypes in a set of 263 patients. The authors discovered three distinct and reproducible imaging subtypes of glioblastoma, with differential clinical outcome and underlying molecular characteristics, including isocitrate dehydrogenase-1 (IDH1), O⁶-methylguanine–DNA methyltransferase (MGMT), epidermal growth factor receptor variant III (EGFRvIII), and transcriptomic subtype composition. The imaging subtypes provided risk-stratification substantially beyond that provided by the World Health Organization (WHO) classification [87, 88]. Within IDH1-wildtype patients (Fig. 5), the subtypes revealed different survival (p<0.001), thereby highlighting the synergistic consideration of molecular and imaging measures for prognostication [72].

Fig. 5.

Imaging subtypes of de novo glioblastoma (A), showing heterogeneity in overall survival beyond the IDH1 mutational status (B). Importantly, within the poor-prognosis IDH1- group of patients, the ones with the rim-enhancing imaging subtype (green) have far better prognosis. These imaging signatures point the way to a new era of imaging-based precision diagnostics which are synergistic with analogous genomic indices. ED = Peritumoral edematous/invaded region

Supervised analysis has also revealed heterogeneity and has characterized glioblastoma patients into different groups based on progression-free survival and overall survival [79, 80].

These studies, as well as similar studies that established molecular subtypes of glioblastoma [89], have moved the field from having a non-specific characterization of glioblastoma as a single umbrella disease to considering it a highly diverse cancer comprising multiple subtypes with imaging and molecular signatures that allow us to better estimate tumor aggressiveness and patient prognosis. More precise diagnosis and prognosis will eventually lead to personalized treatments.

Imaging signatures of the invading tumor front: Toward personalized targeted treatments

Malignant parenchyma in glioma outspreads beyond the enhancing borders of the tumor [90], and stereotactic biopsies have demonstrated the presence of infiltrating glioma cells in peritumoral edema region [65, 91, 92]. Using localized biopsies, Chang et al. have generated a model that illustrates a quantitative and significant relationship between MR signal of T1-postcontrast subtraction, T2-FLAIR, and apparent diffusion coefficient sequences and cell density in 36 glioblastoma patients [65]. Combining these relationships yielded a multiparametric model with improved correlation between the signal intensities and cell density, suggesting that each sequence offers different and complementary information. However, stereotactic biopsies are not always possible, therefore non-invasive methods that detect highly infiltrated tissues are considered useful for understanding the behavior of the tumor and characterizing the heterogeneity within edema region.

Beyond its role as a prognostic and predictive biomarker, radiomic analysis has shown potential at this front as well. Akbari et al. combined preoperative mpMRI signals from 31 patients using machine learning method to create predictive spatial maps of infiltrated peritumoral edema region [93]. Spatial maps representing the likelihood of tumor infiltration and future early recurrence were compared with regions of recurrence on postresection follow-up studies with pathology confirmation, leading to hazard ratio of ~9 on an independent replication cohort of 34 patients. Expanding this characterization to include advanced radiomic features of peritumoral voxels improved the hazard ratio to ~11 on an independent replication cohort of 59 patients [73, 94]. These studies highlight the role of machine learning based imaging signatures in developing personalized treatments that target the infiltrating front of the tumor beyond the visible margins, using either surgical or radiotherapeutic procedures.

4.2. Brain Cancer Radiogenomics

The emerging field of radiogenomics attempts to investigate if tumor molecular characteristics are manifested by distinct imaging patterns, which are derived using machine learning methods. Clinically-acquired mpMRI has generally been used in the past to extract simple tumor characteristics, such as volume, diameter, necrosis, and edema. However, in recent years, there is mounting evidence that extensive analysis of mpMRI can reveal visual (e.g., intensity, volume) and sub-visual features (e.g. morphologic, textural, kinetic), which when integrated via advanced machine learning and pattern analysis methods can identify underlying tumor molecular characteristics [70, 95-100]. Therefore, patterns extracted from mpMRI for the various tumor sub-compartments (i.e., edema, enhancing and non-enhancing regions [35, 37, 101, 102] can be used as in vivo surrogate imaging markers of molecular characteristics, without the need to radiochemically label molecular targets.

Imaging signatures of EGFR Mutations in Glioblastoma.

The EGFR pathway is one of the most important pathways involved in gliomas, and a target of many experimental treatments [103-115]. Therefore, up front evaluation of associated EGFR mutations in pre-operative scans, or even immediately after surgery, that could expedite mutational identification and hence patient stratification to associated clinical trials, is of significant importance. A data-driven approach to construct an mpMRI signature of the EGFRvIII mutation, utilizing multivariate machine learning (i.e., SVM) in independent discovery and replication cohorts of 75 and 54 de novo glioblastoma, respectively, is described in [116]. The accuracy of this mpMRI signature in classifying the mutational status in individual patients of the independent discovery (cross-validated) and replication cohorts was 85.3% (specificity=86.3%, sensitivity=83.3%, AUC=0.85) and 87% (specificity=90%, sensitivity=78.6%, AUC=0.86), respectively. Interpretation of the signature was consistent with EGFRvIII+ tumors having increased neovascularization and cell density, as well as a distinctive spatial pattern involving relatively more frontal and parietal regions compared with EGFRvIII− tumors (Fig.6). Notably, the finding of varying spatial distributions between EGFRvIII+ and EGFRvIII− tumors was also reported in another study [117, 118]. Although the spatial locations reported between the two studies were different, the larger sample size and the inclusion of a biophysical tumor growth model to account for the mass effect of these tumors in [116] potentially enables more robust statistics of spatial patterns.

Figure 6.

Descriptive characteristics of EGFRvIII mutational status in glioblastoma. (Figure adapted from Akbari et al. [106]).

The same approach was applied to other EGFR mutations, namely the missense mutations of EGFR A289D/T/V, EGFR R108G/K, and EGFR G598V. Specifically, EGFR A289D/T/V was found to be the most frequent event in both a retrospective cohort from the Hospital of the University of Pennsylvania (HUP) and from the publicly available data of The Cancer Genome Atlas (TCGA) [119, 120]. The identified mpMRI signature of the EGFR A289V missense mutants suggested an invasive and proliferative phenotype, which was then evaluated in mice models with implanted GBM tumors. Kaplan Meier survival curves comparing mice implanted with either U87 or HK281 tumors expressing either WT EGFR or EGFR A289V, demonstrated decreased overall survival, increased proliferation, and increased invasion. A mechanistic exploration revealed increased MMP1 expression driven by ERK activation leading to both the increased proliferation and invasion. Finally, the tumor driver status of EGFR A289V was demonstrated by in vivo targeting via mAb806, increasing animal survival and inhibiting tumor growth. This study serves as an exemplary case of an imaging signature generating hypothesis for further investigation towards opportunities for therapeutic development.

Imaging Signature of IDH1 Mutation in Glioblastoma

The presence of an IDH1 mutation is fundamentally important for the 2016 WHO-based classification of gliomas [88]. Having an imaging signature of IDH1 mutation is important, since knowledge of IDH1 mutational status at initial presentation can influence therapeutic decision-making, which will have a significant impact on patient care. In particular, as mutant-IDH enzyme inhibitors and immunotherapy targeting IDH-mutant tumor cells are developed, imaging to diagnose and follow IDH-mutant tumors can be important. Bakas et al. [121] has developed an mpMRI signature of IDH1 mutation in a cohort of 86 high grade glioma, with 15 of them harboring an IDH1 mutation. 342 quantitative imaging features were extracted per subject from all mpMRI, describing volume, morphology, texture, location, and biophysical growth model parameters, and 61 of them were selected as the most discriminative. These features were primarily including a distinct spatial location and texture descriptors. Radiographic interpretation of these features revealed that the IDH1 mutant tumor when compared to the IDH1 wild type, were located more in the frontal/occipital lobe with decreased blood flow and cell density. Using this imaging signature, our machine learning predictor classified IDH1 mutational status with an accuracy of 88.4% (sensitivity=66.7%, specificity=92.9%, AUC=0.81).

Imaging Signature of MGMT promoter methylation in Glioblastoma

MGMT promoter methylation is also an important prognostic indicator in glioblastoma, as it affects effectiveness of chemotherapy, with methylated tumors being relatively more responsive. Rathore et al [122, 123] has developed an mpMRI signature for the MGMT promoter methylation status towards addressing the limitation of the current determination, which can be limited by inadequate specimen or assay failures. The discovery of the signature was based on a retrospective cohort of 122 pathology-proven de novo glioblastoma patients with available pre-operative mpMRI data, 46 of which were MGMT-methylated. The status of MGMT promoter methylation status was obtained through pyrosequencing across 4 CpG sites in the MGMT promoter. A total of 330 features were extracted per subject, including descriptors of volume, morphology, texture, and location, 46 of which were selected following cross-validated forward sequential feature selection. Our marker was consistent with MGMT-methylated tumors having lower neovascularization and cell density, as well as a distinctive spatial pattern lateralized more to the left hemisphere compared with MGMT-unmethylated tumors lateralized to the right hemisphere (Fig.7). The cross-validated accuracy of our MGMT mpMRI signature was 84.43% (sensitivity=80.43%, specificity=86.84%, AUC=0.85).

Figure 7.

Descriptive characteristics of MGMT promoter methylation status in glioblastoma.

4.3. Breast Cancer Radiomics

Developments of machine learning tools and the increased use of Dynamic Contrast-Enhanced (DCE)-MRI for breast cancer screening has led to recent studies exploring the associations between radiomic features and breast cancer risk, suggesting a high correlation between quantitative radiomic features and conventional measures of breast parenchymal enhancement and breast percent density [124-126]. Applications of radiomic analysis of breast lesions have been widely explored, examining the efficacy of radiomic features as diagnostic, prognostic, and predictive breast cancer biomarkers [127]. Nie et. al. [128] extracted radiomic features capturing breast lesion texture and morphology to investigate their utility in distinguishing between benign and malignant lesions. The authors used an artificial neural network to select features with the highest diagnostic performance and found that features capturing heterogeneous texture and morphology allowed for a discriminatory AUC of 0.86. Similar conclusions reported by additional studies [129-133] suggest architectural differences between the imaging presentation of different breast lesions, leading to studies employing feature selection, classification, and unsupervised techniques to identify personalized prognostic and predictive radiomic breast lesion signatures.

To distinguish between invasive lobular and invasive ductal breast cancers, Waugh et. al. [134] extracted texture features from DCE-MRI images of breast lesions to train a cross-validated k-nearest neighbor model. Prospective applications of the trained model to a test set resulted in an AUC of 0.82. Further studies exploring the relationship between radiomic data and breast lesion characterization using classification methods have concluded that high throughput features extracted by radiomic analysis can delineate between molecular and histopathologic differences in breast lesions [135-138]. Wang et. al. [136] utilized support vector machines to discriminate between triple negative breast cancer and other histopathological subtypes characterized by morphologic and texture features, giving an AUC of 0.78. Expanding this characterization to include radiomic features from peri-tumoral background enhancement improved classification performance to an AUC of 0.88.

The independent and added value of radiomic analysis as a prognostic biomarker for breast cancer has been explored, investigating the relationship between radiomic signatures and tumor-free survival or tumor metastases [139-144], Li et. al. [142] performed multiple linear regression analyses on radiomic signatures generated from radiomic features characterization tumor morphology, enhancement, and kinetic behavior, to predict the risk of recurrence in 84 women. Using gene expression assays as surrogate measures of recurrence, the authors found a high association between radiomic signatures and risk of recurrence scores, giving an AUC of 0.88. Using a gene expression derived risk of recurrence as a surrogate measure for outcome, Mahrooghy et. al. [141] developed a set of radiomic pharmacokinetic features derived from the dynamic behavior of breast lesions as seen in DCE-MRI. Using a genetic algorithm for feature selection, the authors used logistic regression to examine the association between these features and risk of recurrence, reporting an AUC of 0.78. Ashraf et. al. [140] performed unsupervised clustering of radiomic features summarizing the kinetic behavior of 56 breast lesions as seen in DCE-MRI to identify intrinsic imaging phenotypes. The authors found an association between the imaging phenotypes and risk of recurrence, additionally finding that phenotype assignment augmented linear regression model performance when predicting risk of recurrence using a baseline model of radiomic kinetic features, improving an AUC of 0.77 to 0.82.

Beyond its role as a prognostic biomarker, radiomic analysis has also shown potential as a predictive biomarker, with machine learning methods implemented to predict response to neoadjuvant chemotherapy (NAC) [145-152]. Ashraf, et. al. [148] trained a leave-one-out logistic regression classifier to predict pathologic complete response (pCR) to NAC using radiomic kinetic features extracted from breast lesions. The authors reported classifier performance with an AUC of 0.84. Similar predictive ability was demonstrated by Johansen, et. al. [145], where both a Kohonen neural network (KNN) and probabilistic neural network (PNN) were trained using radiomic features summarizing image intensity in order to predict pCR to NAC. Both the KNN and PNN demonstrated a high predictive performance, with sensitivity and specificity ranging from 80%-92%. Chamming’s et. al. [149] utilized multiple logistic regression analysis to predict pCR using radiomic features extracted from contrast enhanced T1-weighted and T2-weighted MR images. The authors found a statistically signilficant association between the multi-parametric features and pCR, reporting a p-value of 0.03.

Radiomic analysis of breast cancer is limited by user-determined parameters for radiomic features and a dearth of large, publically available datasets for independent validation. As such, literature largely lacks analysis conducted on repeated datasets, preventing large scale radiomics standardization. Additionally, while machine learning advancements have allowed for the quantitative characterization of breast cancer towards various clinical goals, the relationship between radiomic features and underlying breast tumor biology has not been widely explored. Leveraging imaging signatures from complementary imaging modalities and exploring the relationships between radiomics, genomics, and proteomics have the potential to elucidate the biologic interpretation of radiomic analysis. Attempts to understand the biologic interpretation of radiomic features found to have high predictive or prognostic relevance could improve the value of imaging signatures for precision diagnostics.

Leveraging advanced machine learning methodologies for the characterization of breast tumor biology using radiomics can allow for a whole-tumor, quantitative, personalized assay to complement proteomic and genomic analyses to improve clinical decisions for breast cancer diagnosis, prognosis, and treatment response prediction.

5. Conclusion

We have presented a number of methodologies and experimental studies, focused on the topic of imaging-based precision diagnostics using machine learning. The work presented herein, as well as a plethora of similar studies in the literature, point the way to a new and exciting era in mpMRI. In particular, by leveraging the power of machine learning to derive broad panels of features that go well beyond current simplistic or visually-focused summarizations of MRI signals, and which allow us to synthesize complex signatures that relate to normal and abnormal biological processes, this field has begun to develop a number of imaging biomarkers of diagnostic and predictive value on an individual patient basis. We have presented several examples that underline the potential uses of such imaging signatures in dissecting disease heterogeneity, and in breaking broad umbrella disease categories into more precise subtypes. These subtypes will better allow us to match treatments to patients, and to derive more accurate prognostic indicators[116].

Fig. 2.

A schematic flowchart of FCN based image registration. FCNs are used to encode a deformation field between a pair of images to be registered, and optimal transformation for the image registration is estimated by maximizing an image-wise similarity metric between the fixed and deformed moving images.