AI in MRI: A Case for Grassroots Deep Learning

Our principled and foundational approaches with Cartesian sampling, Fourier reconstruction, and assessment of contrast to noise with well-defined point spread functions seem quaint when compared with the rapid innovation in data-driven interpretation and inference. Futurists are ever present in predicting that artificial intelligence (AI) is already here and, therefore, Radiology, if not all of society, will never be the same. In this special issue, we present eighteen perspectives from researchers on the frontline of applying machine learning to the context of magnetic resonance imaging (MRI), spanning perspectives across acquisition, processing, and modeling.

Much emphasis has been placed on the data hungry nature of modern machine learning approaches [1, 2]. National efforts are rising to the challenge to produce exceptionally well characterized datasets that drive innovation for population imaging [3–5] and specific clinical use cases [6]. In this special issue, a common theme among the works is that they are not Herculean efforts. This collection does not capture the results of “big science”; the longest author list has a modest 13 authors (mean: 7.6, median: 8 authors/paper). Rather, these studies achieved successful data-driven approaches with tractable datasets that are commonly captured in modern research labs.

We find the results in this issue as an encouraging counterpoint to the popular drumbeat for more data, structured learning, and larger teams. While we acknowledge and emphasize the importance of large datasets and flagship learning efforts for establishing game-changing paradigms and effecting clinical change [7, 8], we choose to see herein the importance of innovation and characterization on smaller efforts.

The breadth and diversity of the applications in this issue illustrate the depth of impact that AI is having on nearly every aspect of MRI study design. Deep learning algorithms are leading to efficient methods to handle the extreme variation in signal contrast, size, and anatomy with the youngest people [9, 10], and new approaches are enabling routine quantitative mapping of small, motion-prone spinal cord anatomy in adults [11]. Similar segmentation approaches are being applied to substantially larger, but also highly variable, abdominal anatomy to enable mapping of body composition [12]. Machine learning methods are even pushing the boundaries between sequences through synthesizing or “hallucinating” acquisitions that could not be ordinarily be acquired [13–15] and re-interpreting low resolution images to achieve super resolution[16].

Data driven approaches are having similar and transformative impacts on how we handle imaging data after it is acquired. Non-rigid image alignment, or registration, has long been a computationally intensive step (e.g., hours to align two 1 mm isotropic brain scans) [17], but deep learning approaches are increasingly able to bypass iterative search in favor of a single passed learned function [18] and may offer opportunities to assess uncertainty in alignment [19]. Similarly, image noise interpretation and systematic approaches to mitigate its negative effects have been of intense interest for over three decades [17]. Yet, the relatively direct application of convolutional networks appears poised to overtake years of expertly designed systems [20] with the added benefit of being strikingly fast to execute [21]. Similarly, hardware and software upgrades have been a mixed blessing. In our experience, new systems improve contrast, reduce artifacts, yield greater uptime, and enable new imaging sequences, but creating consistent images from the new and old systems has been an art (and a statistical confound). The same family of algorithm designs that are used for synthetizing contrast or aligning images also show great promise at reconciling differences across scanners and could transform how studies apply traveling subjects for study calibration [22].

Finally, MRI offers unique capabilities for capturing dynamic and extended processes within tissues. In diffusion tractography, the core idea of connecting points with locally consistent orientation structure [23] has opened the entire field of structural connectivity. Recent data driven innovations are enabling digital reasoning at higher conceptual levels consisting of extended bundles, which may enable the community to resolve long standing issues specificity and sensitivity [24]. Similarly, machine learning approaches are changing the way we consider functional connectivity from seed-based correlations or linear maps of consistency of resting state functional MRI to allow us to capture and characterize non-linear patterns of brain dynamics [25]. Meanwhile, data driven approaches, including machine learning and radiomics textures, are offering new opportunities for interpreting dynamic contrast enhanced imaging [26]. When combined with specific clinical outcome measures of interest (e.g., Alzheimer’s Disease diagnosis), the data driven approaches are offering intriguing performance for detecting abnormalities and improving precision diagnostics[27–29].

Only in the last few years has data driven learning become feasible for nearly all graduate students. We have seen exceptional performance with deep learning methods on the generic tasks of regression[30], classification [31] and segmentation [32]. At their core, the authors in this special issue are applying core learning from machine learning to Radiological data. Yet, the practical and scientific contributions of solid characterization of these approaches is critical for the advancement of science.

The fragility of data-driven methods is unfortunately also becoming understood [33]. We would like to take this editorial opportunity to emphasize the importance of evaluating algorithms on imperfect datasets. In clinical practice, variations in imaging sequence, artifacts, patient characteristics are ever present. We find that practical / qualitative impressions of an algorithm’s performance are not based on the absolute performance on ideal data, but rather the rate of catastrophic failure when “minor” input variations occur. As designers and users of machine learning, we need these near-outlier data to be available during training and comparison. Yet, simulation or construction of real-world data is incredibly difficult.

Fortunately, code sharing/model zoos are becoming available for both PyTorch[34] and TensorFlow [35] to allow users to apply methods from literature to their data (e.g., https://modelzoo.co, https://github.com/NifTK/NiftyNetModelZoo, etc.). These are important scientific contributions that highlight the needs for proper ontologies and commonly available representative datasets. Challenges are becoming an increasingly successful manner of pursuing community building science [36]. As reviewers of articles and editors of journals, we must realize that replication of model performance on new datasets or application of existing paradigms at new sites is important work. Perhaps controversially, we posit that “This is the second/third work to show…” should not be a negative clause to include in a medical imaging abstract. Literature review mechanisms need to make it clear not only when an article has been cited, but also indicate context for when a study has reproduced, applied, or validated.

Similarly, imaging datasets represent humans with diverse life experiences, genetic/environmental backgrounds, medical histories, and symptomatic presentations. Structuring learning in a classical classification/regression/segmentation framework is convenient, but limiting in terms of the model bias and ability to capture personalized (“n=1”) factors in a data driven approach. Electronic medical records, albeit limited and imperfect, offer one such avenue to capture personalized information. Structured coding of “phenomes” [37], medical procedure spaces [38], laboratory values, and medicine dosages appears to be a convenient/promising avenue for exploration while integration of imaging and natural language processing evolves [39]. We are particularly intrigued by work that seeks to align automated MRI analysis more closely with the patient-context with which a human radiologist would interpret a scan.

In conclusion, despite the numerous successful AI in MRI research efforts and grand proclamations, there are numerous obstacles to researchers seeking to apply AI. The deep learning frameworks provide train tracks to apply machine learning in MRI. Yet, it is critical to note that carefully curated databases have biases and it is exquisitely difficult to predict the in-the-clinic imaging conditions without challenging multi-site studies. Moreover, current machine learning approaches often suffer from a lack of transparency as to which factors are driving inference. As scientists, we need the ability to rapid prototype machine learning algorithms (fail-fast, fail-hard) while precisely articulating the methods in a reproducible manner and reporting on both successes and failures (to minimize publication bias). With an increasing number of possible approaches, we must remain skeptical of the work of others (as readers/editors) and of our own work (as authors). In general, computation is vastly less expensive than the human time necessary to interpret the results, so we must look for effective platforms to use computation such that we can make more effective use of our efforts. While free open source software has the potential to bring communities together, the pathway for patient impact / commercialization is not always apparent. Clear institutional / industry standardization of research-compatible licensing pathways could simplify academic-industry collaboration and streamline accountability for intellectual property in the space of medical image processing algorithms. As captured in this special issue, clever application of data-driven machine learning to focused problems in MRI can have direct impacts on our capabilities to perform, interpret, and generalize MRI studies. As scientists working with magnetic resonance imaging, we live in interesting times.