Artificial Intelligence in Computer Vision: Cardiac MRI and Multimodality Imaging Segmentation

Abstract

Purpose of Review.

Anatomical segmentation has played a major role within clinical cardiology. Novel techniques through artificial intelligence-based computer vision have revolutionized this process through both automation and novel applications. This review discusses the history and clinical context of cardiac segmentation to provide a framework for a survey of recent manuscripts in artificial intelligence and cardiac segmentation. We aim to clarify for the reader the clinical question of “Why do we segment?” in order to understand the question of “Where is current research and where should be?”.

Recent Findings.

There has been increasing research in cardiac segmentation in recent years. Segmentation models are most frequently based on a U-Net structure. Multiple innovations have been added in terms of pre-processing or connection to analysis pipelines. Cardiac MRI is the most frequently segmented modality, which is due in part to the presence of publically-available, moderately sized, computer vision competition datasets. Further progress in data availability, model explanation, and clinical integration are being pursued.

Summary.

The task of cardiac anatomical segmentation has experienced massive strides forward within the past five years due to convolutional neural networks. These advances provide a basis for streamlining image analysis, and a foundation for further analysis both by computer and human systems. While technical advances are clear, clinical benefit remains nascent. Novel approaches may improve measurement precision by decreasing inter-reader variability and appear to also have the potential for larger-reaching effects in the future within integrated analysis pipelines.

INTRODUCTION

Anatomical segmentation, or the process of identifying local structures to determine size or shape, of medical imaging has a major role throughout clinical cardiology. All modalities of cardiac imaging incorporate this segmental information to determine critical measurements such as cardiac function, wall thickness, and other metrics of risk assessment. Artificial Intelligence (AI) based automated cardiac anatomical segmentation has received extensive interest in recent times due to the potential to automate, streamline, or improve assessment of medical imaging. At the same time, many of these projects risk falling into the concept of the “AI Chasm,” defined by Topol and Keane as scientifically sound or accurate algorithms but with limited actual clinical utility.[1, 2] In this review, we provide a context of anatomical segmentation, a survey of the past 5 years of publications regarding cardiac segmentation and AI, and a perspective on future directions of AI cardiac segmentation within clinical cardiology. One of our primary goals is to clarify for the reader the clinical question of “Why do we segment?” in order to understand the question of “Where is current research and where should be?”.

THE CLINICAL USE OF CARDIOVASCULAR SEGMENTATION:

Traditional Approaches to Cardiac Segmentation

Two-dimensional echocardiography (echo) emerged in the late 1970’s with significant advancement in our ability to assess and quantify cardiac structures. Subsequent guidelines recognized the importance of the standardization of measurements and generated the 16-segment model, as well as definitions of standard distances and measurements.[3] These segmentation models do not represent precise anatomical contouring, however, provide a framework to standardize interpretation which persists to this day. This purpose can be divided into multiple steps: 1) the use of standardized segmentation models allows for quantification of qualitative observations, 2) this creates the ability to directly compare between patients/timepoints, anatomical structures and regions such as coronary distributions versus myocardial regions across imaging, and 3) these comparisons between patients define clinical diagnoses, predict outcomes, and monitor therapies.[4]

In the early 1990’s, guidelines for standardized views and definitions within multi-modality tomographic imaging followed[5] with the explicit recognition of the need to standardize interpretation between modalities such as single photon emission computed tomographic (SPECT), positron emission tomographic (PET), cardiac magnetic resonance (CMR), and cine computed tomographic (CT) imaging. In later years, the current 17 segment American Heart Association (AHA) model was standardized within recommendations and also included echo.[6] Coronary segmentation underwent a similar growth from the 1975 original AHA segmentation[7] to cardiac CT guidelines with only minor modifications,[8] thus creating a standard nomenclature to specify locations of disease.

The ability to pursue precise anatomical contouring instead of regional segmentations has emerged with improved spatial and temporal resolution but faces unique barriers. Guidelines related to segmentation explicitly recognize limitations within anatomical segmentation methods. Increased uncertainty and lack of standardization for certain tasks are highlighted within ventricular segmentation by high-detail intra-ventricular structures such as trabeculations, papillary muscles, and the moderator band. Echo and CMR guidelines agree on the inclusion of the trabeculations and moderator band within the right ventricular (RV) cavity but disagree on the inclusions/exclusion for papillary muscles and trabeculae for left ventricular (LV) mass quantifications.[9, 10] These differences emphasize the balance between anatomical precision provided by specific modalities, as well as consistency with historically established ranges and cutoffs.

Computer Vision and Semantic Segmentation

The goal of Computer Vision was succinctly described in Simon Prince’s classic book as “to extract useful information from images.”[11] This has conceptually existed since the 1960’s but has experienced massive recent advances with the development of convolutional neural networks (CNN), exponential increases in computational processing power as described by Moore’s law,[12] and increasingly available large image datasets.[13] As a brief introduction to terminology relevant within artificial-intelligence based computer vision, neural networks are mathematical algorithms which enable machine learning. These structures are loosely based on the human brain, and in fact, the basis was initially described by McCullough and Pitts in 1943, not in the context of programming, but as a means to model neuronal activity.[14] These networks are collections of artificial “neurons”, which receives a set of inputs, passes these inputs through “hidden layers” which perform mathematical processes on the input, to result in a discrete output or set of outputs. By providing data of matched inputs and outputs, the hidden layers are “trained” by incremental adjustment of the hidden layers in order to “learn” the association between the input and output. By combining multiple neurons together, high accuracy of these networks for even complex tasks can obtained as long as the computing power to train and execute the networks are sufficient. The relations, functions, and quantity of the different elements of the neural network determine the network architecture, which in turn plays a major role in its success for a particular task.[15]

Within computer vision, there are multiple potential tasks, which include image classification, localization, detection, and modification.[16] Cardiac imaging characterization, particularly for assessing form and function, most frequently fall under the concept of “semantic segmentation”. This task classifies objects in an image to concepts, mapping each pixel in the original image to a set of constrained values, and is particularly well suited for CNNs. A convolution is a function which is performed in a stepwise fashion across an image which can used to detect imaging features. These features can be simple, such as a horizontal or vertical edge, or a gradient; by combinations of simple features, more complex features can be created to identify and classify everyday objects, such as cars, animals, or faces.[17] Within medical imaging-related semantic segmentation, the network receives the image pixels as inputs and pixel-wise labels as outputs to train the model, and the CNN analyzes the anatomical features to assign labels (e.g. myocardium, aorta, chambers, valve, everything else).[18]

The overarching term for architectures that utilize series of convolutions in order to produce the output is a CNN; however, structural changes may improve computational efficiency and accuracy. Specific well-known architectures for semantic segmentation exist.[19] Fully convolutional networks (FCN) utilize an encoder-decoder structure by downsampling in the encoder, which is an efficient way to create feature maps for semantic segmentation, with upsampling in the decoder to recover resolution to the level of the original image. This provided a significant efficiency benefit, as pixels of the same label typically localize together within a segmentation, but had difficulty creating smooth segmentation edges due to the degradation of resolution in during the downsampling.[20] Skip connections between the different resolution levels of the encoder and decoder arms was added in the U-Net structure, which has been extremely popular and broadly used. This allowed for retention of higher resolution information, which helped resolve the issue with edge resolution.[21] Other modifications have included the addition of non-segmentation modules such as residual layers which improve the ability to train networks with more layers[22] and different types of convolutions which improve spatial context (dilated/atrous convolutions).[23, 24] Architectures are provided to give clinical readers recognition of common methods; however detailed discussion of the architectures are outside of the scope of this article and can be found in these excellent reviews.[25–28]

Defining the Challenges in Human Evaluation of Medical Imaging

In order to define the optimal direction of artificial intelligence-based cardiac segmentation, we need to define the problems with human vision. Human vision’s limitations may be best conceptualized as inherent limitations and tradeoffs in realms of physical perception and time/attention. Attention-wise, with an infinite amount of time, human viewers could scrutinize all portions of an image systematically; however, clearly the needs of clinical practice make this impossible. In human evaluation of medical imaging, attention fatigue is combatted by efficiency-based techniques such as search patterns and standard frameworks. Even with these techniques, weaknesses persist. For example, inattentional blindness where priming to search for a specific finding makes us less likely to see unrelated findings but clinically relevant findings, such as incidental malignancy.[29] Computational approaches do not have the same limitations with regard to fatigue, boredom, or distraction.

From a perception standpoint, edge detection and motion tracking are typically considered relatively simple tasks for human observers,[30–34] whereas detection of contrast differences are weaker, requiring high contrast-to-noise (CNR) ratios of 3–5 per the Rose Criterion.[35] The primary challenge to anatomical contouring is less the ability to identify the appropriate location of the contours and more the time and effort required to draw them. Within this context, semi-automatic and automatic methods have been successful. Conventional approaches have included thresholding by intensity and clustering methods with minimal pre-existing inputs; methods with moderate user-inputs such as region-growing and classification; or methods with more extensive pre-existing rules/knowledge, such as deformable model and atlas-guided methods. More recently, neural networks and probabilistic models such as Markov random fields have been introduced to augment conventional approaches.[36]

A SURVEY OF CARDIOVASCULAR SEGMENTATION AND ARTIFICIAL INTELLIGENCE

Search Parameters and Results

We performed a survey of papers published within the past 5 years by searching in PubMed for “cardiac segmentation” and “artificial intelligence”, limited to non-review articles, with 300 manuscripts identified (Figure 1). Manuscripts were individually reviewed and limited to cardiac anatomical segmentation (i.e. not inclusive of heart sound, electrocardiogram, or non-cardiac anatomical segmentation), which resulted in 149 papers (Supplemental Table 1). These papers were reviewed for their imaging modalities, basis architecture, number of patients used for training and testing, source of patient data, anatomy assessed, purpose of segmentation, and whether or not the code was explicitly released as noted in the manuscript. Information was not available for all papers due to either not being reported in the manuscript or limited access to the manuscript, and therefore we primarily report metrics as a percentage of the total manuscripts which were assessable within each aspect.

Figure 1: Summary figure.

Semantic segmentation exists within computer vision, and more recently has used deep learning techniques to augment traditional methods. While classical segmentation defined clinically relevant regions, anatomical segmentation follows anatomical boundaries, and enables more precise definitions of structure and function, which may be integrated into analysis pipelines. The 5-year literature survey shows recent growth in AI and cardiac segmentation, with MRI as the most common modality. External dataset use was most frequent with the ACDC 2017 set, and U-Net was most commonly used for segmentation. MRI: Magnetic resonance imaging, CT: Computed Tomography, TTE: Transthoracic echocardiography, CXR: Chest X-ray, CNN: Convolutional neural network, FCN: Fully convolutional network, ACDC 2017: Medical Image Computing and Computer Assisted Interventions (MICCAI) 2017 Automated Cardiac Diagnosis Challenge, MICCAI 2009: MICCAI/STACOM Sunnybrook Cardiac MR 2009 Left Ventricle Segmentation Challenge, LV Seg 2011: MICCAI/STACOM Left Ventricle Segmentation Dataset and Challenge from 2011

The number of AI-based cardiac segmentation manuscripts has been increasing significantly over time, with the lowest in 2016 (n=11) and increases to 48 in 2019 and 45 in 2020. Slightly over half of the manuscripts were published in clinically-oriented journals (54%), with a significant remaining portion published in technically-oriented journals. A large portion of the technically oriented journals were in IEEE journals (26%), in particular IEEE Transactions on Medical Imaging. Within the articles, the plurality focused on cardiac MRI segmentation (49%), both individually and combined with other modalities, followed by CT-based segmentation (27%) and echocardiography (13%). While most papers introduced novelties within their neural network architectures, the most frequent underlying architecture was U-Net (37%) followed by FCN (12%), though a significant portion (14%) did not report a basis architecture or stated that the architecture was an original CNN. Approximately one quarter of the papers explicitly released their code, with a direct reference in the manuscript.

Notable innovations were made in specific manuscripts. One recent paper incorporated a Monte-Carlo dropout in conjunction with a U-Net to provide uncertainty estimates for the segmentation.[37] This approach may help obviate unpredictable failure, and provide clinicians some degree of understanding of the expected quality of images. Other approaches combined images from separate views,[38] separate modalities,[39, 40] and different contrasts[41] to improve accuracy of segmentation or advantageous transfer of information from one modality to another. Serial processing such as image quality upscaling prior to segmentation,[42] or pipelines such as image processing pipelines,[43] quality control pipelines,[44] radiomics pipelines,[45] or even integration with robotics frameworks for procedures[46] were noted. Also, while diagnoses directly suggested by left ventricular mass or anatomy were proposed, we found segmentation that leads to indirect diagnosis of diseases such as pulmonary hypertension as a particularly intriguing approach as well.[47]

One particularly notable paper which balances technical innovation with a strong clinical application is presented by Augusto et al.[48] In this paper, the research team sought to automate the measurement of maximal wall thickness (MWT) measurements. A U-Net based model was used to segment the inner and outer myocardial contour, and subsequently solve a Laplace equation to determine the true MWT. The advance from anatomical segmentation to creational of a MWT line is conceptually simple but represents an important advance. Training a model to precisely mimic segmentations drawn by humans limits the maximal performance of the model to equal to or below the human-generated training data, and thus the segmentation model becomes a matter of convenience, rather than progress. The use of a segmentation model to define the potential borders of line enables the machine performance to become “superhuman” by “outworking” human observers. That is, by contouring every slice in the short axis stack, and solving the Laplace equation for every slice in order to identify the MWT, the model is systematic to a degree that would be infeasible for humans to perform during clinical reads. The authors highlight that with more precise wall thickness measurements, the sample size needed for clinical trials could be significantly smaller (2.3 times smaller to detect a 2 mm difference), which is a concrete benefit of using this measure.

Training and Testing Data Sources

The source of training and testing data varied, with slightly more than half (56%) reporting use of local institution-specific data, and a similar portion (52%) utilizing external datasets (a small proportion used both). The publicly available datasets varied in modality and implementation. Given the high proportion of MRI manuscripts, the most frequently used dataset (29% of the manuscripts using external data) was the Medical Image Computing and Computer Assisted Interventions (MICCAI) 2017 Automated Cardiac Diagnosis Challenge (ACDC), dataset which consisted of MRI images from the University Hospital of Dijon.[49] Notably, MICCAI/STACOM challenges including the Sunnybrook Cardiac MR 2009 Left Ventricle Segmentation Challenge[50] (17% of external data), and the Left Ventricle Segmentation Dataset and Challenge from 2011[51] (11% of external data) were also well-represented, with the UK-Biobank cardiac MRI dataset growing in popularity in more recent years[52] (11% of external data). Models most frequently focused on structure and function of the left ventricular myocardium and blood pool (50%), while right ventricle, and atrial segmentation was less popular. Outside of myocardial segmentation for cardiology and cardiac-imaging specific purposes, there was a notable group of applications based around anatomical planning for radiation oncology radiotherapy, though this was lower in frequency (9%). Applications outside of myocardial segmentation were even less common, however, epicardial adipose tissue,[53] coronary arterial lumen,[54] and invasive coronary imaging was also noted.[55]

It is notable that the most frequently used datasets are relatively small (100 training cases for the ACDC and MICCAI/STACOM 2011 datasets, 45 for the Sunnybrook MICCAI 2009 dataset), emphasizing the availability of data as a key bottleneck. In some ways, the popular use of smaller datasets has brought forth extra creativity in data augmentation, which should have durable benefits as limited training set sizes within medical imaging will continue into the near future. At the same time, we are beginning to see larger datasets in cardiovascular imaging, which can further advance the field. The UK Biobank and UK Digital Heart datasets contain >48,000 and ≈1200 patients’ CMRs respectively,[56, 52] and the Cardiac Acquisition for Multi-structure Ultrasound Segmentation (CAMUS) dataset[57] or the EchoNet-Dynamic dataset which contains 500 and 10,030 patients’ echocardiograms respectively. These datasets highlight the vanguard of training with larger training sets.[58]

FUTURE DIRECTIONS IN CARDIAC SEGMENTATION

Segmentation as the Primary Problem or a Pipeline Task

The popularity of the ACDC dataset highlights the major role that competition events and leaderboards have had in pushing forward the field of computer vision both within and outside of medical imaging. Within the primary publication, they summarize the accuracy of the top segmentation models from the ACDC competition.[49] These models had similar accuracy (by Dice score and Hausdorff distance) to inter/intra expert reader variability, and therefore a strong argument could be made that the problem is solved to within human accuracy. Areas of high-error rates were seen in the base and apex, which inherently have more uncertainty due to partial volume effects and variability in anatomical structure.

We view these “competition” datasets as providing a major benefit to medical computer vision. The competition approach with pre-labeled datasets help break down barriers between clinical imaging experts and non-medical computer vision experts, and provide a cross-pollination where non-medical computer vision experts benefit from access to expert labels in a curated dataset which would be otherwise inaccessible due to privacy protections, and clinical imaging experts benefit from additional computer vision expertise to determine top architectures in an appropriate context for medical imaging. This functions as a decentralized way to enable multidisciplinary collaboration, which is essential for accurate and clinically relevant models. Acknowledging the success of segmentation accuracy relative to human expert contours raises questions about how these models can be innovated further to enhance clinical benefit, affect prognostic and diagnostic assessment, and improve delivery of care. Further collaboration will be needed in order to optimally direct the field of medical computer vision.

Next Steps in Data Availability

In order to further progress towards clinical relevance and connection to clinical outcomes, issues regarding access to larger quantities of diverse data are growing in relevance. These needs are balanced against ethical implications regarding patient protection, codification of bias, data security, and ethical application of models. The ethical concerns are now being formally addressed.[59] We believe that these concerns will continue to grow with the field of machine learning in cardiac imaging. With respect to data availability, while the small ACDC dataset reported excellent results, larger datasets, have already begun to provide dividends within deep learning. In particular, the UK Biobank with expected enrollment of 100,000 patients with at least 10,000 follow up scans can be expected to provide unique benefits related to the data quantity and availability.[52] In general, large publicly-available datasets can face barriers related to protection of patient privacy, however, a new approach called “Federated Learning,” whereby the transfer of partially trained models instead of data is performed. These models can be trained on local datasets, and then transferred back to a secondary site for further training or a central site for aggregation, to avoid public release of patient data.[60, 61] This is particularly important for obtaining diversity within training datasets, both in the demographic (less-frequent racial/ethnic groups) as well as diagnostic (less frequent disease groups) as use of biased datasets will result in biased models.[62]

Next Steps in Model Explanation

In order to effectively translate technical advances into clinical application, the ability to understand a model’s decision-making process is important. Understanding our tools allows us to predict their strengths, weaknesses, and applications. With the use of complex architectures, increases in accuracy have often come at the cost of model explicability. That is, we know the training input, the model type, and can rate the output, but the actual process of “decision-making” is obscured to our understanding. Apart from overfitting to training sets, this can result in unexpected failures. There have been advances in attempting to explain computer vision models, most notably Grad-CAM, which can create an attention-based heatmap for multiple CNN types, in order to show how classification tasks are being decided.[63] While this is not directly applicable to semantic segmentation models, this movement highlights the growing understanding that both performance and interpretability is critical for broader uptake of computer vision model tasks. Within (and outside of) medical imaging, there are known examples of “specification gaming” where models literally address the stated objective but in ways that are not intended or useful by the designers.[64] Examples include identifying increased risk of skin lesion malignancy by the presence of a ruler in the picture,[65] or predicting pneumonia by identifying the location of a chest X-ray’s acquisition instead of the lungs themselves.[66] These outcomes highlight the need for model interpretability as a backstop to unpredictable behavior. These concerns are currently being addressed in the regulatory environment, with active international work through the International Medical Device Regulators Forum (IMDRF), and the US Food and Drug Administration, which continues to iterate on definitions and regulations for Software as a Medical Device.[67–69]

Future Directions: Replacement or Assistant

Clinicians recognize that we need help to deliver optimal medical care – The demands on our time relative to the accessible patient-related and evidence-related data have never been so high. At the same time, the fear of obsolescence has been historically deep-rooted in the relationship of the physician to the rapid pace of medical advancement, let alone new technology.[70–72] As these segmentation models are incorporated into more complex decision-making machines, the question arises of whether or not the resultant product will appear in a form more consistent with a clinical-decision support system to a physician, or as an autonomous decision-making system structure meant to replace physician judgement.

We believe that even in the face of technological advancement, the most optimal combination will remain the partnership of human and machine to bridge the AI chasm. This is not discounting the ability of advances in computational power, which will allow more efficient solutions. At their heart, the problems of clinical medicine are difficult to define in a way that is solvable in a formal quantitative manner while still remaining patient-focused. The imperfect nature of patient interviews, symptoms, and subjective experience create an environment which resists the development of formal rules. The abstract concept of patient preferences creates a unique solution for every case; that is, even death, or major adverse cardiac events, are not necessarily the endpoints that are most important to an individual patient. This creates a “wicked world”[73] type of scenario where the difficulty is not in solving the problem, but defining the entire question as well as appropriate benchmarks of success. The integration of clinician and computer should be the optimal direction, and AI-based segmentation provides a foundation for this interaction.

CONCLUSIONS

Visualization of cardiac structures using medical imaging has significantly improved over the last thirty years, increasing the relevance of segmentation, but also the amount of manual time spent on high-detail anatomy. The use of computer vision technologies to automate segmentation has grown out of this need, and while somewhat effective with semi-automatic statistical methods, has progressed significantly with the advent of higher computing power, data availability, and CNNs. The task of cardiac anatomical segmentation has experienced massive strides forward within the past five years, and provides a basis for streamlining image analysis, as well as a foundation for further analysis both by computer and human systems. Within the recent literature, cardiac MRI is the most common data source and U-Net is the most commonly used deep learning semantic segmentation architecture. The availability and visibility of datasets remains relevant, highlighted by the significant impact of MICCAI/STACOM challenges; however, larger volume datasets are becoming more common and will likely have a major effect. While technical advances are clear, clinical benefit remains nascent. Novel approaches may improve measurement precision by decreasing inter-reader variability and appear to also have the potential for larger-reaching effects in the future within integrated analysis pipelines. The future relationship of the physician to AI systems is uncertain, however, additional direction is needed for clinically relevant development of AI models, within which cardiac segmentation will play a major role.