Artificial intelligence and automation in valvular heart diseases

Abstract

Artificial intelligence (AI) is gradually changing every aspect of social life, and healthcare is no exception. The clinical procedures that were supposed to, and could previously only be handled by human experts can now be carried out by machines in a more accurate and efficient way. The coming era of big data and the advent of supercomputers provides great opportunities to the development of AI technology for the enhancement of diagnosis and clinical decision-making. This review provides an introduction to AI and highlights its applications in the clinical flow of diagnosing and treating valvular heart diseases (VHDs). More specifically, this review first introduces some key concepts and subareas in AI. Secondly, it discusses the application of AI in heart sound auscultation and medical image analysis for assistance in diagnosing VHDs. Thirdly, it introduces using AI algorithms to identify risk factors and predict mortality of cardiac surgery. This review also describes the state-of-the-art autonomous surgical robots and their roles in cardiac surgery and intervention.

Introduction

Artificial intelligence (AI), which is a branch of computer science that attempts to create machines that perform tasks as though they possessed human brainpower, and this is gradually changing the landscape of the healthcare industry [1]. Clinical procedures that were supposed to, and could only be performed by human experts previously can now been carried out by machines in a more accurate and efficient way. Recently, cardiologists and radiologists have cooperated with computer scientists and have developed a fully automated echocardiography interpretation system for clinical practice [2]. Cardiovascular diseases affect 48% of the population (≥ 20 years old) and is the leading cause of death [3]. Echocardiography is an effective method to monitor the state of the heart and allows early diagnosis before the onset of symptoms. However, in primary care clinics and poorer regions, there are inadequately experienced radiologists for echocardiography interpretation, let alone patient follow-up. There is group of researchers who trained AI systems using over 14,000 echocardiograms for multiple tasks including view classification and quantification of chamber volume, ejection fraction and longitudinal strain. The AI system validated its accuracy in diagnosing several cardiac diseases after testing on over 8000 cardiograms obtained from routine clinical flow [2]. Such an AI system holds great potential for the transformation of current clinical practice models as well as the popularization of quality healthcare.

The above example is only the tip of the iceberg in the application of AI in medicine for over half a century. Since “AI” was first proposed by John MacCarthy at the Dartmouth College Conference in 1956, great efforts have been made for applying AI to almost all phases of clinical practice. In 1960s, the concept of “computer-assisted diagnosis” emerged with attempts to use mathematical formalism to interpret clinical problems, although they have little practical value, it has paved the way for subsequent expert systems [4]. In 1970s, researchers began to shift attention to studying the reasoning process of clinicians and simulate it on computer systems, which gave birth to the first generation of medical AI products, i.e. MYCIN. It attempts to diagnose infectious diseases and provide appropriate medication therapies based on over 200 rules that in the form of “if (precondition), then (conclusion or action)”. Although MYCIN showed reliability in 63% of the cases of bacteremia [5], it is more like an advanced version of a textbook which has the function of automatic retrievals with little “intelligence”. Moreover, the interactions between each rule should be defined by human experts which is extremely arduous with increasingly new knowledge added to the system. However, automatic interpretation of electrocardiograms has gained great benefits from the rule-based AI system. During the same period, a new model, i.e. the probabilistic reasoning was used to simulate the process of expert decision making [6]. Unlike the categorical mode of reasoning utilized by MYCIN, the probabilistic model assign weight to every symptom and clinical finding to indicate its possibility of occurrence for a certain disease. Based on this, new systems including the present illness program, INTERNIST, CASNET were developed [4]. With the introduction of “artificial neural network” in 1980s, machine learning began to flourish. Machine learning endows computers with the ability to learn patterns from data and perform tasks without explicit programming [7]. It has showed great power in developing robust risk prediction models, redefining patient classes and other tasks with the popularization of electronic health records and digital imaging systems [8]. In 2006, the introduction of deep learning by LeCun et al. [9] achieved unprecedented progress in image recognition and other domains, which brought a new renaissance to the development of AI and made automated medical imaging interpretation one of the hottest topics in AI. AI can generally be classified as either weak AI or strong AI. Strong AI is an intelligence construct that has the ability to understand and think as a human being. Weak AI is limited to a specific or narrow area, which is often designed to perform time-consuming tasks and analyze data in ways that humans sometimes cannot. In this review, a brief introduction to AI is provided and highlights its application in the clinical flow of diagnosing and treating valvular heart diseases, including heart sound auscultation, medical image analysis (echocardiography, cardiac computed tomography [CT] and cardiac magnetic resonance [CMR]), risk factor identification, mortality prediction and robotic surgery.

Key concepts in AI

Machine learning

In essence, machine learning is an extension to traditional statistical methods for dealing with large data sets and variables. It enables computers to learn rules and even uncover new patterns from data through a series of algorithms. Most importantly, machine learning can gradually optimize the “reasoning process” between inputs and outputs. Machine learning can be broadly classified into three classes: supervised learning, unsupervised learning and reinforcement learning.

In supervised learning, machines learn the mapping relations between input variables and labeled outcomes and are able to predict outcomes according to inputs. For example, after training with echocardiograms that are labeled with given disease categories, machines are able to assign these disease labels to new echocardiograms. Common computational approaches in supervised learning include artificial neural network (ANN), support vector machine, k-nearest neighbor, naive Bayesian model and decision tree model, among others. Unsupervised learning aims to uncover hidden structures in unlabeled data and classify it into separate categories, which means machines are only trained with input variables and automatically find potential classification rules. For example, with unsupervised learning, machines uncovered three cardiac phenotypes in patients diagnosed with type 2 diabetes mellitus according to their echocardiograms [10]. Common computational approaches include hierarchical clustering, k-means clustering and principle component analysis. Reinforcement learning signifies that machines learn strategies which can obtain the maximum reward through interaction with the environment or outcomes. Unlike the static mode of supervised or unsupervised learning, it is a dynamic learning process. Q-learning, an example of reinforcement learning, has been utilized in clinical trials of lung cancer to find the optimal individualized therapies [11].

Artificial neural network and deep learning

Artificial neural network is composed of multiple interconnected artificial neurons which mimic the biological brain. As mentioned above, ANN is actually a computational model. ANN learns from large amounts of data and continuously optimizes the weight of each connection between neurons. Figure 1B shows a typical ANN, which is also called multilayer perceptron, is composed of three layers: input layer, hidden layer and output layer. Each layer contains multiple neurons that are responsible for different functions. The input layer receives numerical variables from data, each value in first layer neurons combines with the weight of the connection, which is then propagated to a hidden layer where the values are integrated, while an output layer generates the final value which represents an outcome. Each neuron in the hidden layer can also set up a threshold to determine whether the information should propagate to the next layer. Now suppose that a database that includes patient data (such as age, body mass index, laboratory results and images results) prior to cardiac surgery and operation results (alive or dead), a mortality prediction model can be built with ANN. Patient variables are input into the neurons in the first layer, the outputs are set as 1 for alive and 0 for dead. The weight of each connection and the excitation threshold of each neuron in the hidden layer are gradually optimized by learning from the data. In this way, after trained with sufficient cases, the ANN model can be accurate enough to predict surgical outcome in new patients.

Figure 1.

Schematic diagram of the evolvement of artificial neural network. Perceptron, also called single layer neural network, was proposed by Rosenblatt in 1958 (A), classic artificial neural network, also known as double layer neural network or multilayer perceptron (B), deep neural network (C).

It can be imagined that, with more hidden layers, the computational results will be more accurate, but this meanwhile requires greater computational power using more training data. Thanks to the rapid development of computer processors and arrival of the era of big data, and also the introduction of “pre-training process”, “fine-tuning technique” by Hinton et al. [12] which decreases the training time substantially, multilayer ANN which is also known as “deep learning” becomes a reality. It now represents the most advanced AI technology. The biggest advantage of deep neural network is that it allows the computing model to automatically extract the features of the data by abstracting the data layer by layer and learning the representative and discriminant features.

Computer vision

Computer vision is a main research field in AI, which simulates the human vision with imaging systems and computers. The main tasks involved in computer vision can be divided into three levels [13]. The basic level includes image acquiring and processing techniques such as noise filtering, image enhancement, image segmentation, pattern recognition. For middle level computer vision, computers should be able to draw conclusions and make decisions based on the information obtained from the basic level. For the top level, computers have the capability to “think” and can understand images like humans do. For now, the first two levels of computer vision are mostly applied to the analysis of medical images and assist in diagnosis. As described below, many algorithms and commercial software have been developed for automating the analysis of medical images, which are more accurate and efficient compared to human endeavour.

Expert system

Expert systems are in their earliest application of AI in medicine, and will keep playing an important role in healthcare. In brief, expert systems are a computer program that simulates human experts in solving problems [14]. It is also called a knowledge-based system because it contains a large amount of knowledge and experience from experts. A typical medical expert system is composed of two modules: knowledge base and control system, the latter can be further divided into a human-computer interactive component, an explanation component, a knowledge acquisition component and an inference engine. The knowledge base is composed of medical knowledge from experts, case-specific knowledge from patients and intermediate results from the reasoning process [14].

Application of AI and automation in diagnosis of valvular heart diseases

AI-assisted “cardiac auscultation”

In spite of the diverse alternatives of imaging tools being the major diagnostic approach to valvular heart diseases (VHDs), physical examination, is the cornerstone of clinical diagnosis and should be the primary screening method for VHDs [15]. Common physical examinations for VHDs include general inspection, pulse palpation, percussion of heart boarder and heart sound auscultation, etc. [16]. Among these, auscultation plays a key role of diagnosing VHDs.

Valvular heart diseases manifest as heart murmurs or/and extra heart sound before the stress of hemodynamic changes causing other signs and symptoms including dyspnea, fatigue, angina, cough and hemoptysis. While cardiac auscultation provides significant diagnostic and prognostic information for cardiac disease, it is not an easy skill to master, especially when innocent murmurs are confronted. Although experienced clinicians have reached an accuracy over 90% in identifying innocent murmurs from pathologic murmurs, less-experienced residents and primary care physicians perform less than satisfactorily [17]. In addition, auscultation is a highly subjective process which may cause bias when evaluating the intensity, location and shape of murmurs. For over a century, tools for cardiac auscultation is a mechanical stethoscope, however, it can neither store nor play back sounds [15]. Thanks to the invention of electronic (digital) stethoscope, the forgoing problems have been well solved, more than that, it provides clinicians with a handy way of “seeing” the heart sound through phonocardiogram. AI-assisted cardiac auscultation in practice refers to the auto-interpretation of phonocardiogram, which belongs to the domain of signal processing. Key steps involved in heart sound analysis could be summarized as segmentation, feature extraction and classification as shown in Figure 2 [18]. Each step is fulfilled through a multitude of algorithms with the ultimate goal of precisely identifying the pathological events underlying heart sounds.

Figure 2.

Process of artificial intelligence-assisted auscultation. A normal phonocardiogram (A), heart sound segmentation (B), feature extraction (C), heart sound classification by deep neural network (D).

Heart sounds segmentation

Segmentation aims to locate the fundamental elements including the first heart sound (S1), systolic period, the second heart sound (S2) and diastolic period in each cardiac cycle. Training machines to think like human so as to solve problems is to mimic the thinking process of human brain to a certain extent. When interpreting heart sounds, A human expert would firstly locate the two fundamental heart sounds then discriminate S1 from S2 by its pitch, intensity and duration, finally the systolic and diastolic regions are determined, as is the process with computational analysis, but in a more logical and mathematical way.

In the early exploration of heart sound segmentation, electrocardiograph or/and carotid pulse was/were obtained concurrently with phonocardiogram [19, 20]. Transformation equation g(n) was used to compute the smooth energy curve of electrocardiograph signal, then the peaks in g(n) was determined as the beginning of S1 and the systolic period, the dicrotic notch of carotid pulse curve was recognized as the beginning of S2 [19]. This approach apparently is impractical in routine clinical work. Envelop-based (or amplitude threshold-based) segmentation algorithm was first introduced by Liang et al. [21] to locate S1 and S2. Envelop of a signal signifies the smooth curves outlining its upper or lower extremes [22]. The envelop of the original heart sound waveform was first extracted using specific transformation equations, then a threshold value was selected to filter noise and low intensity signals, the peaks exceed the threshold value were recognized as the S1 and S2, S1 was differentiated form S2 based on an assumption that the systolic period was shorter than the diastolic period [21]. A well-fitted extraction equation would allow only two peaks to exceed the threshold when interpreting the normal heart sound, therefore identification accuracy depends largely on an envelope extraction equation. Diverse methods are used to calculate the envelope were created including normalized average Shan-non energy [21], single-DOF analytical model [23], wavelet decomposition method [24], Viola integral approach [25], and others. However, the envelope-based segmentation method often leads to problems like weak peaks or extra peaks which reduce accuracy of the results.

The application of machine learning algorithms to heart sound segmentation based on features substantially improves the segmentation results (see feature extraction below). Hidden Markov Model (HMM) is the most widely used in the task [26–28]. To train the HMM, adequate samples pre-labeled with the accurate location of S1, S2, systolic and diastolic period are required at the output end; sequences of feature vectors extracted from original phonocardiogram or transformed envelope are used as the observation end of HMM. Recently, utilization of deep learning achieves even higher precision compared to other classification methods in heart sound segmentation [29, 30].

Feature extraction and feature selection

This step aims to select and extract discriminative features either for more precise segmentation of heart sound or for following the disease classification step. Feature means the characteristic of an object by which the human brain recognizes and distinguishes it automatically. The concept of feature is similar to “variable” in regression analysis. Features that can be recognized by machines tend to be presented in the form of numbers or symbols. A human expert draws physiological or pathological information from heart sound through features including heart rate, heart rhythm, timing and shape of the murmur, pitch of the heart sound, and extra heart sound among others. As for phonocardiogram, the features are based on time-domain, frequency-domain, and time-frequency complex domain [31]. Time-domain features include intervals (interval of S1 and S2, systolic intervals, diastolic intervals, the ratio of each component to the cardiac cycle etc.) and amplitude (mean absolute amplitude of S1 and S2 interval). Frequency-domain features refer to the power spectrum of each heart sound component across frequency bands [29]. Theoretically the more input of features into machine training, the better the classification performance it will achieve. In practice, with the number of training samples being set, the classification performance will drop off when the number of feature inputs exceeds a certain value. In order to exclude the redundant features and improve the classification efficiency, feature selection is often required [32].

Classification and detection of VHDs

This step aims to classify the phonocardiograms into cardiac disease categories using suitable classifiers. Classifier in machine learning refers to the algorithm that learns to assign labels to testing data from training data. This learning process can be supervised or unsupervised. Briefly speaking, supervised learning assigns given labels to data while unsupervised learning seeks labels that could be assigned to data. In the present case, labels are given disease categories like mitral stenosis, mitral regurgitation, aortic stenosis, etc. therefore, supervised learning algorithms/classifiers are utilized. The common classifiers used in heart sound classification include a support vector machine [24, 33–36], neural networks [37, 38], HMM [39], etc. After a careful combination of algorithms used in segmentation, feature extraction and classification, previous studies showed promising and inspiring results in detecting VHDs from phonocardiograms (Table 1). For instance, an intelligent diagnostic system developed by Sun [24] could discriminate aortic regurgitation, mitral regurgitation, pulmonary stenosis with accuracy of 98.9%, 98.4% and 98.7%, respectively. Thompson et al. [17] utilized a murmur detection algorithm developed by CSD labs to distinguish no murmurs and innocent murmurs from pathologic murmurs. 3180 phonocardiograms recorded at five different chest location from 603 cases were tested, the algorithm had good sensitivity, specificity and accuracy in detecting a pathologic murmur, which are 93%, 81% and 88%, respectively. The algorithm showed the highest accuracy (90%) using recordings from the left upper and lower sternal boarder. However, it was unable to analyze the recordings with precision which had a high heart rate and low signal quality [17].

Table 1.

Sensitivity/specificity or accuracy of artificial intelligence-assisted auscultation system in detecting valvular heart diseases.

Reference	Year	Algorithm	Testing samples	Normal (%)	AS (%)	AR (%)	AS+AR (%)	MS (%)	MR (%)	MS+MR (%)	PS (%)	PR (%)
Fahad et al. [39]	2018	HMM	80 abnormal HSs and 150 normal HSs	100/99.33		92/100		96/98.54	93.33/99.50
Sun [24]	2015	SVM	588 abnormal HSs from 49 patients, 325 normal HSs from healthy individuals	99.8		98.8			98.4		98.7
Zhang et al. [33]	2014	SVM	40 abnormal HSs and 75 normal HSs	98.67	85			95
Safara et al. [34]	2013	SVM	43 abnormal HSs and 16 normal HSs	98/100	97.44/98.40	100/100			95.65/98.31
Kwak et al. [35]	2012	HMM	53 abnormal HSs, 80 normal HSs	99	67	67	67	83	67	80
Uğuz [37]	2012	ANN	40 abnormal HSs, 20 normal HSs	95				95			95
Kumar et al. [36]	2010	SVM	81 abnormal HSs from 51 patients		93.02/91.29	94.78/93.98			95.20/96.33		100/100	93.21/95.80
Olmez et al. [38]	2003	ANN	240 abnormal HSs from 20 patients, 48 normal HSs from 4 healthy individuals	100	97.9	100		100	100		97.9

If more than one testing methods were utilized in the study, only the method with highest sensitivity/specificity or accuracy is presented; ANN — artificial neural network; AR — aortic regurgitation; AS — aortic stenosis; HMM — hidden Markov model; HS — heart sound; MR — mitral regurgitation; MS — mitral stenosis; PR — pulmonary regurgitation; PS — pulmonary stenosis; SVM — support vector machine

AI-assisted interpretation of echocardiography

Although heart sounds auscultation is convenient, cost-effective and a quick way to diagnose VHDs, it only provides qualitative diagnostic information. To further confirm the diagnosis of VHDs as well as to assess the etiology, severity, ventricular responses and prognosis, imaging tools including echocardiography, CMR, multi-slice CT (MSCT) or even cardiac catheterization are indispensable [40].

Echocardiography is the “heart” of cardiology. It is the preferred method of diagnosing and guiding the treatment of VHDs as well as other cardiac diseases. However, echocardiographic examination is a time-consuming process which usually takes hours from inspection, analysis to formal report which make it impractical in emergency settings. Moreover, typical echocardiographic examination produces large amounts of data including images from multiple views and up to 70 videos which would cause cognitive overload and be prone to error. In addition, due to the characteristic of multiple views in examination, the problems of intra-observer and inter-observer variability inevitably arise. The application of AI and automation in echocardiography would a allow consistent, quick and accurate measurement which promises a more accurate diagnosis and improved patient care.

Image segmentation-valve leaflet detection and tracking

Segmentation and recognition of anatomical structures from original medical images are preprocessing steps for subsequent quantitative analysis and diagnosis. Morphological characteristics and motion pattern of the heart valves are key information to diagnose VHDs and to assist surgical valve repair or percutaneous intervention. While many studies have sought the approaches to segment heart chambers and detect endocardial boarder which were reviewed by Noble and Boukerroui [41], relatively few studies of the segmentation of heart valves and annulus have been published. The reasons for difficulty in segmenting and tracking the heart valves in echocardiographic sequences can be summarized as follows: (1) poor image quality due to low frame rate, speckle noise and artifact which may result in missing boundaries; (2) lack of features to discriminate heart valve from adjacent myocardium for those structures have similar intensity and texture; (3) fast and irregular valve motion making it hard to establish correspondence between frames. Reported approaches for valve segmentation including active contour models [42, 43], thin tissue detector combined with graph cut [44], outlier detection method [45, 46], J-spline method [47], multi-atlas joint label fusion [48], trajectory spectrum learning algorithm [49] and neural network [50].

Active contour models (ACM) is the first and most widely used segmentation algorithm for the heart valve [51, 52]. The segmentation process is based on ACM and is initiated by manual placement of contour close to the target, which means it is not fully automated. Zhou et al. [45] formulated the problems of valve detection and tracking as outliner detection in the low-rank representation based on different motion patterns between leaflets and the heart wall, which is fully automated requiring no user interaction [46]. Recently, there is a growing trend of applying machine learning to valve segmentation. This approach shifts the manual input to a training phase which makes segmentation fully automated [50]. Ionasec et al. [49] utilized spectrum learning algorithm and boundary detectors to locate and refine mitral leaflets based on anatomical landmarks in four-dimensional (4D) transesophageal echocardiograms and CT which allows precise morphological and functional quantification, and a large amount of data as well as high computational power were required to train the model. Recently, a novel UNet architecture based on a convolutional neural network was utilized by Costa et al. [50] to segment mitral valves in PLAX and A4C views. The biggest strength of UNet architecture is that it does not require a large training dataset to produce accurate results.

Automated quantitative analysis

Precise and reliable quantitative analysis of stenosis and regurgitation is crucial to severity assessment, prognosis prediction and to evaluate whether surgery or intervention is needed. Proximal isovelocity surface area (PISA) method is widely used in measuring mitral valve orifice area and calculating the regurgitant volume [53]. Conventional PISA method of two-dimensional echocardiography is based on the assumption that the shape of the proximal flow convergence region is hemispheric, which is not quite true [54]. The development of three-dimensional (3D) color Doppler echocardiography enables direct measurement of PISA without assumptions, thereby reducing errors [54]. However, it is not indicated for routine clinical use because it is a time-consuming procedure, thus in desperate need of an automated algorithm. Grady et al. [55] proposed the first automated PISA measurement system based on a random walker algorithm. The system was initiated by the manual input of two points (one at the valve annulus and one at the coaptation site), followed by automated segmentation of valve annulus and the isovelocity region. The segmentation results were then used to generate 3D meshes for computing PISA. In vitro experiments validated its accuracy in measuring PISA, effective regurgitant surface area (EROA) and regurgitant volume. Further in vivo experiments showed that measurement of EROA on patients with magnetic resonance (MR) by automated algorithm was significantly correlated with manual measurement of vena contracta [55]. Cobey et al. [56] proposed another novel automated method based on Halcon HDevelop Machine Vision. 3D images of PISA were first sliced into 2 mm-thick sequential cuts, the algorithm traced the boundary of each slice and the arc lengths between each slice from which the 3D surface area was generated and computed. Another method of quantitatively assessing MR is to use regurgitant fraction which is calculated from mitral inflow and stroke volume [53]. Wang et al. [57] proposed a novel automated system for estimating mitral inflow and aortic outflow. The system first detected the left ventricular wall, mitral annulus and left ventricular outflow tract using marginal space learning and placed the measurement plane. The 3D motions of these structures were tracked through a whole cardiac circle to construct and adjust the measurement plane, then the volume of mitral inflow and left ventricular outflow was computed by aggregating color flow values in the 3D space [57].

In addition to these pioneering studies, several commercial software have been developed including Mitral Valve Quantification (Philips Medical Imaging, Andover, MA) [58], eSie Valve (Siemens Healthcare, Mountain View, CA, USA) [59], Mitral Valve Navigator (Philips Medical Systems) [60], Auto Valve Analysis (Siemens; California, USA) [61], eSie PISA Volume Analysis (Siemens Medical Solutions USA, Inc., Mountain View, CA) [62] which aim to automate the quantitative analysis of 3D echocardiography. These approaches can reduce the measurement time substantially and provide more accurate and reproducible results. The aforementioned automated algorithms and commercial software have validated their reliability in quantitatively measuring aortic and mitral valve apparatus parameters and regurgitation volume [54, 63, 64] as shown in Table 2.

Table 2.

Automated software and algorithms in quantification analysis of three-dimensional echocardiography.

Reference	Year	Software/algorithm	Function	Reference method	Samples	ICC/intergroup difference between software/ algorithm and reference method)	Intra-observer variability (ICC)	Inter-observer variability (ICC)
Mediratta et al. [58]	2017	Mitral Valve Quantification (QLAB version 9.0.1; Philips Medical Imaging, Andover, MA; USA)	For measurement of aortic annulus area, perimeter and diameter	Computed tomography	52 patients	ICC of annulus area, perimeter, area-derived diameter, perimeter-derived diameter: 0.91, 0.90, 0.91, 0.89	0.83, 0.85, 0.81, 0.85, respectively	0.70, 0.72, 0.72, 0.71, respectively
Aquila et al. [59]	2016	eSie Valve Software (Siemens Healthcare, Mountain View, CA, USA)	For measurement of mitral valve parameters	Manual measurement	61 patients	ICC of inter-commissure diameter, mitral annular area, anterior leaflet length, posterior leaflet length: 0.84, 0.94, 0.83, 0.67	0.999, 0.992, 0.984, 0.937, respectively	0.997, 0.957, 0. 963, 0.936, respectively
Kagiyama et al. [60]	2015	Mitral Valve Navigator (Philips Medical Systems)	For measurement of mitral valve parameters	Mitral Valve Quantification (Philips Medical Systems)	74 patients	Cronbach’s alpha of commissural width, anteroposterior diameter, annulus area: 0.902, 0.934, 0.938	Not available	0.945, 0.959, 0.961, respectively
Calleja et al. [61]	2013	Auto Valve Analysis (Siemens; California, USA)	For measurement of aortic annulus diameter, leaflet anatomy and coronary ostium height	Computed tomography	40 patients	Intergroup difference (p value) of left, non-coronary, right leaflet: 0.76, 0.06, 0.29; left and right coronary height: 0.003, 0.001	Annulus diameters at sagittal and coronal view: 0.97, 0.92	Annulus diameters at sagittal and coronal view: 0.91, 0.93
Agustin et al. [62]	2013	eSie PISA Volume Analysis (Siemens Medical Solutions USA, Inc., Mountain View, CA)	For measurement of EROA and RVol	Manual measurement by 3D planimetry	90 patients	ICC of EROA, RVol: 0.97, 0.97	Not available	Not available
Choi et al. [63]	2015	Based on Wang [71]	For measurement of aortic RVol	CMR	32 patients	ICC of aortic RVol: 0.93, AR severity: k = 0.94	0.98	0.96
Thavendiranathan et al. [64]	2013	Based on Grady [69]	For measurement of mitral RVol	CMR	35 patients	ICC of Integrated PISA-derived RVol, flow volume-derived RVol: 0.92, 0.91	0.99, 0.93, respectively	0.95, 0.96, respectively

Not all results are listed in the table; 3D — three-dimensional; AR — aortic regurgitation; CMR — cardiac magnetic resonance; EROA — effective regurgitant orifice area; ICC — interclass correlation coefficient; PISA — proximal isovelocity surface area; RVol — regurgitant volume

AI-assisted interpretation of cardiac CT images

Computed tomography is not employed as the preferred diagnostic method of VHD. It generally serves as a complementary role when echocardiography is insufficient or inclusive. Nevertheless, cardiac CT has prominent advantages in the evaluation of valve calcification and annulus geometry. It is the golden standard for annulus sizing and an indispensable tool in preoperative planning of transcatheter aortic valve replacement (TAVR). Reliable measurement of aortic valve annulus size, aortic root dimension and the height of coronary ostia are crucial for appropriate transcatheter valve prosthesis selection and intraprocedural valve positioning which may improve success rate and reduce postoperative complications. In recent years, advances in 3D image techniques have enabled reconstruction of patient-specific models of aortic valve apparatus. Several automated algorithms and commercial software for aortic valve segmentation and quantification have been developed for preoperative planning of transcathether aortic valve implantation (TAVI).

Patient-specific 3D modeling of aortic valve

Ionasec et al. [49, 65] proposed the first dynamic patient-specific aortic valve model from 4D CT images. A generic physiological model which can represent aortic valve and its pathological variations was constructed first, patient-specific parameters of the model were estimated from volumetric sequences by trajectory spectrum learning, marginal space learning and discriminative learning. Their study was further improved with shape forest to constrain the classical statistical shape model [66]. Waechter et al. [67] extracted the aortic valve geometry from CT images using model-based segmentation and applies pattern search method to detect the coronary ostia. The whole heart geometry (including heart chambers and great vessels) was first roughly modeled, a generic aortic mesh model was established and boundary detector were trained by annotated images aiming to build more detailed models of the aortic valve and root, the coronary ostia was then detected on the surface of the aortic root. Segmentation results allowed a series of clinical measurements including the diameters of annulus and the distance between ostia and aortic valve, etc. Considering that VHDs often involve multivalvular lesion which require joint assessment, Grbic et al. [68] proposed an integrated model for quantification of all heart valves from 4D CT images based on marginal space learning and multi-linear shape models. Another study of Grbic et al. [69] extracted both the volumetric model of the aortic valve and calcification, which is closely correlated with postoperative regurgitation [70].

Landmarks detection and quantitative measurement

A number of parameters should be measured with precision for surgical planning before TAVI. These include annulus diameter, annulus area, angulation of annulus plane and the distance from annulus to coronary ostia to name a few. The annulus diameter is crucial for selection of the appropriate valve prosthesis. Size mismatching may either cause post-operative perivalvular regurgitation or annulus rupture. A short distance between aortic annulus and coronary ostia indicates elevated risk of coronary obstruction after valve deployment. Annulus plane angulation determines the position of an X-ray tube C-arm during operations to achieve an optimal view of valve delivery [71].

Accurate segmentation of the aorta, aortic valve apparatus and detection of anatomic landmarks (coronary ostium, aortic commissures and aortic hinges) is the prerequisite for reliable measurement of the above parameters. Zheng et al. [72] proposed a robust hierarchical approach by first segmenting the global aorta using marginal space learning from which the position of the anatomical landmarks can be roughly inferred, followed by using specific landmark detectors to refine each landmark. Elattar et al. [73, 74] used thresholding and connected component analysis to detect the region of interest, from which aortic root was extracted using 3D normalized cut. Two coronary ostia and three valve hinge points were then detected on the surface of aortic root by intensity projection map and Gaussian curvature map. Lalys et al. [75] used a hybrid approach which integrated thresholding, model-based method, statistical-based method and a 3D contour model in the procedure, however, a user-specific point is needed to define the volume of interest. Recently, colonial walk algorithm, which is a machine learning method, was utilized by Al et al. [76] to automatically localize the landmarks. Several commercial software including three mensio valves (3mensio Medical Imaging BV, the Netherlands) [77, 78, 81], Syngo (Siemens Healthcare, Erlangen, Germany) [79], IntelliSpace Portal (Philips Medical Systems, Cleveland, OH) [80], have been available for routine clinical use. Together with the aforementioned algorithms, these pioneering techniques showed reliable aortic annulus measurements as shown in Table 3.

Table 3.

Automated software and algorithms in quantification analysis of aortic valve apparatus.

Reference	Year	Software/algorithm	Reference method	Samples	Correlation/bias	Inter-observer variability	Intra-observer variability
Dankerl et al. [79]	2017	Syngo, CT TAVI Planning (version VB10A, Siemens Healthcare, Erlangen, Germany)	Manual measurement (software: NA)	30 patients	Distance from annular plane to coronary ostia: r = 0.97 (right); r = 0.93 (left) Annulus area: r = 0.89 Annulus diameter: r = 0.89 (maximum diameter); r = 0.82 (minimum diameter)	NA	NA
Guez et al. [80]	2017	Intellispace Portal (Philips Medical Systems, Cleveland, OH)	Manual measurement using 3D Brilliance workstation (Philips Healthcare)	74 patients	Annular diameter in systole: r = 0.85 (minimum), r = 0.87 (maximum diameter) Annular diameter in diastole: r = 0.89 (for both minimum and maximum diameter) Annular area: r = 0.93 (systole), r = 0.94 (systole)	NA	NA
Watanabe et al. [77]	2013	3mensio Valves (version 5.1, 3mensio Medical Imaging BV, the Netherlands)	Manual measurement using dedicated workstation (EBW, Philips Healthcare)	105 patients	Average bias: mean annulus diameter: 1.15 mm, left coronary ostium height: 1.61 mm, right coronary ostium height: 3.45 mm	ICC: 0.93 to 0.98	ICC: 0.94 to 0.98
Delgado et al. [78]	2011	3mensio Valves (version 4.1.sp1, Medical Imaging BV, Bilthoven, The Netherlands)	Manual measurement using Vitrea2 software (Vital Images, Minneapolis, MN)	90 patients	ICC of annulus diameter, LVOT diameter, sinus of Valsalva height, coronary ostium height: 0.97 to 0.99	ICC: 0.90 to 0.99	ICC: 0.95 to 0.99
Lalys et al. [75]	2019	Hybrid approach: model-based method, active contour model, atlas-based registration, learning-based detection	Manual measurement (software: NA)	50 patients	Detection bias of aortic leaflets 1.96 ± 0.84 mm, 2.34 ± 0.78 mm and 1.23 ± 1.12 mm (right, left and non-coronary ostia) Measurement bias: annulus diameter: 1.31 ± 0.75 mm, coronary ostia heights 1.96 ± 0.87 mm (right), 1.80 ± 0.74 mm (left)	NA	NA
Al et al. [76]	2018	Colonial walk	Manual measurement (software: NA)	71 patients	Mean detection bias of hinge points, commissure points and coronary ostia: 2.09 ± 1.20 mm Measurement bias: aortic annulus diameter: 0.94 ± 0.83 mm, annulus to ostia distance: 0.98 ± 0.90 mm	NA	NA
Elattar et al. [73]	2016	3D normalized cut and intensity map	Manual measurement (software: NA)	40 CTA images	landmark segmentation bias: 2.38 ± 1.56 mm Measurement: ICC of annulus radius: 0.84, annulus to ostia distance: 0.73	NA	NA
Elattar et al. [74]	2014	3D normalized cut method	Manual delineation	20 patients	Dice coefficient of aortic root segmentation: 0.95 ± 0.03	Dice coefficient: 0.951 ± 0.03	NA
Zheng et al. [72]	2012	Marginal space learning and optimization-cased method	Manual measurement (software: NA)	319 C-arm CT volume images	Segmentation bias: aortic hinges 2.09 ± 1.18 mm, coronary ostia 2.07 ± 1.53 mm, aortic commissure: 2.17 ± 1.88 mm Measurement bias: distance from annulus plane to coronary ostia 2.10 ± 1.86 mm (left), 2.51 ± 2.03 mm (right)	NA	NA

Not all results are listed in the table; 3D — three-dimensional; CTA — cardiac magnetic resonance; CT — computed tomography; ICC — interclass correlation coefficient; LVOT — left ventricular outflow tract; NA — not available

A comparative study between 3mensio, Intelli-Space Portal (version 7.0), IntelliSpace Portal (version 9.0) and manual measurement concerning annulus parameters, time-cost and reproducibility was performed [81]. Results showed that IntelliSpace Portal (version 9.0) allowed the fastest and most reproducible measurements, parameters derived from either method can be used interchangeably in prosthesis sizing. Moreover, Samim et al. [71] performed a prospective cohort study to compare angiography and MSCT (using 3mensio) in predicting the annulus plane and choosing the C-arm position [71]. The study included 35 patients in an angiography cohort and 36 patients in an MDCT cohort. The utilization of MDCT was associated with a significant reduction of prosthesis implanting time, radiation exposure and contrast delivery. A reduction of postoperative complications and 30-day mortality was observed in patients in the MDCT cohort.

AI-assisted interpretation of CMR

Cardiac magnetic resonance is the second-line technique in the assessment of VHDs. A standard CMR scan takes about an hour followed by tedious image processing which is both time and labor consuming. However, it is the golden standard in the assessment of cardiac morphology, volume and function due to its high spatial resolution [82]. MR cine imaging and phase-contrast velocity imaging are gaining increasing importance in studying valvular function. Particularly, CMR have advantages over other imaging tools in the quantification of regurgitation when the regurgitant jet is highly eccentric.

Automated cardiac chamber segmentation and cardiac volume measurement have been widely studied in MR images as reviewed by Bernard et al. [83] and Petitjean et al. [84]. Several large CMR image datasets (Sunnybrook, STACOM, MICCAI RV and kaggle), which are free to access, have been released in the last decade in conjunction with international challenges to automatically segment left ventricle, right ventricle, end-diastolic volume and end-systolic volume [83]. Commercial software such as SuiteHEART^® (Neosoft, Pewaukee, Wisconsin, USA) which allows automated quantification of biventricular volumes and function is now available [85]. The regurgitant volume and fraction in isolated aortic or mitral regurgitation can be calculated from left ventricle and right ventricle stroke volume [82], however, to date, the aforementioned automated methods have not applied in quantification of valve regurgitation. AI-assisted CMR which allows automated diagnosis of VHDs is underexplored. However, several pioneering studies have achieved many valuable results. Fries et al. [86] developed a novel supervised deep learning model for aortic valve malformation classification using unlabeled MR images. 570 patients were classified as bicuspid aortic valve from a cohort of 9230 patients from the UK biobank. These individuals showed a significant lower major adverse cardiac event-free survival rate compared to individuals with a normal aortic valve.

Risk factor identification and in-hospital mortality prediction of cardiac surgery

Preoperative assessment of surgical risk is an important procedure in cardiac surgery which guides the selection of surgery, intervention or nonsurgical treatment. Great progress has been made in risk factor identification and mortality prediction. Risk score models including EuroSCORE, STS score and ACEF scores have been widely used [87]. The linear regression model is the most widely used in analyzing a correlation between risk variables and cardiac adverse events. However, there is a growing trend of utilizing machine learning methods in developing prediction models which outperform traditional scoring systems.

Nilsson et al. [88] were among the first to use an ANN model to identify risk factors and predict mortality in cardiac surgery. 72 risk factors were evaluated from 18,362 patients, 34 of the factors were identified as relevant to mortality. Receiver operating characteristic (ROC) area of the ANN model for mortality prediction was significantly larger than logistic EuroSCORE model (0.81 vs. 0.79). For isolated valve surgery (with or without coronary artery bypass grafting), the ROC area of the ANN model was 0.76 vs. 0.72 of the logistic models. Celi et al. [89] achieved an even larger ROC area using ANN, Bayesian network and logistic regression (0.941, 0.931, 0.854) compared to EuroSCORE (0.648). Allyn et al. [90] utilized a novel ensemble machine learning method which integrates the results from four isolated machine learning algorithms to predict cardiac surgery mortality. 6250 patients were enrolled in the study, Chi-square filtering was utilized to extract relevant variables. Results showed that the ensemble machine learning model had a significantly stronger discriminatory power for operative mortality than EuroSCORE II (ROC area: 0.795 vs. 0.737). Recently, Hernandez-Suarez firstly applied four machine learning algorithms in mortality prediction after TAVI [91]. 10,883 patients were enrolled in the study. Logistic regression, ANN, Naïve bayes and random forest which are the four top supervised learning algorithms all showed good discriminative performance. The best prediction model obtained by logistic algorithm (ROC area: 0.92) have close discriminative power with state-of-the-art National Inpatient Sample TAVR score model.

Moreover, ANN and machine learning algorithm have been applied in other fields such as prediction of length of stay in intensive care unit after cardiac surgery [92], post-operative complications [93], both short-term and long-term mortality after heart transplantation [94]. However, the downside of current studies is that few studies have been dedicated to isolated cardiac surgery procedures which is partially due to a lack of sufficient samples to feed machine training.

Intelligent cardiac operating room: Towards autonomous robotic surgery

The introduction of minimally invasive strategies and a surgical robot in cardiac surgery promises quicker recovery and less postoperative complications and mortality. While having many superiorities over human hands such as tremor resistance and scalable motion, current robotic systems are merely teleoperated devices under human control which entirely possess no autonomy. Since automation has gained great success in other robotic fields which have increased safety, accuracy and efficiency, it is reasonable to assume that the same benefit will be gained by developing autonomous surgical robots. To achieve the level of fully “autonomous”, the surgical robot should possess the ability to “see”, “think” and “act”. “see” refers to perception of the surgical field and itself through sensors. “think” is the process of receiving information and calculating the future status that it needs to achieve in the following “act” [95]. Recently, European research council launch the Autonomous Robotic Surgery (ARS) project aiming to developing a unified framework for the autonomous execution of robotic tasks [96]. Main research objectives include establishment of global action model by analyzing current robotic surgical data, patient specific intervention models, design of controllers, perception of overall surgical situations, and assessment of the surgical robot capability.

Though at an experimental stage, the feasibility of autonomous robots performing simple surgical tasks like suture and knot typing have been demonstrated. Penesar et al. [97] developed a novel Smart Tissue Autonomous Robot (STAR) which was able to perform linear continuous suture, it successfully completed in-vivo end-to-end anastomosis of porcine small intestine with few suturing mistakes, no complications were observed in 7 day follow-up. Although the STAR system realized autonomous suture, the step of knot typing remained manual. In the context of endoscopic surgery or robot-assisted surgery, the task of knot typing is cumbersome which requires manipulation of many subtle movements in a confined space [98]. Through training a recurrent neural network on 3D loop trajectories generated by human surgeon, the Endoscopic Partial-Autonomous Robot (EndoPAR) was able to accurately perform the winding portion of a knot-tying task [99]. To date however, robots are unable to fully perform autonomous knot tying.

Transcatheter therapy has been accepted as an alternative to traditional cardiac valvular surgery especially in patients with high surgical risk in recent years. Catheters are inserted either from peripheral vessel or the cardiac apex to deploy valve prothesis or occlusion device. In either case, the catheters need to be precisely navigated to the intervention site which is a very challenging task in a beating heart. Inspired by wall following, which is used by thigmotactic animals to locate and navigate themselves in low-visibility environments, Fagogenis et al. [100] developed an autonomous intracardiac catheter navigation system. With hybrid imaging and touch sensor installed on the catheter tip, which can provide clear images of what it has touched and identify it as blood, valve or myocardium. The catheter created continuous low-force contact with the surrounding tissue and followed the cardiac wall to achieve autonomous navigation. With the navigation system, authors designed a robotic catheter, inserted from the cardiac apex, which can autonomously navigate to the aortic valve and deploy an occlusion device into the leak site on a porcine model. The in-vivo animal experiments demonstrate that an autonomous robot catheter was non-inferior to human experts [100].

Conclusions

Artificial intelligence is changing the landscape of healthcare. The inclusion of AI and automated algorithms in medical image analysis are very promising for they require less measurement time and meanwhile provide more accurate and reproducible results, which make them ideal helpers in a busy clinical flow. Machine learning methods can make full use of patient data and build more powerful prediction models of cardiac surgery compared to traditional statistical approaches. Moreover, the autonomous surgical robot, although in its infancy, holds great promise for improved safety and efficiency in cardiac surgery and intervention. In general, AI has enhanced the ability of diagnosis and clinical decision-making in VHDs and taken another step towards precision in medicine.