Application of Neural Networks to 12-Lead Electrocardiography　― Current Status and Future Directions ―

Abstract

The 12-lead electrocardiogram (ECG) is a fast, non-invasive, powerful tool to diagnose or to evaluate the risk of various cardiac diseases. The vast majority of arrhythmias are diagnosed solely on 12-lead ECG. Initial detection of myocardial ischemia such as myocardial infarction (MI), acute coronary syndrome (ACS) and effort angina is also dependent upon 12-lead ECG. ECG reflects the electrophysiological state of the heart through body mass, and thus contains important information on the electricity-dependent function of the human heart. Indeed, 12-lead ECG data are complex. Therefore, the clinical interpretation of 12-lead ECG requires intense training, but still is prone to interobserver variability. Even with rich clinically relevant data, non-trained physicians cannot efficiently use this powerful tool. Furthermore, recent studies have shown that 12-lead ECG may contain information that is not recognized even by well-trained experts but which can be extracted by computer. Artificial intelligence (AI) based on neural networks (NN) has emerged as a strong tool to extract valuable information from ECG for clinical decision making. This article reviews the current status of the application of NN-based AI to the interpretation of 12-lead ECG and also discusses the current problems and future directions.

The 12-lead electrocardiogram (ECG) is a non-invasive and easy to conduct, but potentially powerful tool for diagnosing cardiac disease or evaluating the risk of future cardiac events. Surface lead ECG was first developed by Willem Einthoven in 1901.¹ It provides important diagnostic and treatment indications for conditions such as arrhythmia,² cardiac hypertrophy,³ and myocardial ischemia,⁴ and for those that require immediate attention, such as acute coronary syndrome (ACS).⁵

While ECG contains rich information regarding cardiac disorders such as myocardial ischemia, interpretation of the results is not easy. Intensive training for physicians is necessary and, even with this, there is still huge interobserver variability.

Artificial intelligence (AI) has emerged as a powerful tool to automatically interpret medical data. It has been applied to medical imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI),⁶ and echocardiography,⁷ as well as to ECG. AI technology has been successful in multiple tasks of automatic interpretation of ECG. These studies have further shown that ECG may contain more profound pathophysiological information that is not understood even by well-trained cardiologists.⁸ This article will review the current state of the application of neural network (NN) AI to the interpretation of 12-lead ECG, and discuss the current problems and future directions of this technology in the current medical field.

AI in the Medical Field

AI is a broad term representing the replication of human “intelligence” with machines that include but are not limited to computers. In the current medical field, the term “AI” mainly refers to a complex machine-learning model that automatically extracts information from data and which is, in most cases, based on an NN. Therefore, in this article, the term “AI” will be used to refer to such statistical models and the methods to create them.

The use of AI is in rapid development in many areas such as image recognition,⁹ language recognition¹⁰ and automatic driving.¹¹ In particular, in the image recognition field, AI has achieved results exceeding human ability in speed and accuracy. The technology has received attention from the medical field because it may extend the utilization of complex medical data beyond the limit of human brain function that is unable to handle high-dimension data.^12,13 As expected, the application of AI to medical still-images has been greatly successful,^7,14,15 but AI has also produced striking results in other tasks such as electronic digitalized health record analysis,^16,17 and prediction of clinical outcome from ECG.^1,18–33 Of note, AI applied to the interpretation of 12-lead ECG has shown a potential beyond the ability of even a well-trained cardiologist, and will be the main focus of this article.

Ability and Limitations of NN

Before going into detailed specific discussion of NN application to 12-lead ECG, this section will explain the general strength and limitation of current NN model. NN is a network constructed by multiple units of statistical model called “neuron” that simulates the function of neuron cells (Figure 1). A single unit of neuron takes multiple inputs and calculate an output by multiplying the input with its internal parameters called “weights” and summing them up. In most cases, the output of the unit will go through a non-linear activation function and passed on to the next neuron unit. NN constructed from these units are capable of replicating a complex non-linear function if the network layers are deep enough. What makes this structure called an AI is the fact that these networks can be trained with pairs of multi-dimensional input and mono/multi-dimensional outputs to produce a model that allows calculation/prediction of the output from the input without prior knowledge of the underlining mechanism or important features. This means that NN can automatically extract important features and automatically create classifiers or predictions models only from data. This characteristics of NN may allow tasks that could not be done by human. Human brains can hardly deal with data with more than 3-dimensions while NN can take an arbitrary dimensional input, which means that it can process extremely complex input and automatically produce meaningful models. Medical data contain large numbers of multiple-dimensional data and is a field of interest on AI application.

Figure 1.

Schematic illustration of neuron units used in neural network models. The neuron unit simulates the function of a biological neuron by multiplying the input by weights and outputting the result through the activation layer.

However, the strength of NN that it doesn’t require human interpretation of data translates into its weakness. Although NN automatically extract important features and allows us to produce meaningful models, this does not mean that we can directly understand the features learned in the NN. In most cases, the features used by NN are difficult to extract in human understandable formats and it is also difficult to combine the human knowledge of mechanism into the model. Therefore, we are currently forced in the situation that we need to use the model without understanding what the AI are doing. This makes the AI model a black box and means that it is difficult to tell if the model is not suitable for the specific dataset that user wants to input. It is known that there are ways to engineer inputs that completely fakes AI models without being noticed by human.³⁴ These results suggest that NN uses different features from human and that the human perception of difficulty may not be useful for selecting suitable dataset for a network. It is necessary to share this common sense regarding the limitation of AI.

Types of NN for Special Tasks

The basic structure of NN as described in the previous section is called a “fully connected neural network” or a “multilayer-perceptron” (MLP). This structure is capable of learning features from multi-dimensional data, but they are not good at explicitly handling the location of data such as spatial or temporal position. Therefore, various special neuron units were developed to handle this problem. Of these special units, convolutional NN (CNN) and recurrent NN (RNN) are the most commonly used neuron units in the medical field.

CNN calculates outputs by performing “convolutions” using kernels. The method of calculation is shown in Figure 2. Instead of multiplying each input element by a weight, the 2-D CNN unit applies a kernel that is moved across the space of input data. The output is the sum of element-by-element multiplication on the area where the kernel is placed. Through training, the 2-D CNN unit then adjusts the kernel to minimize the difference between the outputs and the labels (ground truth). Kernels in the field of computer vision are used to extract specific features from images. Manually constructed kernels have been used for processes such as edge detection. CNN can automatically develop kernels that are suitable for extracting features that are important for the task it is used for. Thus, 2-D CNN emerged as a powerful tool to automatically extract features and classify images.

Figure 2.

Schematic diagram of a 2-D convolutional neural network (CNN). A 2D-CNN unit calculates the output by applying a kernel to the input. The output is the sum of element-by-element multiplication on the area where the kernel is placed. This calculation is repeated by moving the kernel across the space of input data, and the CNN adjusts the kernel to minimize the difference between outputs and labels.

RNN, in contrast, returns the output of the neuron unit to an auxiliary input of the same unit. This can be understood as multiple units with a rectified data transfer between the units next to them (unrolling) as shown in Figure 3. This structure is sensitive to changes in data order in the input. Therefore, the neuron unit is useful in learning the order of data and can be used for time-series data and languages.

Figure 3.

Schematic diagram of an unrolled recurrent neural network (RNN). Each neuron unit (yellow rectangle) has a rectified transfer of processed data to the neuron unit next to it. This structure allows learning of the order of data, which in most cases is time series data. Blue rectangles, elements of time-series input data.

Application of AI to 12-Lead ECG

The early AI applied to 12-lead ECG used NN structures other than CNN or RNN (in most cases MLP with or without slight modification). They used manual methods or technologies other than NN to obtain the features to train the model. This approach was successful on lead reversal detection,²³ incomplete right bundle bunch block pattern detection,²⁸ classification of ST-T segment,³⁵ classification of ECG into disease patterns (such as myocardial infarction and left ventricular hypertrophy) and detection of acute^24,27,32 or healed^25,33 myocardial infarction. These approaches, however, obviously did not fully use the ability of AI automatic feature extraction. The use of these approaches was limited by the selected features, which were usually based on human knowledge of ECG features. Thus, the applications were mainly focused on automating human tasks.

Improvements in computer and NN technology have allowed the development of deeper network patterns, which are capable of analyzing more complex data. These improvements, along with the improvement in neuron units themselves, such as the introduction of CNN, RNN and so on, have allowed direct input of ECG data into the network. This cleared the way for AI models to perform tasks that are beyond human ability.

Using 2D-CNN on raw ECG data, Attia et al have predicted age and sex,¹ and detected cardiac contractile disfunction²⁰ and atrial fibrillation from sinus rhythm ECG;¹⁸ Yang et al have detected the origin of premature ventricular contractions;²¹ and Tison et al have predicted the values of echocardiographic parameters and related heart diseases such as pulmonary artery hypertension, hypertrophic cardiomyopathy and cardiac amyloidosis.²⁶ Baloglu et al utilized a 1-D CNN network to extract lead features and combined all the features in the fully connected neuron. They successfully classified the location of myocardial infarction from ECG.³⁰ Our group used a combination of 1D-CNN with RNN and constructed a model that successfully identified patients who need urgent revascularization in the emergency room only from the 12-lead ECG.⁸ These new approaches allowed tasks to be carried out that were beyond the ability of even a well-trained expert cardiologist.

CNN or RNN for 12-Lead ECG

When applying AI technology to 12-lead ECG, the use of CNN and RNN can be seen as reflecting differences in how the ECG data structure is recognized. Using CNN is similar to the physician’s way of dealing with ECG, treating the data as an image. Even though the input is structured as raw voltage data, this approach is similar to interpreting images because the CNN approach extracts features with kernels. CNN kernels are pattern recognizers and will be activated by specific wave patterns.

In contrast, RNN explicitly deals with the ordering of data. Thus, the RNN approach treats the ECG data as a time-series. The effect on model performance of selecting CNN or RNN, however, is unclear, and the choice is currently made empirically.

There is no consensus on what tasks are suitable for CNN and what tasks are suitable for RNN. One way of choosing which structure to use involves understanding the difference in computational resource requirements for these units, especially in the ECG field. Generally, CNN are computationally cheaper than RNN. ECG is a field of rapid development of carriable devices, including implantable ones.^36–39 Some devices such as implantable cardioverter defibrillators require real-time classification of ECG data, which is a possible field of AI application.⁴⁰ These devices usually possess limited computational resources. In these situations, CNN would be an attractive choice, and in some cases the only choice because of the limited resources, even when RNN can achieve better accuracy. If there is no limitation on computational resources, the networks should be compared head to head. If there is no merit in using RNN over CNN, CNN should be preferred. Theoretically, however, RNN can learn more precisely the time-series voltage data, which is the rawest form of ECG data. It is not unreasonable to assume that RNN can extract more data from ECG than CNN. Thus, some complex tasks may still require RNN. Identifying the tasks suitable for CNN and RNN is an interesting topic for future AI research in this field.

Current Problems and Future Directions

There are some limitations of AI usage that should be specifically noted with regard to the medical field. The first problem is the need for large-scale data with which to train NN. This is a general issue in NN usage but is especially important in the medical field, where we model diseases that are rare. Ways to reduce the required input are being actively developed such as reinforcement learning. From the medical side, however, we should develop a system for data sharing in order to maximize the utilization of available data. We have already created a way to digitize and share the information in the field of financial data. This information is almost as important as health information and requires similar security. Thus, digitizing and sharing health information should also be able to be achieved (Figure 4).

Figure 4.

Schematic diagram of data sharing.

The second problem is lack of human understanding of the ECG features. NN are capable of automatically extracting features from complex inputs such as raw ECG data. This is powerful when the objective is to create an automatic classifier or automatic prediction tool. The features identified by AI models, however, are difficult to extract in a human-understandable format. This makes the model a black box and limits the clinical use of these models because it is difficult to tell if the model works for a specific population or not. Another shortcoming is that the model is not able to enhance the understanding of the disease. Contemporary models such as logistic regression modeling provide information about which features are independently associated with outcomes and so on, which give mechanistic insights into the disease. There are ongoing efforts to extract human-understandable features from AI models and this is an important topic for future AI research.

Conclusions

AI models based on various NN complexes have a huge potential to assist in clinical decision making. Limitation in the understanding of the mechanism on how the AI model makes predictions, however, should also be noted.

Disclosures

Shinya Goto acknowledges financial support from Bristol-Myers Squibb from their independent research support project (33999603). Shinya Goto received research funding from Sanofi, Pfizer, and Ono. Shinichi Goto declares no conflicts of interest.