Fusion Modeling; Combining Clinical and Imaging Data to Advance Cardiac Care

Abstract

In addition to the traditional clinical risk-factors, an increasing amount of imaging biomarkers have shown value for cardiovascular risk prediction. Clinical and imaging data are captured from a variety of data sources during multiple patient encounters and are often analyzed independently. Initial studies showed that fusion of both clinical and imaging features, results in superior prognostic performance compared to traditional scores. There are different approaches to fusion modeling, combining multiple data resources to optimize predictions, each with their own advantages and disadvantages. However, manual extraction of clinical and imaging data is time- and labor-intensive and often not feasible in clinical practice. An automated approach for clinical and imaging data extraction is highly desirable. Convolutional neural networks and natural language processing can be utilized for the extraction of electronic medical records (EMR) data, imaging studies, and free-text data. This review outlines the current status of cardiovascular risk prediction and fusion modeling, and in addition gives an overview of different AI approaches to automatically extract data from images and EMR for this purpose.

Introduction

Cardiovascular disease (CVD) is a major contributor to global mortality, representing 31% of all global deaths¹. Traditional risk factors obtained from population-based studies are used to predict adverse cardiac outcomes such as the pooled cohort equations CVD risk calculator^2,3.

Cardiac imaging may be seen as a bridge between upstream risk-factors and CVD, as several cardiac imaging modalities are currently clinically well integrated for the evaluation of subclinical vascular and ventricular function. Echocardiography, cardiac nuclear imaging, coronary computed tomography angiography (CCTA) and cardiac magnetic resonance (CMR) imaging play an important role in CVD diagnosis and prognosis^4,5. With the rise of artificial intelligence (AI), the role of cardiac imaging is being leveraged for several different applications, including data analysis, quantification, and prognostication⁶.

Clinical and imaging data are captured from numerous data sources during patient encounters and are often analyzed independently of each other. However, such models fail to exploit longitudinal and complementary information from different data streams. Overall, AI enables the development of models that combine multi-modal data from large populations for the prediction of CVD outcomes. Initial studies, integrating both clinical and imaging features using an AI-based fusion approach, showed superior prognostic performance compared with traditional scores to predict major adverse cardiac events (MACE)^7,8. Although the combination of multi-source shows promising results, manual extraction of clinical and imaging data is time- and labor-intensive and not feasible in clinical practice. Different AI approaches can be utilized for the extraction of clinical data from the electronic medical records (EMR), imaging studies, and the identification of important factors from multi-modal data.

Automated feature extraction makes fusion modeling clinically applicable. This review will discuss the current status of cardiovascular risk prediction, multi-modality data fusion approaches and their limitations. In addition, it gives an overview of automated AI based extraction of data that can serve as input for fusion modeling.

Prognostication Using Clinical Risk Factors and Imaging Biomarkers

Clinical CVD Risk Factors

The use of clinical risk factors for CVD risk prediction is a cornerstone of preventive cardiology. Hypertension, dyslipidemia, diabetes, smoking, and obesity are among the five strongest and most common modifiable risk factors^2,3. While traditional risk factors are associated with a higher incidence of CVD events, there is considerable heterogeneity among persons who have elevated CVD risk factor burden^9,10. Overall, such data suggests that traditional risk factors incompletely explain CVD risk, and further support the concept of cardiac imaging to improve risk stratification.

Imaging-Based CVD Risk Factors

There are several different imaging modalities for CVD risk stratification, including ultrasound, CT, CMR, and nuclear imaging. Table 1 summarize the major fields of CVD imaging for each modality and the most relevant AI applications ⁶.

Table 1.

Integration of common cardiac imaging modalities and

Imaging Modality	Important Measures	Select Clinical AI-Based Applications
Coronary Calcium CT	- Coronary artery calcium  - Extra-coronary artery calcium (including thoracic, aortic and mitral valve)	- Improved workflow for traditional coronary artery calcium scoring  - Automation of extra-coronary artery calcium scoring
Cardiac Computed Tomography Angiography	- Calcified and non-calcified plaque  - Plaque characteristics   ○ Number of coronary segments with atherosclerosis   ○ High risk plaque features: napkin ring sign, low attenuation plaque, positive remodeling	- Integrating risk scores for high-risk plaque features
Echocardiography	- Systolic function   ○ Ejection fraction  - Diastolic function   ○ E/e, E/e’, isovolumetric relaxation time, deceleration time, left atrial maximum volume index, peak tricuspid regurgitation velocity  - Global longitudinal strain	- Deep-learning models for phenotyping diastolic dysfunction for early Heart Failure with preserved Ejection Fraction (HFpEF) risk stratification
Cardiac Magnetic Resonance Imaging	- Delayed gadolinium enhancement  - Cardiac function	- Scar assessment for post-myocardial infarction and sudden cardiac death risk stratification  - Automated Left ventricular function analysis
Nuclear Imaging	- Myocardial Perfusion   ○ Coronary artery disease   ○ Microvascular disease	- Automated Perfusion Quantification

Computed Tomography

Cardiac CT has become widely utilized because of its non-user dependence, reproducibility and short acquisition time ^11,12. Non-contrast cardiac CT has emerged as a powerful technique because coronary artery calcium (CAC) is a direct measure of subclinical atherosclerosis burden, strongly associated with CVD risk¹³, predicting downstream mortality^14,15, and can help identify those who are most likely to benefit from pharmacotherapy. The most commonly used method for quantifying CAC was introduced by Arthur Agatston and colleagues in 1990 ¹⁶(Figure 1A). Coronary CTA has been recently recommended by several societies and guidelines as a first-line diagnostic test for patients with suspected stable CAD^5,17. Traditionally, coronary CTA (Figure 1B) is used to exclude CAD, having excellent negative predictive value. However, it also has been used to assess stenosis severity and guide follow-up examinations and treatment, as outlined in the CAD-RADS 2.0 classification¹⁸. Studies have shown that total plaque volume increases the accuracy of outcome prediction compared to stenosis classification alone^8,19–21. Non-calcified plaque volume might indicate more vulnerable plaque and therefore might be a more accurate biomarker compared to calcified plaque volume. However, these analyses are time and labor intensive, using AI allows for a standardized analysis in a time efficient manner, making it more practical to use in clinical practice. Examples of AI supported analyses for CAD-RADS 2.0, plaque volume quantification and high-risk plaque feature detection are shown in Figure 2(A–D).There are several high-risk plaque features that are important to consider for risk prediction, such as positive remodeling, low attenuation plaque and the napkin-ring sign^22–24, see Figure 2 (E–F).

Figure 1.

Coronary Artery Calcium (CAC) acquisitions and Coronary CT angiography (CCTA) allow a direct measure of atherosclerosis and plaque vulnerability features strongly associated with Cardiovascular Disease (CVD) risk. Panel A, Agatston calcium and MESA score are a powerful cardiovascular risk predictor. Panel B, CCTA can concurrently identify obstructive and non-obstructive atherosclerotic lesions. (LM- left main, LAD-Left Anterior Descending, CX- Circumflex, RCA- Right Coronary Artery)

Figure 2.

AI-based algorithms for coronary evaluation and plaque burden quantification on CCTA. Panel A, AI prototype for automated plaque detection, stenosis severity quantification, and Coronary Artery Disease – Reporting and Data System (CAD-RADS) classification (HeartAI Siemens Healthineers). Panel B, plaque burden quantification and identification of high-risk plaque features (blue arrow) can be automated using AI approaches (Cleerly Inc). Panel C, shows an example of plaque component quantification (Elucid Bioimaging Inc.) and several imaging biomarkers of plaque vulnerability, that may be used for Cardiovascular Disease (CVD) risk stratification, including low plaque attenuation (C), positive remodeling (D), and spotty calcifications (E).

Echocardiography, Cardiac Magnetic Resonance

Beyond atherosclerosis-based imaging, the relationship of ventricular structure and function with upstream risk factors is also important in CVD risk assessment, particularly for heart failure with preserved ejection fraction (HFpEF), which has a rapidly growing incidence and societal burden²⁵. While the diagnosis of heart failure with reduced ejection fraction has become largely standardized, the early identification and assessment of HFpEF on imaging has been more challenging. This latter concept has led to an effort to diagnosis early diastolic dysfunction using both echocardiography and CMR²⁶, and to identify potential risk factors associated with this subclinical phenotype to help prevent progression to HFpEF.

Overall, coronary microvascular dysfunction, systemic vascular dysfunction, as well as skeletal muscle, renal, and adipose pathophysiology have all been thought to be notable pathways involved in HFpEF²⁵. Speckle tracking echocardiographic measurement of global longitudinal strain and myocardial tissue characterization via extracellular volume fraction by CMR may also be important sensitive tests of diastolic dysfunction and HFpEF risk assessment. Of note, CMR also has an important role in characterizing heart failure with mid-range ejection fraction, which appears to share many phenotypic attributes, including diffuse fibrosis and hyperemic myocardial blood flow, with HFpEF²⁷. Figure 3 shows an example of AI based left ventricular function and volume assessment.

Figure 3.

Cardiac MR AI-based algorithms for ventricular volume and function assessment can provide rapid and accurate analysis for cardiovascular risk prediction. Images show automated cardiac chambers segmentation for left ventricle volumes and segmental contraction evaluation (Circle Cardiovascular Imaging Inc.).

Nuclear Imaging

Nuclear imaging techniques such as single positron emission computed tomography (SPECT) and positron emission tomography (PET) have been the backbone of cardiac imaging and risk prediction for many decades, having the capability to accurately depict myocardial perfusion abnormalities related to coronary disease and microvascular dysfunction^28,29. Myocardial perfusion is a powerful predictor of CVD events. The number of cardiac SPECT and PET studies performed annually in the US still outnumbers the CMR and CCTA studies, being the most frequent and often first imaging test for non-invasive ischemia detection³⁰.

Multi-Modality Data Fusion

Patient information, recorded from different data sources such as genetic information, imaging data, ECG signals, and tabular EMR data, are often processed in isolation from each other. Current focus is on integrating multiple sources in a single model, often termed as a fusion model, to build more comprehensive prediction models that outperform single-source models. Data fusion has several advantages such as increasing the accuracy of diagnosis, ease of interpretation, and summarizing and sharing information. Fusion modeling can be used in a multitude of cardiac applications such as multi-omics projects including data from genomics, radiomics and proteomics, prediction of drug efficacy and drug interactions in addition to prognostication and risk stratification.

Clinical Use Cases

Although fusion modeling is mostly used in a research setting there are promising clinical scenarios.

Scenario 1:

With increasing options for cardiovascular diagnostics and therapy a more personalized approach seems to yield better results. Fusion modeling will allow for an additive approach using patients’ symptoms, demographics and risk factors, known at first visit to the cardiologist, combined with later performed follow-up testing such as ECG and CT imaging, see Figure 4. The combination of these features can subsequently be used to predict which course of treatment would benefit this specific patient the most and optimize the type of therapy prescribed. With new developments in medication, fusion modeling could be used to select the most optimal type of medication to get optimal results while reducing unnecessary therapy and procedures. This is especially relevant with novel expensive drug developments. AI can assist in the identification of novel drug targets, design and select new drug molecules with favorable drug properties, predict drug/target interaction, and assist in patient selection for clinical trials or creating virtual trials using existing databases³¹. An example is Biotech InSilico Medicine using AI to create the drug INS018–055 to help treat idiopathic pulmonary fibrosis, the first drug with both a novel AI-discovered target and AI-generated design currently undergoing Phase 2 testing.

Figure 4.

The use of demographics, risk factors, symptoms and clinical risk scores can be combined with information from non-invasive imaging test such as CAC and CAD-RADS to create a personalized treatment plan. This includes medication that is predicted to be more beneficial to specific patients, personalized treatment goals and follow-up schedules. (CTA- CT Angiography, CAC-Coronary Artery Calcium, CAD-RADS- Coronary Artery Disease – Reporting and Data System, RF- Risk Factors, DM2- Diabetes Mellitus type 2, CHD- Coronary Heart Disease, Lp(a)- Lipoprotein(a), ASA-Aspirin, LDL-C- Low-Density Lipoprotein Cholesterol)

Scenario 2:

Fusion modeling can be used to create a digital twin for ablation procedures, as utilized by Siemens Healthineers. The combination of imaging, ECG and electrophysiology mapping allows for anatomical and electrophysiology modeling of the cardiac substrate. With the combination of these multi-source models, a digital twin can be created on which a personalized virtual ablation plan can be created and the most likely outcome can be visualized before the actual procedure to determine the most successful approach. During the procedure the digital twin can help guide the procedure, see Figure 5.

Figure 5.

By using the CT and MRI images for segmentation and anatomical modeling and using ECG and electrophysiology mapping to create a digital twin of the heart, the combination of these data sources can be used to identify ablation target and plan out the procedure and visualize the outcomes before the procedure to identify the most optimal procedure strategy. The virtual model can be integrated into the procedure to guide the actual procedure. Figures courtesy of Siemens Healthineers. (EP- Electrophysiology, ECG- Electrocardiogram, EAM-Electroanatomic Mapping, VT-Ventricular Tachycardia)

Current Literature

Fusion modeling has traditionally been performed through late, early, or middle fusion³² (Figure 6). Amal et al.³³list a number of fusion models designed to improve care for patients with CVD Recently, graph mining frameworks have also been developed to fuse and analyze multi-modal data^34–36, where graph convolutional neural networks (GCNN) can jointly process multi-source data to make predictions, see Figure 7. When facing missing data in certain modalities, generative models such as generative adversarial networks (GANs) and deep diffusion networks have the potential to jointly enable cross-modality data imputation and downstream diagnosis/prognosis predictions. Research has also focused on combining sequential information collected over multiple patient interactions with the healthcare system at varying time intervals. Complex representation learning techniques have been designed to fuse several structured data modalities that can be later used for CVD risk prediction.

Figure 6.

Traditional fusion modeling approaches; late fusion, early fusion, and middle fusion. Late fusion methods can preserve the complete information of each data modality, but they cannot fully explore the interactions among data modalities. Early fusion methods, on the other hand, can potentially find complex cross-modality features but are often harder to properly supervise. In consequence, middle fusion methods offer a compromise, however, their designs are often ad hoc and require domain knowledge toward relations between the modalities

Figure 7.

Graph neural network for fusion modeling. Different data elements may be used as node and edge feature vectors. Edge features vectors of two samples are evaluated for similarity to decide an edge between the nodes corresponding to these two samples. Graph neural network learns updated node representations based on original node features as well as edge structure formed based on edge features, hence, fusing the two data elements.

In the field of cardiology, the term ‘fusion’ modeling traditionally referred to AI models that combined clinical information such as age, comorbidities, etc. with information extracted from imaging data. Motwani et al.⁸ investigated 25 clinical (age, sex, gender, risk factors, and Framingham risk score (FRS)) and 44 CT parameters (segment stenosis score (SSS), segment involvement score (SIS), modified Duke index (DI), number of segments with plaques) for prognosis. They showed that a fusion AI approach exhibited higher AUC’s (0.79) for predicting all-cause mortality compared with FRS (0.61) or CT based scores alone (0.64). Betancur et al.³⁷ fused clinical information with information extracted from SPECT myocardial perfusion imaging (MPI) for prediction of MACE, showing increased AUC for the fusion approach compared to imaging only (0.81 vs. 0.78). Al’Aref et al.⁷ combined clinical factors with CAC score estimated from cardiac CT scans for estimating risk of obstructive coronary artery disease (CAD), with AUC’s of 0.88 for fusion vs. 0.87 for imaging and 0.77 for EMR data only. These models, while innovative, fall short of fusing imaging data directly with clinical information. Chaves et al. ³⁸ fused L3 slices from CT for body composition analysis, and known clinical risk factors through late fusion, for opportunistic screening for ischemic heart disease. They observed that fusion modeling outperforms single-modality models in terms of screening performance. Fusion models are not limited to EMR and imaging data. Experiments have been performed to expand the scope of fusion to include ECG-signals, phonocardiograms, various wearable sensors, as well as genetic data. Li et al.³⁹ combined visual features from ECG and PCG to improve prediction of CVD. Zhao et al.⁴⁰ combined EMR data with genetic features through late fusion for CVD event prediction. Ali et al.⁴¹proposed a smart healthcare monitoring system for prediction of CVD that fuses EMR data with wearable sensors like respiration rate sensor, an oxygen saturation sensor, a blood pressure sensor, a cholesterol level sensor, a glucose level sensor, a temperature sensor, an EMG, ECG and EEG sensor. Zhang et al.⁴² developed a screening tool to discriminate acute chest pain patients with cardiac cause from and non-cardiac causes. They fused features from ECG, phonocardiograms, echocardiography, Holter monitors, and biological markers in an early fusion setting. Huang et al.⁴³ used the latest GCNN based model to predict CAD using vascular biomarkers derived from fundus photographs by fusing imaging information with patient characteristics.

As with all AI modeling, it is essential to choose the right approach for the problem. It is essential to choose a model with the highest accuracy and the lowest complexity. Parameter selection will always be an important step to avoid overfitting of models⁴⁴.

Limitations of manual data extraction

The extraction of EMR data and imaging-based biomarkers can be time and labor intensive. Because of this, only few studies are performed using a combination of clinical and imaging data on very select populations. In addition, the a-priori selection of parameters greatly limits the information available and inherently introduces bias. By included only a select few biomarkers, the opportunity is missed to create a model that considers thousands of different parameters for detecting currently unknown patterns and identify novel prognostic variables. The use of a large number of variables also requires large populations in which this is investigated and validated. The labor intensity of creating these populations without automating the process is severely limiting fusion modeling. Manual extraction of data also suffers from interpretation errors and variability between institutes, hospitals and countries. Automating the process using AI could be used to standardize the EMR and imaging data extraction process, making it feasible to create large diverse populations with standardized biomarkers, allowing large scale evaluation of CVD patients using fusion modeling.

Automated Data Extraction for Fusion Input

Role of NLP for EMR data Processing

As healthcare databases are growing exponentially in size, more information is hidden in the form of free-flowing text such as clinical notes and radiology reports. The domain of NLP is concerned with both the syntactic and semantic understanding of free-flowing text at multiple levels such as words, sentences, paragraphs, and documents. We refer the reader to the foundational book of NLP by Jurafsky et al.⁴⁵ NLP can be used in different ways, to extract features that can be relevant for the model as a predictive factor or to extract reference labels from radiology reports for training AI algorithms without needing manual annotations. In addition, NLP can be used to mine data on drug properties and literature to optimize novel drug development.

NLP Approaches for Cardiology

To provide an overview of the vast amount of literature, we will divide NLP models for cardiology into two categories; rule-based NLP algorithms and AI-based NLP algorithms.

Rule-based NLP algorithms have been generally used to extract information from large collections of documents such as coronary catheterization reports, radiology reports, and clinical notes. Rule-based systems had been developed for extraction of coronary anatomy related terms from clinical text, such as coronary catheterization reports, as early as 1998^46,47. In early 2000, rule-based models were developed to extract useful information, including diagnoses of heart failure, chest pain⁴⁸ and QT prolongation⁴⁹ from EMR and ECG reports. Information extraction methods have gone beyond simple extraction to understanding relationships between extracted pieces of information from large repositories. Hamon et al.⁵⁰ extracted relationships between pathologies and 22000 risk factors from the MedLine repository of research papers in 2010. Sophisticated concept extraction models such as UMLS, MedTagger, and cTAKES ^51–54 have been developed recently which have been used for extraction of clinical concepts from thousands of clinical notes ^55,56. However, rule-based techniques suffer from poor generalization capabilities, in addition to requiring input from domain-experts to craft these rules. Rules defined for one data source may not be applicable to another data source. With no automatic method for updating or fine-tuning rule-based methods, the field of NLP started to shift toward statistical machine learning. Often, rule-based techniques were used to annotate a dataset for training a statistical machine learning classifier resulting in a hybrid approach⁵⁷.

AI-based NLP algorithms can be used to automatically label cases with a certain disease. More recently, deep learning-based models are being applied for information extraction. Pandey et. al⁵⁸ labeled CT reports with presence or absence of radiologic findings like aortic aneurysm or cardiomegaly using a deep learning-based model. Datta et. al⁵⁹ extracted complex relationships between radiologic findings, their locations and their characteristics. Instead of extracting relevant information for clinicians to process, Sung et al.⁶⁰ created an AI-based NLP model to directly predict poor functional outcome among patients hospitalized after ischemic stroke using free-text of CT reports and history of present illness from clinical notes.

AI-based NLP algorithms have been used to extract information from unstructured radiology reports for automated structured reporting. Tariq et al.⁶¹ developed a 1D convolutional neural network classifier to predict Coronary Artery Disease - Reporting and Data System (CAD-RADS) score from unstructured reports of coronary computed tomography angiography (CCTA) reports. Faryna et al.⁶² designed a weakly-supervised system for CT report classification with disease labels covering several organs including liver and lungs. Hassanpour et al.⁶³ designed a pattern mining based AI-based NLP module to automate the process of converting free-flowing information contained in reports to structured information in well-designed databases. All these data can be subsequently used in fusion models to add high level image interpretation to the prediction models.

Role of AI based Image Processing

The challenge of processing high-dimensional data such as medical images was first tackled by radiomics approaches, focused on extracting high-throughput quantitative features from medical images using statistical techniques like histograms, and texture and fractal analysis⁶⁴. Radiomics enables the quantification of image features like size, shape, heterogeneity, or repetitive patterns. Although the field of radiomics is promising, radiomic features are known to be subjective to interrater variability and often need manual identification of the region of interest⁶⁵. Radiomics features have been widely used as input for AI models for downstream prediction tasks, including diagnosis^66,67

Deep learning greatly enhanced the capabilities of image processing by introducing learnable convolutional filtering which allows models to learn spatial characteristics from images tailored to the down-stream prediction tasks while keeping model complexity in check. The drawback of this approach is the requirement of large training datasets. This problem has been tackled by pre-training models on large public datasets like ImageNet.

We will divide deep learning based architectures into three categories; 1) Segmentation models, 2) Prediction (classification or regression) models, and 3) Image generation models (Figure 8). Models of the first two categories have similar initial layers, i.e., multiple convolution filtering layers, but differ in the design of their final layers and computation of loss for model training. Generative adversarial network (GAN) is the most popular architecture of category 3. Litjens et al.⁶ provided a detailed overview of the applications of image processing in the cardiovascular imaging domain.

Figure 8.

Broad classes of image processing models for cardiac imaging studies with main purpose of 1) segmentation, 2) numerical prediction and 3) image generation. Depending on the purpose, convolutional neural networks (CNNs) consists out of convolutional and fully connected layers and are the current standard to process imaging data.

Segmentation of anatomical regions has been a well-known barrier for the application of image processing on cardiovascular scans. Examples include semi-automated segmentation of right ventricle from short axis CMR⁶⁸, endocardial contouring for left ventricular (LV) volume and ejection fraction (EF) estimation from 3D transthoracic echocardiography⁶⁹, and lumen vessel segmentation from contrast enhanced imaging modalities for atherosclerotic plaque detection⁷⁰. Deep learning model architectures like U-Net can perform tasks like coronary artery calcium scoring⁷¹ and left ventricle function estimation from CMR. Chen et al.⁷² provided a detailed survey of latest segmentation techniques applied to CMR, CT, and echocardiography.

For many image processing models for cardiovascular images, the goal is categorical (classification) or numerical (regression) output prediction. For example, echocardiography has been used for direct classification of disease rather than LV volume of EF estimation^73,74. Other classification labels for echocardiography have included hypertrophic cardiomyopathy (HCM), cardiac amyloidosis (amyloid), pulmonary hypertension (PAH), presence of pacemaker and wall motion abnormalities⁷⁵. Automation of CAC from coronary CTA has been a popular target for prediction models⁷⁶. AI-based CAC scoring has excellent agreement with human readers in a fraction of the time and as a result is being employed clinically ⁷⁷. In addition, several companies have received FDA approval for AI-based plaque analysis, enabling plaque burden quantification as well as the identification of different plaque compositions and high-risk plaque features.

Regression models may predict continuous value output such as LV volume⁷⁸. Model architecture is often very similar for regression and classification, but different loss functions are used.

Scarcity of data with desired properties like a specific imaging modality (e.g. CT or MR) or image quality in terms of noise or contrast for training of complex deep learning-based image processing models has led to a vast amount of research in generation of synthetic imaging data. Generative adversarial network (GAN) is the most popular architecture for imaging data generation. GANs have been employed to generate MR images from available CT scans⁷⁹ and to reduce noise in low-dose CT scans⁸⁰.

Limitations and Challenges Fusion Modeling

Fusion Modeling

Although studies have shown the benefit of combining clinical and imaging data for risk prediction and prognostication, fusion modeling is a nascent field. Each fusion model has its own strong and weak points, which makes them suitable for different types of problems and different types of datasets. Things that should be taken into consideration are different dimensionality of input data, the size of the datasets and number of features extracted and the amount of missing data per input entry. With the increased interest in fusion modeling, new fusion methods are being developed continuously. Future studies should explore optimization of these fusion models and creating pipelines for the automated extraction of data from both EMR and imaging biomarkers. Interoperable pipelines will allow larger databases for fusion imaging, benefitting the development of more complex and accurate AI algorithms. In addition, it is important to address issues such as data quality and consistency, inappropriate model selection and computational burden.

Bias and generalizability in AI

One of the major challenging with AI algorithms in medicine in general is the introduction of bias which can be introduced during development and deployment⁸¹. The major forms of bias that are of concern include when the training dataset does not reflect the clinical use population and when the reference labels are subject to inconsistencies. Data curation and labeling are subject to human bias and can introduce bias to the algorithm. Careful consideration should be made when considering data selection and reference labeling, including methodological approaches to avoid over or underfitting and model evaluation metrics. Transparency about the chosen population and methodology are necessary to avoid adverse effects of non-intentional created biases.

Ethics

As AI-based approaches are introduced into the clinical workflow, experts will have to consider who becomes responsible for the modeling outputs and the consequences thereof. Very few autonomous AI application in medicine have been introduced, most AI application aim to assist clinical experts instead of fully taking over tasks. With AI implementation it is essential to consider the ethical consequences of the use of AI and the impact it will have on patients and healthcare providers.

Clinical Implementation

There are several challenges with fusion modeling/AI that limited clinical implementation. In addition to the issues mentioned above, workflow compatibility, reimbursement and legal implications all hamper clinical use of these AI models. There is little information on the efficiency of AI models and the effects on patient outcomes. In addition, a challenge specific to predictive and prognostic models, is that they aim to improve patient outcomes by guiding treatment and intervention. This requires a specific type of performance assessment and algorithm maintenance to ensure consistency in the performance over time. Workflow integration, making sure use of AI is time-efficient while giving accurate predictions, that are easy interpretable by the user is essential. Guidelines are needed to guide clinical implementation and ensure adequate use of AI in clinical practice.

Future developments

Fusion modeling is at a beginning stage for cardiovascular disease and has shown promising results for prognostication purposes. In the future it is expected that fusion modeling can play a larger role in the design of clinical trials, identifying patients’ specific profiles for drug and therapy development. The concept of a health digital twin, encompassing both clinical and imaging data, could potentially allow more accurate phenotyping of individual patients with the same condition or presentation, using multiple clinical, imaging, molecular and other variables to guide diagnosis and treatment. The next steps should focus on how these techniques can be implemented into clinical practice and fit into clinical workflow. Infrastructure for the technical implementation need to be created in addition to a legal structure and user guidelines. For clinical adaption to happen, clinical trials that show that fusion modeling approaches are safe, efficient and indeed improve patient care are needed.

Conclusion

There is an urgent clinical need for computational models that can aggregate multiple heterogeneous streams of data to facilitate patient-centric care. The use of NLP and AI-based image analysis can facilitate the extraction of valuable information from free-text and imaging data. The use of multi-modal data fusion approaches allows for the combination of both clinical EMR and imaging data and can be used to drug development, optimize risk and outcome prediction and create personalized treatment strategies. While the combination of multi-modal data sources is not new, these novel AI-based approaches greatly reduce the time and labor intensity making it possible to clinically implement the use of automated personalized multi-modal cardiovascular risk prediction.