A generalizable deep learning system for cardiac MRI

Abstract

Cardiac MRI allows for a comprehensive assessment of myocardial structure, function and tissue characteristics. Here we describe a foundational vision system for cardiac MRI, capable of representing the breadth of human cardiovascular disease and health. Our deep-learning model is trained via self-supervised contrastive learning, in which visual concepts in cine-sequence cardiac MRI scans are learned from the raw text of the accompanying radiology reports. We train and evaluate our model on data from four large academic clinical institutions in the United States. We additionally showcase the performance of our models on the UK BioBank and two additional publicly available external datasets. We explore emergent capabilities of our system and demonstrate remarkable performance across a range of tasks, including the problem of left-ventricular ejection fraction regression and the diagnosis of 39 different conditions such as cardiac amyloidosis and hypertrophic cardiomyopathy. We show that our deep-learning system is capable of not only contextualizing the staggering complexity of human cardiovascular disease but can be directed towards clinical problems of interest, yielding impressive, clinical-grade diagnostic accuracy with a fraction of the training data typically required for such tasks.

Offering unparalleled diagnostic clarity, cardiac magnetic resonance imaging (CMR or cardiac MRI) is the reference standard for assessing cardiac anatomy and function^1,2. Depending on the technique used, it enables clinicians to capture videographic sequences of cardiac and valvular motion, quantify scarring or tissue oedema within the cardiac musculature, and identify regions of poor perfusion—all without exposure to ionizing radiation². Despite this wealth of data available, deep-learning systems capable of learning high-quality representations of human cardiac disease from CMR have not yet been described.

Deep learning has shown incredible promise in the diagnosis of cardiovascular disease with electrocardiography (EKG), retinal scans and echocardiography^3–7. Traditionally, these networks are trained to detect a handful of pre-defined and curated ‘disease’ conditions in the background of negative ‘normal’ cases. While superficially successful, many of these systems fail when tested on real-world clinical data that are often heterogeneous with numerous concomitant abnormal findings⁸. Patients with inherited cardiomyopathies, for example, may present with severe valvular disease; those with evidence of ventricular thrombus, on the other hand, may have severe heart failure from remote ischaemic insults. With the traditional supervised deep-learning approach, it remains challenging to contextualize this diversity of disease presentation from CMR scans, with parameters learned for one problem rarely generalizing to others⁹. These systems, therefore, must be re-trained from scratch for every new clinical task of interest, requiring thousands of training examples each time. Unlike human clinicians, deep-learning models do not have a baseline foundation of clinical and pathophysiological understanding over which learning specific tasks can be accelerated. Ultimately, this has restricted research in the field to tasks that either automate objective structural measurements or to diseases that are simply more prevalent¹⁰.

Here we describe a transformer-based vision system that learns complex pathophysiological visual representations from a large multi-institutional dataset of 19,041 CMR scans, guided by natural language supervision from the text reports accompanying each CMR study. We use a large language model to help ‘teach’ a vision network to generate meaningful low-dimensional representations of CMR studies, by showing examples of how radiologists describe what they see while drafting their reports. We describe how these models group together patients with similar pathophysiological and even socio-demographic characteristics, with no explicit supervision for these tasks. We fine tune this deep-learning system on smaller expert-labelled datasets for a range of clinically relevant tasks ranging from left-ventricular ejection fraction estimation to the detection of 39 different cardiovascular conditions. We validate our system on multiple external datasets from distinct geographical sites and health systems. The result is a generalizable and data-efficient CMR deep-learning system capable of representing the breadth of cardiovascular disease and health. Figure 1 details an overview of the project.

Fig. 1 |. Project overview.

a, Large-scale contrastive pre-training. CMR cine sequences in the form of video data are fed to a video transformer network, and the paired text reports are fed into a parallel text transformer network. The embeddings produced by each (V, video; T, text) represent compressed representations of the complex inputs to each network. The networks are trained to maximize agreement between true pairs of video and text that arise from the same CMR scan (that is, the diagonal of the similarity matrix highlighted: V₁T₁, V₂T₂ … V_nT_n) and minimize the agreement between samples that come from different scans. b, In the process in a, we see that as training progresses, more complex local and global structural features emerge in the video embeddings (V). The scans initially viewed as an indistinct amalgam in high-dimensional space at the start of training (epoch #1) begin to develop separations and localized clusters that are more distinct by epoch #100. c, Evaluating the ability of the network, on external datasets, in separating diseases or probing the UK BioBank as described in later sections is achieved by freezing all the parameters of the video encoder (at this point, the text network is discarded). Embeddings produced from different views can then be plotted in 2D via standard dimension reduction algorithms. d, Finally, leaving the last layer unfrozen (~600,000 trainable parameters), we showcase data-efficient fine tuning towards specific clinical tasks of interest. Of note, the embeddings are fed into a secondary network designed to aggregate information from different views into a single prediction, with a few examples listed: regression of left-ventricular ejection fraction (LVEF%) and diagnosis of amyloidosis or ischaemic cardiomyopathy. A single vision model can thus be applied to a wide range of different tasks.

CMR imaging data

The inputs to our networks are steady-state free precession (SSFP) cine sequences taken along multiple different cardiac view planes. Typically, SSFP sequences are characterized by a high blood signal intensity, with lower intensity myocardial signal^11,12. This technique allows for the acquisition of high-contrast dynamic-motion scans of the heart during a breath hold timed with EKG gating to capture images between heart beats. These cine-sequence videos are captured along various spatial view planes as described below. Our data are sourced from a heterogeneous sample population with substantial variability in acquisition and scanner vendors, with training data sourced from Stanford Medicine, MedStar and UCSF. Additional data from the UK BioBank, Kaggle Grand Data Challenge, Automated Cardiac Diagnosis Challenge (ACDC) dataset and the University of Pennsylvania were sourced to rigorously assess model generalizability. Specific descriptions of each cohort are provided in Methods and Supplementary Fig. 1.

Different cardiac structures may be visualized better in one view plane than in another. As a result, findings indicative of pathology may only be visible in a specific MRI view plane. This is conceptually analogous to the challenges faced in deep learning for histopathology, where only a fraction of the biopsied tissue might contain findings diagnostic of disease¹³. We limit ourselves to short-axis stacks (SAX), four-chamber (4CH), three-chamber (3CH) and two-chamber (2CH) views for this work as these are most consistently captured irrespective of query pathology. Demographic data available for certain subsets of the data are detailed in Supplementary Table 1.

Pre-training framework and evaluation of low-dimensional representations

The traditional approach for deep learning with cardiovascular imaging has been to assign some label to each imaging scan via tedious manual annotation. Large-scale labelling of training data requires extensive domain knowledge and clinical expertise, but ultimately constrains the models to features explicitly labelled in the dataset. Models trained this way require thousands of examples for adequate performance and are unable to account for findings outside the scope of what they were trained to identify. Recent advances in self-supervised deep learning have shown promise in reducing this reliance on vast quantities of expert-labelled data across a wide range of modalities ranging from pathology slides to chest X-rays^14–21. A framework wherein unstructured text could be used as a method of self-supervision for a chest X-ray deep-learning system has been described¹⁶. In this method of contrastive learning, two neural networks are used to produce a pair of low-dimensional representations for each pair of contextually related inputs from two separate modalities²². We extend these concepts to the spatiotemporal and multiview problem of cardiac MRI. During the pre-training process, the networks are trained to match true pairs of text report and MRI scans by optimizing for a contrastive objective²². Since the release of our initial preprint, others have reported promising results using similar contrastive-learning methods to train generalizable ‘foundation models’ for computed tomography (CT) and echocardiography^23–25.

Visual features of a particular disease are iteratively associated with textual cues, with a training process guided by a rich unstructured description of disease created by clinicians as part of routine clinical workflow. We use an implementation of a multiscale vision transformer (mViT) for the CMR vision encoder and a bidirectional encoder representations from transformers (BERT) text transformer for the MRI report encoder^26,27. We pre-train and validate on 14,073 cardiac MRI scans (12,707 unique patients) from Stanford, UCSF and MedStar. Specific implementation details are described in Methods.

We plot the embeddings generated by the vision encoder on the validation set using standard dimensional reduction algorithms (Methods), demonstrating the progressive emergence of local and global structure in the two-dimensional (2D) projections of these embeddings as training progresses (Fig. 1). Once training is complete, the network is frozen and deployed onto a publicly available external dataset of cine-CMR sequences sourced from France (ACDC dataset)²⁸. We plot the embeddings for each study in the ACDC dataset onto two dimensions and confirm that compared to a baseline network of the same architecture, our vision system is able to separate different disease conditions with remarkable consistency (Supplementary Fig. 4). This is despite the lack of any directed supervision during the training process in separating conditions such as hypertrophic cardiomyopathy from dilated cardiomyopathy or right ventricular dysfunction.

Validation on the UK BioBank

We sourced 159,883 cine-CMR scans representing 45,623 participants from the UK BioBank and assessed the performance of large-scale contrastive pre-training on a dataset representing a relatively healthy participant population from a geographically distinct continent. We hypothesized that if useful representations are learned by our deep-learning system during the pre-training phase, it should be trivial to separate participants on the basis of features indicative of disease without any additional supervised training. We use our contrastive pre-trained network and freeze the weights to prevent any additional learning, following which we pass each available cine-CMR sequence through the network. The network generates a low-dimensional embedding of each input. We process each view separately and store the embeddings generated for further processing. As with the experiments on the ACDC dataset, we repeat the experiment with an identical network with weights initialized from a training run on the Kinetics-400 action recognition dataset, a video dataset of natural scenery and activities²⁹.

We elected to use the unsupervised t-distributed stochastic neighbour embedding (t-SNE) algorithm to dimensionally reduce the embeddings generated for each view and visualize them in two dimensions³⁰. We expected the networks at baseline to at least be capable of generating embeddings of sufficient quality to separate different views when visualized with t-SNE, as they are obviously distinct even to the untrained eye. In Fig. 2(left), we show the unsupervised clustering of embeddings produced by our contrastive pre-trained network in 2D t-SNE space, for all 45,623 UK BioBank participants. We find that the pre-trained network is essentially view invariant. Key to driving invariance based on view planes was to ensure that all view planes from the same study are aligned with a text embedding from the same text report during pre-training. This allows for attention to be placed instead on features of importance highlighted in the text. We see a relatively dense cluster of patients with ejection fractions <35%. Surprisingly, without any explicit instruction, we find that contrastive pre-training allowed for sharp delineation of sex and separation by age. It is likely that the presence of textual information containing phrases such as ‘adult female’ within some of the MRI reports allowed the vision networks to learn features that identify demographic characteristics. Importantly, all of these findings are seen with the networks entirely frozen, with no additional learning or model parameter updates possible. Comparison with similar experiments using the Kinetics-400 pre-trained baseline is instructive in demonstrating the lack of any such biomedical latent knowledge via traditional training methods (Fig. 2 right). No distinct clusters beyond those of the major MRI view planes studied are seen without our contrastive pre-training methods.

Fig. 2 |. Contrastive pre-training enables learning of demographic and pathophysiological representations.

2D t-SNE plots of low-dimensional embeddings generated from a forward pass on 34,490 unique scans from the UK BioBank (n = 31,693 unique individuals) using a Kinetics-400 pre-trained checkpoint (baseline, right column) compared to a contrastive pre-trained checkpoint (ours, left column). The colour labels help with understanding the basis of clustering. While at first glance it may appear that the Kinetics baseline model produces a set of 5 readily separable clusters vs the more homogeneous-appearing plot for the contrastive pre-trained model, the basis of the baseline Kinetics-initialized clusters are simply the various MRI view planes. On further exploration we find that the Kinetics-400-generated embeddings fail to capture the required information to separate low-ejection fraction states, cardiomyopathy, gender or age (lack of separation of colour for each variable). This is different from the contrastive pre-trained generated embeddings that show clear demarcations by physiologically relevant characteristics, allowing for zero-shot separation of low-ejection fraction states, cardiomyopathy, gender and age. Colour legends for each subplot are shown on the right of each t-SNE figure. View invariance is a built-in feature of the contrastive pre-training process, evidenced by the characteristic absence of clustering by view plane.

Self-attention-based aggregation of embeddings from multiple CMR views

We perform a series of experiments to define the impact of contrastive pre-training when these models are tasked to a variety of clinically relevant regression and classification problems. Each view in a cardiac MRI study may be of varying diagnostic utility, depending on the presenting pathology. To mimic the approach of clinicians reporting cardiac MRI scans, a secondary neural network is used to aggregate useful information from each view plane to generate a final scan level output (Methods). As a result, our models use all available views (4CH, 2CH, 3CH and SAX) while analysing each patient examination. We treat the presence of multiple MRI view planes as a weakly labelled multi-instance learning problem. Videos from each view plane are processed via our pre-trained video transformer encoder, and the resulting embeddings are fed into a self-attention-based multi-instance learning module. The self-attention module is trained to identify relevant features across each available view plane, independent of the actual number of views supplied³¹. The relative contribution of one view over another towards the final output is calculated via a learned weighted average of the output embeddings. This assigns importance to different views depending on the pathological features present in them. This is conceptually identical to assigning higher importance to a selection of zoomed-in patches of histopathology whole-slide images that contain features of malignancy, in the background of a large number of non-diagnostic patches^13,31.

Automated estimation of left-ventricular ejection fraction

We define the impact of contrastive pre-training on the problem of predicting left-ventricular ejection fraction (LVEF), a commonly calculated metric of cardiac function, on two external datasets: the UK BioBank referenced above and a publicly available CMR dataset (Kaggle) for this problem. Deep-learning-derived automated LVEF measurements in echocardiography and cardiac MRI are typically based on segmentation models or hybrid regression-segmentation systems^4,28,32,33. Such networks are trained to outline left ventricular chambers during end systole and end diastole, and as such have no further inherent understanding of the disease space. Nonetheless, these networks perform exceptionally well in the task of LVEF estimation as they replicate the measurement workflow of routine clinical practice. For instance, a mean absolute error (MAE) of 3.2 has been reported for the task of predicting LVEF in a subset of the UK BioBank participants using short-axis sequences³⁴. Two recent papers released while our manuscript was in review are notable for their technical advance in the area of deep-learning-powered LVEF estimation: ref. 35 describes CineMA, a masked-autoencoder system utilizing a hybrid-segmentation supervision; and ref. 36 describes masked autoencoders combined with a contrastive objective. While both achieve competitive performance on LVEF estimation in the UK BioBank (CineMA with a MAE of 3.34% and ref. 36 with a MAE of 2.95%), critically, these systems were both pre-trained and fine tuned on the UK BioBank itself, making direct comparisons of generalizability challenging.

Prospective studies have shown that there is considerable variability in institutional and dataset-specific protocols for calculation of certain metrics, but clinicians can typically be expected to make estimates of LVEF within Bland–Altman limits of −12% to +12%^37,38. With our pre-trained vision encoder frozen, barring the last linear layer, we fine tune the network on 34,488 unique CMR scans from the UK BioBank for the task of predicting LVEF. The vision encoder can no longer learn dataset-specific features with this approach and must rely on learned representational abilities from previous pre-training. To incorporate information from each available view to produce a ‘patient level’ prediction of LVEF, we use a multi-instance self-attention regression head as described above and input cine-CMR sequences from all available views without any additional quality control steps (Methods). On a hold-out test subset of UK BioBank participants using our approach, we report a MAE of 3.344 (s.d. 3.615), with Bland–Altman limits of agreement of −9.91% to +9.61%. At baseline, using a traditionally trained deep-learning system of the same model architecture as our contrastive pre-trained models, we report an MAE of 4.603 (s.d. 4.409), with Bland–Altman limits of agreement of −12.15% to +12.8%.

These metrics rival those of hand-crafted deep-learning systems to calculate LVEF using segmentation masks on manually selected end-systolic and end-diastolic frames, and are well within error limits of clinicians following strict annotation protocols^35,37. We freeze this fine-tuned model and evaluate it on cardiac MRI scans from a publicly available dataset, where patients were recruited from hospital systems based in the United States (Kaggle Data Science Bowl)³⁹. The Kaggle external dataset contains a larger proportion of patients with diseased hearts and additionally utilizes a slightly different scanning protocol and method for determination of ground-truth LVEF labels. On this dataset we report a MAE of 6.880 (s.d. 5.309), with Bland–Altman limits of agreement fo −18.7% to +8.03%. Given the differences in annotation methods compared to data from the UK BioBank, the frozen fine-tuned network showed a modest systematic underprediction of LVEF by 5.36% (95% CI 4.86–5.87) (Fig. 3b). Diagnostic plots for test-set results and additional results with improvements on the Kaggle dataset with bias correction are available in Supplementary Figs. 7 and 8 (final bias-corrected MAE 4.861), showing superior performance in the high and low LVEF range, with smaller residual errors when models are initialized from contrastive pre-trained weights. Finally, we manually review CMR studies for model predictions with the largest absolute errors and find that the majority of these stem from incorrect ground-truth labels or artefact/noise-degraded images (Supplementary Fig. 9 and Table 4).

Fig. 3 |. Clinical-grade LVEF% regression using contrastive pre-trained vision encoder.

a, Dashed horizontal line represents Kinetics-400 baseline performance for the problem of LVEF% regression with 100% of the UK BioBank training data (for fine-tuning and transfer-learning modes). Each point represents validation-set results from a randomly initialized subset (n = 5 independent replicates) of either 1% or 10% of the available training data (x axis). Superior performance (lower is better) is seen with fine tuning, whereas unfreezing all mViT layers (transfer learning) is detrimental to performance. Fine tuning with just 1% of available data yields superior validation results compared to the Kinetics-400 baseline. Box limits represent the interquartile range (IQR), whiskers 1.5× IQR, and the horizontal marker is the median. b, Bland–Altman (BA) plots for test-set results on the UK BioBank and an additional external test set using the Kaggle Dataset. MAE on UK BioBank test set is 3.344 (s.d. 3.615), with BA limits of agreement of −9.91% to +9.61%. Kaggle test results: MAE of 6.880 (s.d. 5.309), with BA limits of agreement of −18.7% to +8.03%. c, Test-set performance of Kinetics-400 baseline performance against contrastive pre-trained models with 100% of training data available for HFrEF (LVEF < 40%) diagnosis based on predicted LVEF%. AUC on the UK BioBank test set is 0.880 (95% CI 0.835–0.925, n = 4,259 scans), and 0.949 (95% CI 0.907–0.990, n = 679 scans) on the Kaggle dataset. Baseline performance on the UK BioBank test set is 0.751 (95% CI 0.692–0.811, Z = 3.93, P = 8.376 × 10⁻⁵) and 0.766 (95% CI 0.697–0.836, Z = 4.08, P = 4.325 × 10⁻⁵) for the Kaggle dataset. Grey-shaded regions represent 95% confidence intervals at different classifier thresholds. Statistical testing performed using two-sided paired DeLong’s tests without multiple-comparison adjustments. d, Relationship of downstream performance with quality of contrastive pre-training. Each point represents a pre-trained model checkpoint saved every 10 epochs of pre-training. Left to right: non-monotonic decrease in loss as pre-training continues; progressive improvement in downstream performance on LVEF% regression task validation MAE; similar improvement in validation MSE as a function of decreasing pre-training loss. For this experiment, fine tuning was performed using 10% of available UK BioBank data.

Nevertheless, if these predicted LVEF values were to be used for identifying patients with heart failure with reduced ejection fraction (HFrEF) < 40%⁴⁰, the area under the curve (AUC) on the UK BioBank test set would be 0.880 (95%CI 0.835–0.925, n = 4,259 scans), and 0.949 (95% CI 0.907–0.990, n = 679 scans) on the external Kaggle dataset (Fig. 3c). This is significantly more performant compared with our baseline, where we fine tune the exact same network architecture from a Kinetics-400 checkpoint. Baseline performance on the UK BioBank test set would be 0.751 (95% CI 0.692–0.811, P < 0.001), and 0.766 (95% CI 0.697–0.836, P < 0.001) for the Kaggle dataset (Supplementary Table 5). This is in line with recent reports of poor generalizability with models trained via more traditional methods on the UK BioBank³⁷. We find that using our contrastive pre-trained vision encoders allow for impressively low error rates when fine tuning on just 1% of the data available in the UK BioBank dataset (344 scans), far surpassing our baselines (Fig. 3a and Supplementary Table 3). We also find that unfreezing the vision encoder, allowing for parameters to be overwritten (transfer learning) with new learned weights led to a marked decline in performance across the board. To further explore the relationship of pre-training quality on downstream performance on the task of LVEF prediction, we fine tune models from checkpoints taken at equally spaced intervals during the pre-training process. We find that downstream validation performance continues to increase with decreasing pre-training loss in a non-monotonic fashion (Fig. 3d). Finally, we studied the scan–rescan variability of LVEF predictions generated by our models for participants with more than one available CMR study acquired at two distinct time periods (n = 311, test set). The mean variance was found to be 5.98% (s.d. 1.53%), with Bland–Altman limits of agreement between scans of −6 to +6%. These figures exceed previous benchmarks of clinical expert-level performance in prospective trials (Supplementary Table 9 and Fig. 14)³⁷.

Efficiently diagnosing disease from cardiac MRI

Cardiac MRI is routinely used to assess patients with left ventricular dysfunction, congenital heart disease, valvular disease and various cardiomyopathies⁴¹. While morphological features seen on cine sequences may be sufficient to diagnose diseases such as hypertrophic cardiomyopathy, others such as cardiac amyloid or the presence of an intracardiac thrombus typically require contrast-enhanced sequences. Previous work in automated deep-learning-based disease diagnosis has relied on segmentation of cardiac chambers or specific patterns of scar, using limited datasets that lacked the comprehensive inclusion of a wide range of cardiac conditions^42,43.

We created a labelled dataset of 4,301 unique patients from Stanford, Medstar and UCSF distinct from the pre-training dataset for 39 cardiovascular conditions using reports generated by expert radiologists as part of routine clinical practice. We intentionally included disease conditions that typically require the evaluation of contrast-enhanced images or T1/T2 sequences for accurate clinical diagnosis. Furthermore, we included certain labels that are more subjective in nature (for example, ‘severe’ valvular disease). These datasets were unenriched, uncurated and reflect real-world clinical prevalence at the various clinical sites (Supplementary Figs. 10 and 11).

With our pre-trained vision encoder frozen, barring the last linear layer, we fine tune the network to predict the binary classification label for each disease task on a subset of 2,145 unique patients (2,414 scans). We generate a ‘patient level’ prediction using a multi-instance self-attention classification head and input cine-CMR sequences from all available views and slices without additional quality control steps (Methods). Models were validated and tested on separate splits of 1,073 and 1,083 patients, respectively. We show significant absolute improvements in the area under the receiver operator curve (AUROC) across most clinical tasks when fine tuning from our pre-trained models, compared to baselines fine tuned from a Kinetics-400 pre-trained checkpoint (Fig. 4a, and Supplementary Tables 6 and 7 and Fig. 12). Furthermore, attention maps plotted from the multi-instance self-attention module show that embeddings from different views are preferentially weighted depending on the clinical task at hand (Fig. 4c).

Fig. 4 |. Data-efficient disease diagnosis from cine sequences alone.

a, Test-set AUCs on the internal dataset for contrastive pre-trained (magenta) and Kinetics-400 baseline (light pink) checkpoints for models fine tuned for the binary classification problem of disease diagnosis. Disease labels are placed along the x axis, with test AUC on the y axis. The dashed grey line marks an AUC of 0.5, the performance expected from a random classifier. Solid grey connecting lines mark the performance difference between contrastive pre-trained and baseline models. Remarkable classification performance is observed for conditions such as hypertrophic cardiomyopathy, amyloidosis, systolic anterior motion of mitral valve and dilated cardiomyopathy, whereas models somewhat struggle for conditions that typically require gadolinium contrast or additional scanning sequences for diagnosis, such as differentiating cardiac masses from cardiac thrombus. The numbers of patients with disease for each disease label are listed along the x axis (n = positive classes), in black text within grey-outlined box. b, Test-set results of the contrastive pre-trained (navy blue) and Kinetics-400 baseline (light blue) models on the UPenn dataset with no additional fine tuning. c, Relative self-attention values calculated by the multi-instance self-attention classifier heads. Heat maps generated for different views (2CH, 3CH, 4CH, SAX) for each disease condition, plotted as a function of the presence or absence of the disease in question. Subjectively, we find that the networks offer more attention to certain views over others depending on the disease condition to make accurate predictions. This is additionally consistent with internal testing that showed performance degradation when certain slices or views are hidden from the networks.

External validation on the Penn cardiac MRI dataset

We follow a similar data labelling technique as described above for CMR scans from the University of Pennsylvania. Final model checkpoints for each disease task from the experiments above were frozen and subsequently used for testing on the Penn dataset. No additional fine tuning was performed on these data. Testing was performed in series for each disease label on either a single Nvidia A40 GPU or Nvidia B200 GPU, taking ~10 min each, for a total of 2,070 scans from 2,033 unique patients. As before, all available view planes (2CH, 3CH, 4CH, SAX slices) were used with no additional quality control or filtering steps. On this external test dataset, we show robust performance for disease conditions such as amyloidosis, hypertrophic cardiomyopathy and tetralogy, but significant degradation for labels such as ventricular septal defects and sarcoidosis. AUROC curves and confidence intervals are shown in Fig. 4b, and Supplementary Table 8 and Fig. 13.

On review of institutional CMR sequence protocols, a possible contributing factor for our findings is that at the University of Pennsylvania, the standard practice is to administer contrast before the acquisition of all cine sequences—a technique employed to reduce scan times. This was not the case at Stanford, MedStar and UCSF where cine sequences are acquired before contrast is administered. This represents a dramatic shift in the underlying pixel-data distribution. Despite these differences in image acquisition protocols, the robust performance on several tasks suggests that our models rely on structural and dynamic-motion features to identify certain disease states, rather than patterns of pixel intensities alone that are not robust to timing of contrast.

Diagnostic performance strengths and limitations

Despite the absence of contrast-enhanced images in the inputs, our systems excelled at the detection of several cardiomyopathies on the test sets: internal test AUCs for amyloidosis was 0.921 (95% CI 0.879–0.963), 0.91 (95% CI 0.872–0.948) for hypertrophic cardiomyopathy, and 0.867 (95% CI 0.843–0.890) for dilated cardiomyopathy. However, performance on tasks involving the detection of intracardiac thrombi (AUC 0.744, 95% CI 0.653–0.836) or certain cardiac tumours (AUC 0.649, 95% CI 0.535–0.764) remained marginal. Differentiating tumour from thrombus, or diagnosing myocarditis can be particularly challenging for clinicians without contrast enhancement or T2 imaging^44–46. The Lake Louise imaging criteria used for the diagnosis of myocarditis in clinical practice, for instance, relies on T1, T2 or late gadolinium-enhanced (LGE) sequences⁴⁷. Our results indicate that cine sequences alone are unlikely to provide enough signal to reliably diagnose such conditions despite the alignment with textual information in the reports. Similarly, the diagnosis of arrhythmogenic right ventricular cardiomyopathy (ARVC) relies on a Clinical Task Force Criteria scoring system: a combination of clinical symptoms, medical and family history, EKG findings and imaging findings⁴⁸. Our model performance returns an AUC of 0.652 (95% CI 0.487–0.818) for ARVC, indicating the challenges of diagnosing conditions that in routine clinical practice rely on a comprehensive evaluation of the patient beyond what images alone can provide.

Valvular heart disease was similarly challenging to accurately grade for our models, although this is probably the result of poorly defined severity labels in the MRI reports themselves. Severe mitral regurgitation (AUC 0.753, 95% CI 0.716–0.79) and severe aortic stenosis (AUC 0.706, 95% CI 0.622–0.79) are characterized by turbulent regurgitation jets and high-velocity stenotic jets, respectively. Detecting bicuspid valves on the other hand requires acquisition of images in the plane of the aortic valve to truly define the location and orientation of fused aortic valve leaflets. While these features are sometimes visible on cine sequences, accurately quantifying regurgitant volumes and the severity of valvular heart disease requires additional post processing of phase-contrast sequences in specific anatomical planes or 4D-flow scans that are seldom performed as part of standard CMR scanning protocols⁴⁹.

Our models performed well in detecting anomalous anatomical configurations such as tetralogy of fallot (AUC 0.966, 95% CI 0.955–0.977), ventricular septal defects (AUC 0.911, 95% CI 0.884–0.938) and changes in ventricular mass. The performance across tasks appears unrelated to the underlying prevalence of the disease labels themselves, with excellent performance for diseases such as amyloid (prevalence ~1%) and worse performance for those such as right atrial dilation (prevalence 13%). Results are summarized in Supplementary Table 6 and Fig. 4. In clinical practice, cardiac MRI is often used to establish differential diagnoses, for example, interrogating the underlying aetiologies of individuals presenting with heart failure. To illustrate this, we analyse a subset of patients with the clinical label of ‘non-ischaemic cardiomyopathy’ (n = 242) in our internal test dataset. This broadly includes cardiomyopathies associated with infiltrative diseases, genetic, hypertrophic or dilated cardiomyopathies and myocarditis among others. Such subgroup analyses demonstrate conditional model performance in clinically challenging contexts; for instance, certain non-ischaemic cardiomyopathies share overlapping phenotypic features of left-ventricular hypertrophy (sarcoidosis, amyloidosis and hypertrophic cardiomyopathy). This makes discerning the underlying aetiology challenging for models initially designed to detect disease in the background of phenotypically very distinct negative classes. Nonetheless, our models demonstrate impressive performance in this subset for amyloidosis (AUC 0.870, 95% CI 0.675–0.957), dilated cardiomyopathy (AUC 0.879, 95% CI 0.836–0.923) and hypertrophic cardiomyopathy (AUC 0.925, 95% CI 0.873–0.977) (Supplementary Table 18).

Mirroring findings on the UK Biobank LVEF% regression task, there is significant performance degradation when attempting to transfer learn with all vision encoder parameters unfrozen, suggesting the fundamental importance of parameters learned via contrastive pre-training. Finally, we find, on scan–rescan testing, that the variance between predicted probabilities of patients with more than one scan is on average less than 3% in both the internal test set and the external UPenn test datasets (Supplementary Tables 10 and 11).

Clinical applications and future directions

The core vision system remains unchanged in our proposed framework: we pre-train one vision network capable of contextualizing and representing the features important for multiple actionable downstream clinical tasks of interest. Our principal contribution is the development of a system that is readily adaptable, with minimal additional fine-tuning data, to a wide range of clinical tasks. As a result, there are numerous immediate clinical applications. We anticipate our models to enable rapid and expert-level diagnosis of relatively rare complex cardiomyopathies, democratizing expertise beyond the confines of large academic centres of excellence. This is particularly relevant for conditions such as hypertrophic cardiomyopathy and cardiac amyloidosis that are often misdiagnosed or underdiagnosed in the community^50,51. Our models may serve as a specialist-level triage, opportunistically labelling patients with a high predicted probability of having disease as soon as scans are completed for physicians to review. At a systems level, our models can help automatically triage and notify clinical services, ensuring flagged patients are formally evaluated by subspeciality cardiology teams. This is especially important for cardiomyopathies that can be effectively managed with novel pharmacological agents with significant reduction in mortality and morbidity^52,53.

Our methods, furthermore, make possible for deep-learning systems to be adapted towards identifying imaging features that are predictive of a histopathological ground truth, narrowing the modality gap between radiological and histopathological diagnoses for diseases such as amyloidosis, sarcoidosis, post heart transplant rejection and cardiac malignancies. This has the potential of reducing our reliance on invasive testing for obtaining actionable diagnostic information, dramatically reducing the risk of iatrogenic injury⁵⁴. We envision that our models could be deployed as a clinical decision aide, potentially allowing us to reserve invasive endomyocardial biopsies for patients who have indeterminate results via our models.

We recently presented work on our pre-trained models in screening the UK BioBank population for hypertrophic cardiomyopathy⁵⁵. CMR studies acquired as part of the UK BioBank are usually not reviewed by an imaging cardiologist unless glaring abnormalities are noted, probably contributing to the 5–10× lower reported prevalence compared with the general population⁵⁶. Given the overwhelming majority of normal studies, cases exceeding a probability threshold of 95%—correlating with the top 0.5% of cases (expected specificity of 97% and sensitivity of 60%)—were screened for manual reading. With this approach, our group screened through over 40,000 CMR scans and identified 200 scans for manual physician read over. From this high-probability subpopulation, we confirmed 112 new individuals with imaging features of hypertrophic cardiomyopathy. Our models thus have substantial potential for clinical expert-grade phenotypic refinement of large research datasets.

Traditional uses for deep-learning systems for CMR are in the automation of otherwise hand-measured metrics such as LVEF, chamber volumes or the estimation of myocardial thickness. Our approach is to use the same pre-trained vision system, obviating the need for labour-intensive data annotation for each new structure or target of interest. While segmentation-based tools are able to achieve similar results, they are critically limited in needing ground-up retraining for every new metric of interest, with considerable human effort required for each new task. Our modular approach of swapping regression or classifier heads depending on the task of interest allows for rapid repurposing of representations learned by our core vision encoder.

Finally, we show that our models demonstrate intriguing emergent properties out of the box, including the ability to separate sex, age and certain disease phenotypes without additional fine tuning or instruction. While traditional approaches are limited to standard structural measurements of the heart, these properties of our models make them powerful tools for efforts directed at phenotypic refinement or genetic discovery in cardiovascular disease in large population-scale data repositories^57–62.

Discussion

The methodology for our foundational deep-learning CMR model incorporates contrastive pre-training over a joint embedding space. Visual features are extracted from scans, and text features are extracted from reports generated by clinical experts as part of routine care. Unlike previous work on chest X-rays where an image is taken from a largely consistent anatomical view, for CMR studies, the inputs are a series of videos acquired along multiple distinct anatomical view planes. Even within a particular view plane, there may be multiple unique parallel videos acquired (for example, cine-bSSFP sequences at the base, mid-papillary and apical cross-sections of the heart in the SAX view). The contrastive pre-training routine was thus designed to maximize agreement with shared text embeddings across all available views present within a single study. We hypothesize that this solves two interdependent and critical problems: First, that traditional supervised learning methods often exacerbate the problem of ‘shortcut learning’, wherein deep-learning systems approximate simple decision rules that allow for networks to perform well for a narrow task⁹. Second, the parameters learned for a specific and narrow clinical problem rarely transfer to new tasks. This demands time and labour-intensive labelling of large amounts of data for each clinical task of interest.

By pre-training and subsequently fine tuning towards specific problems, we show that with remarkable consistency, our networks perform superiorly to baseline methods. We show that across numerous unrelated tasks, superior performance can be achieved with up to two orders of magnitude less data. This data efficiency is a critical advance for the development of fine-tuned models for the diagnosis and characterization of complex inherited cardiomyopathies, where expert adjudicated registries with paired imaging data remain scarce^63,64.

While pre-training itself is computationally expensive, fine tuning on each task can be achieved within a few hours on consumer grade GPUs. In inference mode with no additional hardware-specific optimizations, our models can process an entire CMR study with multiple views, sequences and anatomical slices in under 400 ms. With additional optimization and quantization, these models can readily run on resource-efficient embedded systems. This allows for clinical deployment strategies ranging from local on-device inference to light-weight cloud-containerized instances that interface with hospital Picture Archiving and Communication System (PACS) servers⁶⁵. Furthermore, the pre-training framework can be extended to data sources beyond linked CMR reports, including text reports from other imaging modalities (for example, echocardiography, cardiac catheterization reports) or histopathological slide features that may further expand the emergent capabilities of vision encoders seen in this work.

Large language models have grown immensely in capabilities and in size. Our implementation of BERT has 110 million trainable parameters and a just over 30,000-token vocabulary, whereas contemporary large language models boast up 80 billion parameters and over a trillion tokens. Contrastive pre-training makes use of embeddings produced by the text encoder (a feature of architectures such as BERT). The ‘decoder-only’ architectures of most contemporary large language models preclude direct extraction of embeddings needed to train on a joint-representation space. Recent advances offer alternatives but are too computationally expensive to be viable in their current forms⁶⁶.

Another limitation of our current work is the reliance on cine-SSFP sequences alone. While the reports describe findings from gadolinium-enhanced imaging, the networks do not make use of LGE sequences as inputs. This is largely because LGE sequences are often captured as images rather than dynamic-motion videos, necessitating a separate and parallel image-based neural network to be built into the current framework. Future research will incorporate non-standard view planes along with T1, T2, LGE and perfusion scans via separate image encoders or dimensionally agnostic vision encoders capable of handling both image and video data. Despite this, our models consistently perform well on tasks with just cine-SSFP sequences in situations where clinicians typically require additional imaging data. LGE, for instance, is routinely used by clinicians for diagnosing amyloidosis; similarly, LGE along with T1 and extracellular volume maps are often used in diagnosing hypertrophic cardiomyopathy^67–69.

We hope our work accelerates research in this field, making it feasible to apply deep-learning techniques to clinical areas of interest on datasets traditionally deemed too small or niche. Fine tuning on smaller, expert-labelled datasets where ground truth is established via either expert consensus or histopathological reads of myocardial biopsies is an exciting avenue of future research. Keeping this in mind, our pre-trained checkpoints have been made freely available for academic use. Our results are an important step in the evolution of deep learning for cardiac MRI. In summary, we describe a generalizable self-supervised deep-learning system for cardiac MRI capable of representing the breadth of human cardiovascular disease. This work lays the foundation for prospective clinical-grade applications in disease diagnosis, with immediate research applications in phenotyping and genetic discovery of cardiovascular disease using cardiac MRI.

Methods

Ethics statement

Data collection and research began following approval and waiver of consent by the Institutional Review Board of Stanford University (Protocol #60342, March 2021). Additional data from the University of Pennsylvania were sourced after retrospective collection was deemed IRB exempt by the University of Pennsylvania Health System (Protocol #852332, November 2022).

Computational hardware and software

MRI DICOM data were pre-processed on siloed HIPAA-certified n2 instances on the Stanford Nero–Google Cloud platform. Specifically, we used an 8-core virtual machine with 52 GB of memory and 6 TB of attached solid state storage. Data from the UK BioBank were pre-processed on the Stanford Sherlock High Performance Computing Cluster, using 24 CPU cores (Intel Xeon Gold 5118, 2.30 GHz). Anonymized reports were tokenized on a local encrypted desktop using 48 CPU cores (AMD Threadripper, Lambda Computers). All models were trained on the Stanford Sherlock High Performance Computing Cluster using servers with 4× Nvidia A100 GPUs, each with either 40 GB or 80 GB VRAM, and 64 CPU cores (AMD Epyc). External validation on data from the University of Pennsylvania was performed on the Penn CUBIC High Performance Computing Cluster on a single Nvidia A40 GPU with 10 CPU cores. Additional external tests took place on the Penn Advanced Research Computing Center (PARCC) Betty cluster, on a single Nvidia Blackwell B200 GPU with 10 CPU cores. Hyperparameter optimization experiments were run on servers with a variety of GPU resources (Nvidia V100, 32 GB VRAM; Nvidia H100, 80 GB VRAM; Nvidia A100, 40 GB/80 GB VRAM; Nvidia P100, 32 GB VRAM; Nvidia Blackwell RTX 6000 MaxQ, 96 GB VRAM). We used the PyTorch deep-learning library (v.1.11.0) and the pytorch lightning framework (v.1.8.6)⁷⁰. Major Python packages used in this work include numpy (v.1.21.2), pydicom (v.2.0.0), transformers (v.4.4.2) and stanza (v.1.5.0).

Datasets

Specifics of the pre-processing pipelines for both the MRI scans and the free-text reports are detailed in the ‘Dataset pre-processing’ section of Supplementary Information. Briefly, from each unique MRI study, relevant scans were extracted (4CH, 3CH, 2CH and SAX cine sequences) as 4D arrays and stored within a single hdf5 file. Free texts from the reports were segmented into individual sentences using the stanza natural language processing pipeline, tokenized using the standard BERT auto-tokenizer, and the resulting anonymized numeric arrays were stored in a single indexed json file^27,71. Across the pre-training datasets, fine-tuning datasets, external test datasets and the UK BioBank, we included 65,492 individuals with ~550,156 unique videos across different view planes and cross-sections.

Clinical CMR dataset.

The total clinical CMR dataset comprised 19,122 unique individuals. Cardiac MRI scans were sourced from 17,088 individual patients from a consortium of academic hospital systems based in the United States (Stanford Healthcare, UCSF, Medstar). Cine MRI scans were procured via Bunkerhill Health (San Francisco, CA) as de-identified DICOM files, and associated radiology reports were sourced as a single csv file (IRB Protocol #60342, March 2021). The total pre-training dataset consisted of 293,110 unique 4CH, 3CH, 2CH and SAX videos. The scans were performed as part of routine clinical practice and reports were generated by board-certified physicians with specific expertise in cardiac MRI. Sequences were acquired on a wide range of scanners including those manufactured by Siemens (Siemens Healthcare), General Electric (GE) and Philips (Philips Healthcare), resulting in substantial variance in the number of frames per slice, imaging resolution and reconstruction techniques (Supplementary Table 2). Demographics wherever feasible are detailed in Supplementary Table 1. The data were first separated into pre-training and downstream datasets in an approximate 75:25 split at the patient level. For the pre-training split, we further divided the data into a training and validation set with an approximate 66:33 split. Similarly, for downstream split (intended to be used as a labelled fine-tuning dataset for clinical tasks of interest) we further divided the data into training, validation and testing datasets with an approximate 50:25:25 split. We did not selectively exclude patients from this dataset; however, a fraction of the dicom files were received as duplicates or were corrupted and were subsequently discarded. Supplementary Fig. 1 details the data splits and enumerates the excluded studies at each stage. Cardiac MRI scans from an additional 2,033 individual patients were secured from the University of Pennsylvania Health System (IRB exempt, Protocol #852332, November 2022). These scans were performed as part of routine clinical practice and acquired on scanners manufactured by Siemens and GE. Data from the University of Pennsylvania were used solely for external testing. While rule-based automated data labelling techniques have been used in the past, these have been superseded by large language models⁷². Building on our previous work in exploring the zero-shot capabilities of large language models for medical text, we utilized a publicly available large language model (medgemma3, 27-billion parameter variant) to parse free-text reports generated as part of routine clinical practice into pre-defined ‘disease labels’ for the disease diagnosis tasks^73,74. Specific prompts, parameters, performance comparisons vs human annotators, and a selection of random non-curated reports with critique of the deep-learning-predicted labels are detailed in Supplementary Fig. 5.

UK BioBank cardiac MRI cohort.

Cine bSSFP-cardiac MRI sequences from 45,623 participants were sourced from the UK BioBank (Project ID: 71226). SAX sequences were available for 11,005 participants and contain stacks of 8–10 individual slices. One slice was available for each of the 4CH, 3CH and 2CH scans. This amounted to a total of 257,046 unique videos available for analysis. Sequences in the UK BioBank were acquired on a clinical 1.5 Tesla scanner using a standardized protocol (MAGNETOM Aera, Syngo Platform VD13A, Siemens Healthcare)⁵⁶. As part of this protocol, the vast majority of ventricular volumes and functional metrics were calculated via automated contouring of the ventricular endocardium and epicardium without manual expert quality controls^56,75. For fine-tuning and transfer-learning experiments to estimate LVEF%, we split the UK BioBank dataset into an approximate 80:10:10 split at the participant level into training (n = 31,693), validation (n = 3,938) and hold-out test datasets (n = 3,938).

ACDC dataset.

The ACDC dataset is a publicly available cardiac MRI dataset of 100 patients from the University Hospital of Dijon, France²⁸. Each SAX sequence was paired with patient-level non-overlapping labels (n = 20 each) for hypertrophic cardiomyopathy, previous myocardial infarction, dilated cardiomyopathy, abnormal right ventricles and normal controls. The scans were acquired on either a 1.5 Tesla (Siemens Area, Siemens Healthcare) or 3.0 Tesla (Siemens Trio Tim, Siemens Healthcare) scanner with a conventional SSFP sequence in breath hold and gating.

Kaggle Data Challenge dataset.

The 2015 Kaggle Data Science Bowl released data from 700 patients compiled by the National Institutes of Health and the Children’s National Medical Center, and was at the time, an order of magnitude larger than any cardiac MRI dataset previously described. Patients were recruited from the United States and scans were performed in the Washington DC area. While demographic splits from the dataset are not available, the original data were sourced from multiple hospital systems across a range of age groups containing both normal and diseased hearts. The competition closed on 14 March 2016, but data from 697 cases remain publicly available in DICOM format³⁹. 2CH, 4CH and SAX cine sequences were available for use, along with expert annotations for left ventricular end-systolic and end-diastolic volumes. The entirety of the available dataset was used for external validation as is, without any quality control.

Neural network architectures

We tested vision encoder architectures including 3D residual convolutional networks and video vision transformers. We settled on using an implementation of a multiscale vision transformer (mViT) with 36.3 million trainable parameters as our video encoder after experiments showing superior generalization and embedding quality²⁶. Vision transformers have recently emerged as a performant alternative to convolutional neural networks, especially in the setting of large-scale self-supervised pre-training^76,77. Vision transformers retain the skip connections seen in traditional convolutional networks, but are also able to attend to local and global features of an image in earlier stages⁷⁸. The mViT architecture is a vision transformer designed specifically for video data, which foregoes the successive layers of convolutional operations seen in typical convolutional neural networks, for a single convolutional layer to divide the input video into a linear series of overlapping cubes. These linear elements are processed by 16 layers of stacked transformer modules, allowing the network to effectively attend to distant input features. Specific to the mViT architecture is a sequential series of pooling and scaling operations that effectively enable the network to attend to simple visual features at high resolution in early layers, followed by complex high-dimensional relationships at a coarser resolution in deeper layers. As a result, compared with other extensions of 2D-image transformers to the video domain, mViT by design has a stronger temporal inductive bias. While more computationally expensive than comparable convolutional networks, mViT is more efficient than comparable vision transformers, requiring remarkably less pre-training data to achieve state-of-the-art results on typical action recognition datasets. Finally, compared to traditional convolutional neural networks, mViT has shown superior performance on large video action recognition datasets despite fewer trainable parameters²⁶.

We elected to use a pre-trained BERT model for our text encoder²⁷. Unlike other language models that have come before it, BERT is trained using a ‘bidirectional’ approach, where the model is trained to learn the structure and context of human language by attending to sentences in both the left-to-right and right-to-left direction. Specific details of the pre-training methods for BERT are detailed in the original paper²⁷. We used a 12-layer variant of BERT base, with 12 attention heads and a hidden dimension of size 786, with a total of 110 million trainable parameters. We tested a combination of different pre-trained weights including those from the original publication, weights fine tuned on the MIMIC dataset, and weights from a model trained on biomedical abstracts from PubMed with a custom vocabulary of 30,522 tokens^79,80 (Supplementary Fig. 3).

Pre-training framework

We built on previous attempts at learning visual representations using naturally occurring pairing of 2D medical imaging and textual data, extending these concepts to the spatiotemporal video-like nature of cardiac MRI scans^{14–17,19,20}. Two parallel encoders were trained: one for processing the MRI cine sequences and the other for processing the subsampled text from paired radiology reports. Self-supervised transformer networks in particular have shown superior performance on downstream tasks compared to traditional supervised techniques^76,81,82. We used an implementation of mViT with Kinetics-400-initialized weights for the vision encoder, and a pre-trained BERT model for the text report encoder. Specifically, we utilized weights from BERT pre-trained on abstracts of biomedical publications on PubMed with a custom vocabulary⁷⁹. Data from 8,513 patients (9,427 scans and paired reports) were used for training, and a separate set from 4,194 patients (4,646 scans and paired reports) were used for validation.

We employed randomized sequential data augmentation schemes (AugMix) to stochastically sample and layer a series of chained transformations including but not limited to resizing, solarization, shear, translate and random rotation of videos in the spatial dimensions, all while preserving the same augmentations along the temporal dimension for temporal consistency⁸³. Uniform temporal subsampling greatly improved downstream performance and generalizability. We augmented the radiology reports by randomly sampling five sentences from the entire report for each scan per training step. The output of each encoder was passed through a one-layer linear projection head to yield a pair of 512-dimensional embeddings. These low-dimensional, 512-dimensional embeddings are a compressed numeric representation of the information contained within the input MRI scan and paired text report.

Previous work has also shown the importance of large batch sizes for effective contrastive representation learning⁸¹. To study this, we pre-trained models with a batch size of 16, 32 and 128 video–text pairs. For the UK BioBank LVEF prediction task, we found that fine tuning from the larger-batch-size pre-trained models led to improved downstream results (Supplementary Fig. 2). While computational budgets did not allow for an extensive hyperparameter search with the larger batch sizes, we note that the downstream benefits did not appear to be clinically significant for this specific task. Nonetheless, this remains an area for additional future exploration.

Vision-only self-supervised methods would be challenging to incorporate where scans from multiple visually distinct view planes exist for the same patient. We focused our efforts on text-to-video approaches given the success with text supervised visual representation learning across radiology and action recognition^14,16,84,85. We considered approaches such as Contrastive Language-Image Pre-Training (CLIP); however, these are limited by a short context length suitable for captions rather than the larger, mostly unstructured paragraphs that are typical of cardiac MRI reports⁸⁵. Similar to the work of ref. 16, we elected to use an asymmetric bidirectional implementation of the InfoNCE loss to maximize mutual information between each MRI video–text report pair^16,22. The contrastive losses used are essentially log-loss of an n-way classifier to predict the correct pair of MRI scan and report (where n = batch size). The first loss function is a video-to-text contrastive loss for the ith pair, where v_i represents a video embedding and u_i represents a text embedding of the ith video–text pair. N here represents the number of video–text pairs in a total batch being evaluated.

$$l_i^{\left(v\to u\right)}=-\text{log}\frac{\text{exp}\left(〈v_i,u_i〉/\tau \right)}{{\sum }_{k=1}^N\text{exp}\left(〈v_i,u_k〉/\tau \right)}$$ (1)

The second loss function is a similarly structured text-to-video contrastive loss. The tunable temperature parameter (τ) controls the strength of penalties on hard negative pairs sampled during training⁸⁶.

The final loss was defined as a weighted combination of the two losses averaged over all positive video–text pairs in each batch of data. The scalar weight is given by λ.

$$ℒ=\frac{1}{N}{\sum }_{i=1}^N\left(\lambda l_i^{(\mathbf{v}\to \mathbf{u})}+(1-\lambda )l_i^{(\mathbf{u}\to \mathbf{v})}\right)$$ (2)

We additionally implemented a ‘flooding’ regularization technique to prevent the training loss (ℒ) to approach zero⁸⁷. We set the flood level (scalar value given by b) to a training loss of 0.05 to allow for better generalization. The final loss ($\widetilde{ℒ}$) is thus given by:

$$\widetilde{ℒ}=\left|ℒ-b\right|+b$$ (3)

The specific pre-trained weights and vocabulary used for initializing the text encoder, batch size, augmentation scheme, InfoNCE temperature parameter and flood regularization were critical for model convergence⁸⁸. The final model was pre-trained with a batch size of 32 per GPU, for 600 epochs. The first 6 layers of the BERT text encoder was frozen, and the entire network was trained with a learning rate of 4.8 × 10⁻⁵ using the AdamW optimizer with weight decay set to 1 × 10⁻⁶ and eps set to 1 × 10⁻⁸. We decayed the learning rate by a factor of 0.1 at 300 epochs. Checkpoints were saved every 10 epochs during the pre-training process and the last checkpoint was used for fine tuning on downstream clinical tasks. The total time taken for pre-training was 13 days and 14 h (4 × 80 GB Nvidia A100 GPUs). The ability of the vision transformer encoder to cluster different disease conditions without any additional explicit supervised training was visualized using the uniform manifold approximation and projection (UMAP) algorithm initialized using default values⁸⁹.

Multi-instance self-attention and downstream evaluation

A gated multiview self-attention network was trained to assign an attention value (a_k) to each MRI view embedding produced by the main vision encoder^13,31. For each embedding within a bag of k embeddings, a high score after softmax activation (near 1) indicates that a particular MRI view plane is highly informative for the downstream diagnostic task. Conversely, a low score (near 0) indicates that the MRI view plane has little to no diagnostic value. For classification tasks, each input embedding was additionally passed through a LayerNorm function before a forward pass into the self-attention blocks (Supplementary Fig. 6)⁹⁰ (w^T, attention scoring vector; V, view level weight parameters; U, view level weight parameters; h_j, low-dimensional embeddings; ⊙, element-wise product; tanh, tanH activation function; sigm, sigmoid activation function; N, total number of MRI view embeddings for a particular study).

$$a_k=\frac{\text{exp}\left\{{\mathbf{w}}^{\top }\left(\text{tanh}\left(\mathbf{V}{\mathbf{h}}_k^{\top }\right)\odot \text{sigm}\left(\mathbf{U}{\mathbf{h}}_k^{\top }\right)\right)\right\}}{\sum _{j=1}^k\text{exp}\left\{{\mathbf{w}}^{\top }\left(\text{tanh}\left(\mathbf{V}{\mathbf{h}}_j^{\top }\right)\odot \text{sigm}\left(\mathbf{U}{\mathbf{h}}_j^{\top }\right)\right)\right\}}$$ (4)

We made use of an attention pooling mechanism to average the embeddings from all MRI views weighted by their predicted attention scores, to return a single 512-dimensional embedding. This embedding can be treated as a ‘feature representation’ of the entire MRI study for a specific downstream task of interest. For each downstream classification task of interest, we used a binary classification head with a sigmoid activation function, as disease labels are usually not mutually exclusive in the setting of cardiovascular disorders. For downstream tasks that involve regression of a numeric variable, we replaced the binary classification head with a single output neuron with a linear activation function.

LVEF regression task

We examined two modes of training for LVEF% prediction: (1) ‘fine tuning’ where the last linear layer of the vision encoder and the classifier head are trainable and (2) ‘transfer learning’ where the vision encoder, linear layer and classifier heads are all trainable. ‘Fine tuning’ allows for some degree of flexibility in the way embeddings are generated but keeps the vision encoder frozen to make use of the learned representations. With the system set to ‘transfer learning’, the network begins from the learned representations; however, since the entire network is unfrozen, it is possible to ‘overwrite’ these parameters with each new update of the training process. For these experiments, we initialized the vision encoder with the contrastive pre-trained weights (ours) or Kinetics-400 weights (baseline), onto which we attached the regression head as described above.

We fine tuned our pre-trained checkpoints with 32-bit precision using the AdamW optimizer, with a learning rate set to 1 × 10⁻⁴ and default value of 0.01 for weight decay. We explored different augmentation schemes and achieved superior validation performance with AugMix on restricted hyperparameter sweeps with 10% of the training data⁸³. For all experiments involving fine tuning with subsets of available data, we used a manual seed value for random subsampling to ensure reproducibility of results. We made use of all available 4CH, 2CH, 3CH and a random subsample of 50% of SAX views per study, with no manual screening for quality control. We elected to train our regression models with a Huber loss function, and we used mean squared errors and mean absolute errors as performance metrics⁹¹. We additionally calculated the AUROC for diagnosing heart failure on the basis of an LVEF cut-off of 40%. We trained models for a maximum of 100,000 steps on GPUs with at least 16 GB of VRAM each. For experiments described in Fig. 3a,d, configuration files were generated for each experimental setup and were trained in parallel across numerous GPUs on Stanford Sherlock.

Disease classification task

We define both ‘fine tuning’ and ‘transfer learning’ as above, and used the same network architecture initialized with Kinetics-400 weights as our baseline. We fine tuned our pre-trained checkpoints with the same overall settings as described above for the regression tasks, except for the use of a weight decay value of 5 × 10⁻⁴ and the addition of a LayerNorm function for the embeddings before a forward pass through the multi-instance self-attention modules to aid with convergence. We empirically used AugMix for our data augmentation strategy, given the successes noted above. We made use of all available 4CH, 2CH, 3CH and SAX views per study with no quality control or screening. We utilized a binary cross-entropy loss function with a sigmoid activation weighted by a scalar multiplier equal to the proportion of positive vs negative classes for each disease (calculated using the internal training set prevalences). We used the AUROC as a performance metric, given the considerable class imbalance of positive and negative classes⁹². For each disease label of interest, we trained models for 24 epochs on GPUs with at least 24 GB of VRAM. For experiments described in Fig. 4a,b, configuration files were generated for each experimental setup and were trained in parallel across numerous GPUs on Stanford Sherlock. External test data were evaluated on the Penn CBICA cluster on a single Nvidia A40 GPU with 40 GB VRAM, and on the PARCC Betty cluster on a single Nvidia Blackwell B200 GPU with 180 GB VRAM. In addition to the losses and metrics, we stored predicted probabilities and relative self-attention scores for each view for downstream processing and statistical analyses.

Statistical analyses

We used the torchmetrics (v.1.0.1) package to calculate MSE and MAE for regression tasks, and AUROC values for classification tasks within the training and validation loops. We additionally manually calculated AUROCs as empirical curves in the sensitivity and specificity space, computed from predicted probabilities generated by our models⁹³. To compare the performance of fine-tuned classifier models (that is, contrastive pre-trained vs baseline), we calculated non-parametric confidence intervals on the AUROC using DeLong’s method (paired)⁹⁴, following which P values were computed for the mean difference between AUROC curves. Additional analyses were performed to calculate the accuracy for each diagnostic label at different thresholds (optimizing for Youden’s statistic, a sensitivity of 0.90 or a specificity of 0.90). Differences between predicted LVEF% values and ground truth were assessed using Bland–Altman plots. Statistical analyses were performed and graphs were plotted using R (v.4.1.0); major packages used included pROC (v.1.17.0), ggplot2 (3.3.5) and blandr (0.5.1). The online test-set leaderboard webapp was created using shiny (1.8.1).

Attention visualizations

For every input scan, we output the raw self-attention tensors from each head of each layer of the MRI vision encoder during evaluation and processed them to yield 65 separate attention heat maps. As described earlier, the spatiotemporal resolution was reduced with each successive stage in the mViT architecture; the self-attention tensors were reduced from an initial spatiotemporal resolution of 8 × 56 × 56 at the first layer, to 8 × 7 × 7 at the last few layers. We kept only the attention values from the output patches for the purposes of visualization, and spatiotemporally interpolated these tensors back to a size of 16 × 224 × 224 via nearest-neighbour resampling. These arrays were exported to mp4 files using imageio and the ffmpeg library (Supplementary Figs. 17 and 18). Aside from the self-attention heat maps for each input video, we also computed the raw self-attention values from the multi-instance classifier head for relevant downstream tasks. After each scan was passed through the vision encoder, the resultant embedding was assigned a leaned raw self-attention score within the multi-instance self-attention modules. We calculated the relative differences in self-attention scores across different view planes for each disease label. These relative self-attention values were visualized as 2D heat maps as shown in Fig. 4c. The multi-instance classifier head self-attention scores showed that the network learns to differentially prioritize view planes for different clinical tasks.

Supplementary Material

Supplementary information The online version contains supplementary material available at https://doi.org/10.1038/s41551-026-01637-3.

Data availability

Cardiac MRI Data from Stanford Medicine, UCSF and Medstar were secured via agreements with Bunkerhill Health. Applications for data access can be found at https://www.bunkerhillhealth.com. Cardiac MRI data from the University of Pennsylvania are available to researchers under appropriate data use agreements. Please contact R.S. for additional information. Applications for data access from the UK BioBank can be found at https://www.ukbiobank.ac.uk/enable-your-research/apply-for-access. The ACDC dataset is publicly available at https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html. The Kaggle MRI dataset is publicly available at https://www.kaggle.com/competitions/second-annual-data-science-bowl/data. Leaderboard for classification and regression test datasets are available at https://rohanshad.shinyapps.io/cmr_leaderboard/. We are happy to accept model weights as submissions to evaluate our existing datasets; please contact R.S. for additional information. Source data are provided with this paper.

Code availability

Model checkpoints, raw table and figure data, and additional scripts to preprocess and analyse data are available in GitHub at https://github.com/rohanshad/cmr_transformer ref. 95. Final contrastive pre-trained checkpoints for our models are available for academic use in Huggingface at https://huggingface.co/rohanshad/cmr_c0.1 (ref. 96). Please contact the corresponding authors for commercial licences. Additional scripts and pre-processing code for handling Cardiac MRI data are available free of cost in GitHub at https://github.com/rohanshad/cmr_toolkit ref. 97.