Combining Artificial Intelligence With Human Insight to Automate Echocardiography

See Article by Asch et al

A quick eyeball estimate of left ventricular function is often the first thing an echocardiographer or clinician performs, whether consciously or subconsciously, when they start to perform an examination.¹ By the time they are writing their report, a precise, quantified measure will have replaced this estimate.² Over time, M-mode quantification³ has been superseded by a series of new ultrasound technologies, to the point where 3D echocardiography is now accepted as providing measures as precise as cardiovascular magnetic resonance.⁴ So what next? Asch et al⁵ in this issue of Circulation Cardiovascular Imaging propose that echocardiography may have reached a stage in life perhaps best described by Marcel Proust in his ‘Remembrance of Things Past’ when he realizes that: ‘the real voyage of discovery is not in seeking new landscapes but in having new eyes’.⁶ The new eyes for echocardiography are those of a computer, trained using artificial intelligence (AI) methodology, to mimic a human expert’s eye.

To develop an AI that can eyeball left ventricular function, the team had to give some guidance to the computer. Based on the a priori observation that the operator does not rely on volume measures when they estimate left ventricular function, they trained the computer to look for proportional, rather than absolute, changes in the size of the heart. When performed along 2 orthogonal axes within the apical long-axis views these proportional changes can replace volume measures in the equations used to calculate ejection fraction.⁵ The approach overcomes several technical challenges, including needing to account for image scaling when generating measures and trying to identify borders in poor quality images, which limits some other automated techniques.^7,8 The major limitation is that, because the computer performs no measures of left ventricular size, no volume or size measures are reportable with the technology and, as only 2 planes are used, unusual pathologies or regional variations are overlooked.

After training on 50 000 real-world patients, the method was tested on scans from 99 new studies, which had also been analyzed by independent experts. The ability to identify patients with severe left ventricular dysfunction (ejection fraction <35%) was particularly impressive achieving sensitivity and specificity of 0.90 and 0.92, respectively. The overall performance across a range of ejection fraction levels was also good but with apparently large limits of agreement of over 10%. This degree of variation from a reference standard was similar to that recorded for clinical readers. If most of the time we are eyeballing within 5% to 10% of the right answer, is that acceptable? Some of this variation may also represent an experimental problem of comparing 2 methods, neither of which generate gold standard measures, against each other. Which is closer to the truth, the AI method or the reference method?

Interestingly, on closer examination, there are subtle differences that differentiate AI eyeballing from human eyeballing. The clinician has a tendency to identify confidently severe or normal but finds intermediate or borderline cases more difficult. In the correlation and Bland-Altman plots, clinical readers appear to have a tendency to report measures closer to upper or lower limits and cluster around cut points such as 35% and 55%. This leads to narrower bounds of variation below 35% and above 55%, and a tendency for intermediate ejection fractions to be classed higher or lower, apparently at random. Severe is obvious; normal is obvious, with a split decision in the middle. In contrast, the AI has more uniform bounds of error across the range of ejection fractions but with a trend to underestimate lower ejection fractions and overestimate higher ejections, similar to the clinician.

As a source of immediate feedback on cardiac function at point of care, possibly with handheld devices, this technology appears attractive. However, 2 apical views are required, and some adaptation and retraining may need to be considered to take account of echocardiography use in emergency departments or the operating theater. The first estimate of ejection fraction is often from single views and, typically, this view is the parasternal or transgastric short axis.⁹ There is also the question of whether the echocardiography community, after years of increasing accuracy of ejection fraction measures, and with new availability of AI-driven tools available to auto-contour and measure function,^8,10 are ready to take an apparent step back to automated eyeballing. A novice only needs training on 50 images to achieve acceptable eyeballed estimates of ejection fraction¹¹ and, despite many publications showing its accuracy,¹² everyone still expects a quantified, contoured measure to end up on the report. Increasingly, this report also now needs to include detailed regional assessment and strain measures.²

But there is an elegance to this work. This is not another description of brute force, black-boxed AI applied to large scale data, but the use of clever human observation and insight into how an operator looks at images to carefully direct machine learning. The tangible excitement in this article is that, because of this approach, a computer that has never seen an image before can now take an echocardiogram, look at 2 views and, without any user input or need to contour the ventricle, estimate what it thinks is the ejection fraction with astonishing precision.

Disclosures

Dr Leeson acknowledges current grant support related to medical imaging from the British Heart Foundation, Wellcome Trust, National Institute of Health Research, and Lantheus Medical Imaging. Dr Leeson is a stockholder and founder of Ultromics a medical imaging artificial intelligence company. The other author reports no conflicts.