Comparison of pulmonary congestion severity using artificial intelligence-assisted scoring versus clinical experts: A secondary analysis of BLUSHED-AHF

Abstract

Aim

Acute decompensated heart failure (ADHF) is the leading cause of cardiovascular hospitalizations in the United States. Detecting B-lines through lung ultrasound (LUS) can enhance clinicians’ prognostic and diagnostic capabilities. Artificial intelligence/machine learning (AI/ML)-based automated guidance systems may allow novice users to apply LUS to clinical care. We investigated whether an AI/ML automated LUS congestion score correlates with expert’s interpretations of B-line quantification from an external patient dataset.

Methods and results

This was a secondary analysis from the BLUSHED-AHF study which investigated the effect of LUS-guided therapy on patients with ADHF. In BLUSHED-AHF, LUS was performed and B-lines were quantified by ultrasound operators. Two experts then separately quantified the number of B-lines per ultrasound video clip recorded. Here, an AI/ML-based lung congestion score (LCS) was calculated for all LUS clips from BLUSHED-AHF. Spearman correlation was computed between LCS and counts from each of the original three raters. A total of 3858 LUS clips were analysed on 130 patients. The LCS demonstrated good agreement with the two experts’ B-line quantification score (r = 0.894, 0.882). Both experts’ B-line quantification scores had significantly better agreement with the LCS than they did with the ultrasound operator’s score (p < 0.005, p < 0.001).

Conclusion

Artificial intelligence/machine learning-based LCS correlated with expert-level B-line quantification. Future studies are needed to determine whether automated tools may assist novice users in LUS interpretation.

Introduction

Acute decompensated heart failure (ADHF) is the leading cause of cardiovascular hospitalizations in the United States.¹ Traditionally, assessment and re-assessment of pulmonary congestion has relied on integrating clinical history, physical exam and chest X-ray with variable success.² Point-of-care ultrasound (POCUS) accurately detects pulmonary congestion by visualizing bilateral B-lines on lung ultrasound (LUS).^3,4 B-lines change dynamically with severity and treatment of ADHF, providing both diagnostic and prognostic information.^5,6 However, LUS relies heavily on examiner competence in obtaining high-quality images and image interpretation. Although LUS acquisition and quantification does not require extensive training,⁷ there is large variability between different operators.⁸ The use of automated B-line software packages to identify and quantify B-lines has the potential not only to aid novice users, but essentially standardize quantification of B-lines across operators, allowing for more objective serial measurements.⁹

We aimed to determine whether an automated artificial intelligence/machine learning (AI/ML) assessment of LUS images from patients with ADHF correlates to an expert’s interpretation of images for the quantification of B-lines.

Methods

This was a secondary analysis from the BLUSHED-AHF study, an NHLBI sponsored, multicentre, randomized controlled trial, designed to test whether LUS-guided treatment of emergency department patients with ADHF resulted in less pulmonary congestion compared to usual care. Methods from this study have been previously published.¹⁰ The study was approved by the institutional review board at all sites (NCT03136198).

The BLUSHED-AHF LUS imaging protocol used the standard eight lung zones. LUS examinations were performed with Zonare ZS3, Z One Pro (Mindray, Mountain View, CA, USA) or Sonosite MTurbo (FUJIFilm Sonosite, Bothell, WA, USA) with the curvilinear C1–5s transducer. Serial LUS images were obtained on patients at multiple time points: enrolment (T0), 2–4 h after T0 (T1), 2–4 h after T1 (T2), and daily throughout hospitalization until hospital day 7 or day of discharge, whichever occurred first.

All POCUS operators were required to undergo a structured training programme. LUS were performed by research associates, emergency medicine residents, ultrasound fellowship-trained and non-ultrasound trained emergency medicine faculty (R0). Two faculty with expertise in LUS and LUS research (E1 and E2) provided blinded interpretations off-line. B-lines were quantified for each lung zone and determined by multiplying the percentage of the intercostal space filled with B-lines by 20, where the maximum total count per individual zone was 20 B-lines.¹¹ Therefore, the distribution of scores ranged from 0 to 20.

Artificial intelligence/machine learning modelling (lung congestion score)

An AI/ML model-based severity score (named LUS congestion score) was computed for 3858 LUS clips. Correlation between these scores and the two expert B-line scores (i.e. counts) as well as the bedside operator were computed to assess concordance.

A detailed description of the dataset and methods used in the development of the AI/ML model for congestion scoring was previously published.¹² In brief, the model was developed using an independent dataset comprised of 1419 LUS clips from 113 patients, acquired using curvilinear C1–5s transducers with Mindray ME8 or M9 (Mindray, Mountain View, CA, USA). Single-point annotations of B-line origins located at the pleura were provided by LUS experts for model training. The network architecture consisted of EfficientNet-B0 as encoder coupled with the decoder structure of U-Net. The AI model combined the predictions of five network instances, trained based on different overlapping partitions of the training dataset. The predicted lung congestion score was defined as the sum of the predicted B-line origins in all frames of a LUS video, divided by the total number of frames in the clip. This was done by the AI/ML counting the number points on the pleural line where a B-line originated from for each frame of a clip.

Statistical analysis

Spearman correlation was computed between sums of per zone counts from each of the raters: R0 (bedside operator), E1 and E2 (expert), and AI (AI lung congestion score). If a rater did not provide a score for at least two left zone and two right zone clips for a patient at a particular time point, that patient was excluded and was not considered in calculating correlations involving that rater at that time point.

Statistical significance for difference in Spearman correlations between expert and AI/ML versus expert and operator was calculated using a two-tailed z-test with Fisher-transformed correlation coefficients. These correlation coefficients were calculated based on taking the sums of per zone counts and averaging them over all time points. Further, the same scores were compared across all three groups when scores were normalized from 0 to 1 to determine if the lung congestion score over- or underestimated the BLUSHED-AHF congestion score.

Results

A total of 7928 LUS clips were obtained on 130 patients in the original BLUSHED-AHF study. Analysis in this paper was performed on 3858 video clips obtained at four time points: emergency department arrival/baseline (t0), 2 h post-baseline, 6 h post-baseline, and 24 h post-baseline. Not all patients had imaging at all time points. Of these 3858 clips, bedside operators (R0) annotated B-line counts for 3842 clips (99.6%). Expert 1 (E1) annotated B-line counts for 3406 clips (88.3%) and expert 2 (E2) annotated B-line counts for 3301 clips (85.6%). The AI model annotated B-line counts for 3858 clips (100%) and calculated a score range from 0 to 16.2 with a mean of 4.2 (± 3.5) for all clips reviewed. A total of 3212 clips (82.0%) had a B-line count from the operator, both experts, and the AI model (Figure 1).

Figure 1.

Flow diagram of BLUSHED-AHF clips through artificial intelligence (AI) model. E1, expert 1; E2, expert 2; R0, ultrasound operator.

Spearman correlations between rater scores were calculated for each time point using the sum of the available rater’s scores across all eight lung zones for each patient. Correlations for each time point for each of the six pairwise combinations of the four raters (R0 through R3) are listed in Table 1. Specifically, the two experts had excellent agreement between each other for all time points (r = 0.938). Both experts’ B-line quantification scores were significantly more correlated with the AI/ML lung congestion score than they were with the operator (r = 0.917 vs. 0.860 for E1, p < 0.005; r = 0.925 vs. 0.836 for E2, p < 0.001). When the congestion scores were normalized on a scale from 0 to 1, the AI/ML congestion score had a non-significant underestimation of scores compared to the operator and non-significant overestimation of congestion scores compared to experts.

Table 1.

Spearman correlation

	Expert internal agreement	Experts vs. AI		Operator vs. expert		Operator vs. AI
	E1 vs. E2	E1 vs. AI/ML	E2 vs. AI/ML	R0 vs. E1	R0 vs. E2	R0 vs. AI/ML
Day 1, T0	0.911	0.861	0.87	0.723	0.72	0.622
Day 1, T2	0.94	0.888	0.862	0.836	0.825	0.791
Day 1, T6	0.944	0.911	0.858	0.826	0.818	0.766
Day 2	0.953	0.901	0.925	0.806	0.816	0.763
All time points combined	0.938	0.894	0.882	0.809	0.808	0.751

Color corresponds to strength of correlation with blue (very strong), yellow and orange (strong), and red (moderate).

AI/ML, artificial intelligence/machine learning; E1, expert 1; E2, expert 2; R0, ultrasound operator.

Limitations

The AI/ML model used in the study does not account for confluent B-lines. If a LUS clip contained confluent B-lines that covered half the intercostal space or a single discrete B-line, the AI/ML model would predict the same LUS congestion score for that same clip. Thus, the LUS congestion score may underestimate B-line counts in the most pathological LUS clips.

Conclusion

Lung ultrasound offers real-time data to guide clinical decision-making and patient care for patients with ADHF. Although widely considered an easy technique to master and interpret, we tested this assumption. Operator experience may impact correct interpretation, thereby restricting its use and scalability. AI/ML technologies can provide automated interpretation of LUS images making POCUS more accessible to a broad range of novice users across a care team, standardizing interpretation allowing it to be both scalable and reproducible. Our data suggest that our AI/ML model performs at the same level as two LUS experts and outperforms a group of mixed-experience operators.

The B-line scoring protocol used in the BLUSHED-AHF trial has been shown to predict 30-day acute heart failure rehospitalization or death in studies of longitudinal LUS monitoring.¹³ As our study shows that an AI/ML LUS congestion score correlated well with experts on external data, it is possible that an automated tool such as this may provide information to help providers determine who may be at risk for rehospitalization and/or death as well as guide therapy. Although there is no agreement upon gold standard for B-line quantification, as we further train our LUS congestion score based on more LUS data with expert annotations, it is reasonable to expect that this AI/ML model will eventually resemble a large consensus of LUS experts in congestion scoring.