Noise in clinical data: insights from assessment of aortic stenosis by echocardiography

Abstract

Noise (synonyms: variability, imprecision) in clinical data is underappreciated. All measurements are affected by noise and bias. Research studies enrol many patients to average out noise and use randomisation and double-blinding to counter bias. These mitigation techniques are often not available in clinical medicine, where individual patients typically have a variable measured only once.

The example of echocardiographic assessment of aortic stenosis severity illustrates how the conventional presentation of data as point estimates masks wide variability in noise between variables, which affects utility. Adding aortic valve area to the quantification of aortic stenosis may result in poorer (noisier) estimates.

Every clinical variable reported as a point estimate comes with an invisible cloud of uncertainty (noise) of varying size across variables. Being mindful of how this noise varies across variables should facilitate optimal clinical decisions.

Graphical abstract

Graphical abstract. 95% confidence intervals for point estimates of aortic peak velocity and valve area. An echocardiogram may be reported as severe AS when the point estimate for peak velocity is 4 m/s and for valve area is 1.0 cm2. Applying 95% confidence intervals from Manna, et al shows that the peak velocity could indicate moderate or severe AS (J Am Soc Echocardiogr 37:370-372.e2, 2024) . The valve area is consistent with anywhere from mild to critical AS. Note that the values are reported with an excess of decimal places, giving a spurious impression of accuracy of the point estimate

Aortic stenosis (AS) severity is a surrogate marker. Similar to cholesterol, in which we are interested in its ability to predict heart attack, AS severity is a surrogate for the likelihood that the stenotic valve will cause death (or heart failure, which is more difficult to define and track and is not part of this analysis). So, how well do the different measures of AS severity predict death, and why do some do better than others?

The severity of AS is assessed by the variables listed in Table 1. Valve area is theoretically less confounded by differences in ventricular function (stroke volume) [1]. If we focus on the surrogate question “How restricted is the valve?”, then valve area seems to provide a more direct answer; it feels closer to what we are seeking than velocity or the gradient across the valve.

Table 1.

Echocardiographic indices of aortic stenosis severity [1]

Aortic stenosis: index of severity (units)	Origin
Peak velocity (m/s)	Point measurement of CW peak velocity
Mean pressure gradient (mmHg)	Tracing of CW flow profile
Valve area (cm2)	(π x Radius2LVOT x VTILVOT)/ VTIAV
Dimensionless ratio	Peak velocityLVOT / Peak velocityAV or VTILVOT /VTIAV

CW Continuous wave, VTI Velocity time integral, LVOT Left ventricular outflow tract, AV Aortic valve

Do some measures of AS severity predict death better than others?

How well does valve area perform when we turn our focus to the true and clinically meaningful endpoint of death? Table 2 summarises five observational studies that compared the prognostic power of the different indices of AS severity. Notably, all grades of severity are represented, and peak velocity is the only multivariate predictor in four of the five studies. Also notable is that the populations studied by Du, et al. and Capoulade, et al. included patients with left ventricular dysfunction (low ejection fraction, low stroke volume), the very circumstances in which valve area should be more accurate than the other indices. However, even then, peak velocity or mean gradient remained the predictors.

Table 2.

Natural history of aortic stenosis studies [2–6]

References	AS severity (peak velocity, m/s)	Outcome	Multivariate predictors
Otto, 1997	Moderate 3.6 ± 0.6	Death or surgery	Peak velocity	Change in peak velocity, Functional status
Pellikka, 2005	Severe 4.4 ± 0.4	Death	Peak velocity	Age, Chronic renal failure, Inactivity
Rosenhek 2009	Very severe 5.4 ± 0.4	Event-free survival	Peak velocity
Capoulade 2016	Mild to severe 3.3 ± 0.9	Death or surgery	Mean gradient	Age, gender, hypertension, diabetes, coronary disease, renal failure, EF
		Death	-	Age, gender, hypertension, diabetes, COPD, renal failure, SVi, EF
Du, 2021*	Mild-moderate 2.1 (1.7, 2.6)	Death	Peak velocity, Ejection fraction	Age, Functional status

EF Ejection fraction, COPD Chronic obstructive pulmonary disease, SVi Stroke volume index

*Importantly, Du and Capoulade included patients with ejection fractions below 50% (28% and 17% of participants, respectively) and indexed stroke volumes below 35 ml/m² (54% and 34% of participants, respectively)

The somewhat surprising finding is that the AS indices differ in their levels of noise (synonyms: variability, imprecision).

Why does peak velocity outperform other measures?

Measurement error

Every measurement is confounded by bias and noise (inset). Stepping on bathroom scales every day and consistently obtaining a high reading is an example of measurement bias. If the scales sometimes read high and other times low, then this random error is termed noise [7]. The systematically smaller left ventricular volumes measured by echocardiography compared with those measured by magnetic resonance imaging are an example of bias [8]. Noise is illustrated by the less certain, more variable estimates of coronary artery stenosis severity by computed tomography compared with invasive angiography. Repeated measurements of the LVOT area are noisy due to variation in measurement site and technique. There is also a bias, with consistent underestimation due to assuming the area is circular instead of elliptical. There are circumstances in which a trade-off between noise and bias occurs: using decision trees for clinical decisions can yield more consistent decisions, but can introduce bias [9, 10]. In the research setting, randomisation and double blinding reduce many sources of bias. And multiple measurements (from the same or many individuals) average out noise. Multiple measurements also allow the degree of noise to be quantified and presented as standard deviation or interquartile range. However, clinical measurements are generally taken only once or a few times, yielding a noisy estimate, and the degree of noise is unquantifiable. The result is that, unlike in research, clinical measurements can be presented only as point estimates, which may allow us to subliminally disregard their inherent uncertainty.

Quantifying noise

Noise can be quantified using the standard deviation, but this cannot be used to compare indices with different units or even with different average values. Coefficient of variation (CV) is the noise (standard deviation) divided by the mean. Small values of CV imply less noise relative to the signal.

Synonyms

Noise

Variability

Imprecision

CV is ideal for comparing noise between variables with different units, since both are dimensionless quantities (the units in the numerator and denominator cancel out).

Measurement error for indices of aortic stenosis severity

Quantifying noise is tedious and not often done comprehensively [11]. However, Manna, et al. studied 30 patients with moderate aortic stenosis (peak velocity 3–4 m/sec) and a regular heart rhythm [12]. Two different sonographers did two immediately sequential transthoracic echocardiograms. Then, two reporters independently and blindly freely chose loops and stills for measurement. This comprehensive assessment of interobserver reproducibility used CV to quantify noise, with the results summarised in Table 3.

Table 3.

Variability of aortic stenosis indices [12]

Aortic stenosis: index of severity	Coefficient of variation, %	95% confidence intervals
Peak velocity	6	4–9
Mean gradient	12	8–17
Dimensionless index*	18	11–25
Aortic valve area	28	17–39
LVOT diameter	9	5–12
VTILVOT	12	7–16
VTIAV	8	5–11

*This study used VTI for the dimensionless index. Other studies indicate that using peak velocity results in a lower CV [13]

Adapted from Manna, et al. [12]

The variation in CV between the different indices is noteworthy. Compared with peak velocity, mean gradients have 2 times the CV, dimensionless index 3 times and valve area 4 ½ times. The authors used the European Society of Cardiology algorithm to grade the severity of AS based on measurements from the two reporters in the same studies (not different) [14]. One quarter of patients were graded differently, with 67% of disagreements due to differences in valve area between the two reporters. This is a practical illustration of how the noisier valve area causes varying estimates of stenosis severity.

When we see a number in an echo report, it conveys an illusion of precision because there is no measure of its noise. Single measurements are noisy but unquantifiably so. Often, the number of decimal places used exaggerates the precision. For example, an echocardiogram report may indicate severe AS with a peak velocity of 4.00 m/s and a valve area of 1.00 cm². Using the 95% confidence intervals from Manna et al., all we can say is that the peak velocity lies between 3.5 m/s (moderate AS) and 4.5 m/s (severe AS). For valve area, the noise is much greater; the valve area could be as small as 0.4 cm² (critical AS) or as large as 1.6 cm² (mild AS). See Graphical Abstract.

A different approach by Pillai, et al. yields complementary results [15]. They took 24,948 pairs of clinical (not research) transthoracic echocardiograms, where the second study reported “no significant change”. The mean delay was 312 days, with a median of 177 days; however, a sensitivity analysis restricting the delay to 90 days yielded similar results. Regarding indices of AS severity, peak velocity had the lowest CV, followed by valve area, then mean gradient. The authors observe “Notably, this measurement variability was seen despite the bias introduced by the real-world clinical practice of examining prior echocardiogram reports, which would dampen fluctuations in reporting.” Indeed, the diameter of the left ventricular outflow tract (LVOT) had the lowest coefficient of variation of any variable, in fact a median value of zero, and markedly lower than even the next lowest. This is consistent with the common practice of using the same LVOT diameter as in previous studies, assuming it doesn’t change over time. The standardisation of this error-prone component of the aortic valve area will falsely reduce noise in the estimate of valve area.

These two studies explain the difference between peak velocity and valve area in their respective abilities to predict death; valve area is over four times noisier than peak velocity. This noise obscures the valve area signal, and it is reasonable to question its use. This may be a situation where more is less; where adding valve area to other estimates of AS severity could yield poorer (noisier) estimates [16]. Over the last twenty years, aortic stenosis has been subclassified according to discordance of peak velocity, valve area and stroke volume. Rather than representing differing physiological states, these subtypes may represent measurement artefact due to noisy measurements, particularly valve area and stroke volume [17–19].

Reproducibility

Reproducibility is commonly reported in research studies, but not all reproducibility evaluations are created equal, and it can be hard to know exactly how it was measured [11]. Reproducibility can assess both bias and noise, but our focus is on noise. The best reproducibility (least noise) is obtained when two reporters measure off the same still frame from the same study. See Fig. 1. Not being blinded to each other’s measurements will further reduce noise, but it introduces subjectivity and bias that render the assessment of little value. Page 257 [20] and p 56 [21].

Fig. 1.

Reproducibility assessments in order of increasing noise (and increasing clinical relevance). After Popovic and Thomas [11]

Conversely, the most noise (and perhaps the most realistic assessment) is obtained when the same patient undergoes two separate echocardiograms by different sonographers (as per Manna et al.), which are then analysed blindly by two different reporters who select their own loops and still frames for measurement. This best-practice, comprehensive assessment of reproducibility is tedious to perform and is rarely done.

Where does noise come from?

Diverse factors generate noise. Within echocardiography, there are biological sources of noise such as day-to-day changes in a patient’s volume status, heart rate, and blood pressure. Longer-term changes, such as anaemia, can markedly affect Doppler velocities [22, 23].Individual sonographers use different scan windows and machine settings that affect subsequent measurements. The two VTI measurements used in the continuity equation for aortic valve area can be noisy. The LVOT VTI requires placement of the pulsed Doppler sample volume in the funnel-shaped LVOT, without a precise landmark, in the presence of respiratory movement [24]. The valvular VTI and peak velocity require interrogation of multiple windows due to variability in jet direction through the stenosed aortic valve [25]. Image analysis can vary with the choice of cine loop and still frame, as well as with individual differences in measurement technique. Linear dimensions are less noisy than biplane volumes, and this may, surprisingly, make them better at tracking changes in volume over time [12, 26]. Echocardiographic images have inherent noise, including dropout parallel to insonation, reduced far-field lateral resolution, and foreshortened apical two-chamber views (for left ventricular volumes).

Calculations on raw measurements can magnify noise, such as when the LVOT radius is squared, resulting in a doubling of the relative measurement error. Subtraction or division typically outputs smaller numerical values than the input variables. Despite this, errors remain additive so that the resulting noise can be markedly increased [27]. Calculation of aortic valve area includes a division that may contribute to the excessive noise of this variable. The upper bound of the error in this division is obtained by adding the errors of each component, using relative error, because the uncertainty is proportional to the values. The division yields a valve area numerically much smaller than any of the equation components. Valve area being numerically small, makes even a small error large relative to the point estimate. Each mathematical manipulation magnifies preceding errors, meaning that the multistep calculation of valve area, with squaring following by a division, can turn even precise raw measurements into a noisy derived variable.

The coefficient of variation can become unstable when the point estimate or the mean is numerically small, as with aortic valve area. In that setting, a slight variation in error can create large swings in CV, with even a small change being large relative to the mean. On occasion, this can make the CV large even when the error is small and not clinically important.

Surrogates can be noisy or less noisy; for example, brain natriuretic peptide is a noisy surrogate for heart failure because it is also influenced by factors such as sex, age, obesity, and renal failure. Cholesterol is a noisy surrogate for myocardial infarction because many other factors, such as age, smoking, diabetes, and blood pressure, influence the latter.

Conclusion

Taking the point estimate as the true value of a variable confers the certainty we crave as decision-makers. However, it doesn’t reflect reality and may not lead to optimal clinical decisions. In 1913, Niels Bohr’s planetary model gave a simple and precise picture of the atom. (Fig. 2). By 1925, Erwin Schrodinger had replaced it with the current, more realistic model. The discrete location of the electron is now a fuzzy cloud of probable (and less probable) locations. Similarly, every clinical variable we encounter comes with a cloud of uncertainty. Acknowledging this noisy uncertainty may promote optimum clinical decisions.

Fig. 2.

Models of atomic structure

The model of Neils Bohr suggested that electrons could be precisely located like moons orbiting a planet (the nucleus). Schrodinger subsequently developed the current model that describes the position of electrons in terms of probabilities. Although less satisfying to our human desire for certainty, this statistical description reflects reality more faithfully.

Authors’ contributions

GA conceived the idea and wrote the original oral presentation, GA and JL wrote the manuscript together and both approved the final version.

Funding

No funding sources.

GA conceived the idea and wrote the original oral presentation, GA and JL wrote the manuscript together and both approved the final version.

Data availability

There are no new data associated with this article.

Declarations

Ethics approval and consent to participate

Human Ethics and Consent to participate declarations: not applicable as no original data were gathered for this study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.