Accuracy of Volumetric Flow Rate Measurements

Abstract

Objective

Volumetric flow measurement with Doppler ultrasound is useful in assessing blood flow as part of an evaluation of arteriovenous fistula maturity in patients undergoing hemodialysis. In this study, we assessed both accuracy and variability in volumetric flow measurements obtained using modern and commercially available ultrasound systems and an in vitro experimental setup.

Methods

Volumetric flow measurements using duplex ultrasound were obtained by 3 users operating 5 different systems for randomized flow in the range of 100 to 1000 mL/min. Users performed 3 consecutive measurements at a given flow rate. Data were analyzed using statistical techniques to assess measurement accuracy and variability.

Results

Over the span of flow rates studied, the root mean square error (RMSE) for the 5 ultrasound systems ranged from 38.8 to 79.7, 36.8 to 52.0, 73.0 to 85.3, 26.7 to 44.6, and 43.9 to 93.5 mL/min. Corresponding average RMSE values were 60.3, 42.7, 81.1, 37.2, and 64.4 mL/min, respectively. A linear regression analysis of mean interobserver measurements revealed an excellent correlation for all ultrasound systems (r² > 99.1%). Assessment of intraobserver measurements revealed no statistically significant differences for any ultrasound system evaluated (P > .94). Comparison of interobserver measurements indicates no statistically significant differences between any of the 5 systems (P > .14).

Conclusions

Modern ultrasound systems are reasonably accurate in blood flow measurement in an experimental setup mimicking clinically relevant blood flow ranges in a hemodialysis fistula. Users need adequate training and experience to perform multiple measurements and use appropriate techniques to minimize errors in flow measurement.

Volumetric blood flow measurement with Doppler ultrasound is desirable for assessing blood flow and has been applied as part of an evaluation of arteriovenous fistula (AVF) maturity in patients undergoing hemodialysis.^1,2 It also has been used to evaluate the increase in blood flow from the preoperative to postoperative state in fistulas.^3–8 It has been used in the past to evaluate cerebral hemodynamics,^9,10 but significant accuracy limitations described by Winkler et al¹¹ led to the abandonment of most clinical uses of blood flow measurements.

However, Doppler imaging remains the current clinical standard for quantifying blood flow with ultrasound, and a wide variety of commercial duplex systems are available to support such measurements. Transonic blood flow measurement during dialysis was validated in part using ultrasound as a reference standard.¹² On the basis of the principle of uniform insonation, the standard method for volumetric blood flow quantification entails measuring the average velocity of flowing blood across a vessel and multiplying it by the cross-sectional area of the lumen. Numerous reports in the literature have documented error associated with ultrasound-based volumetric blood flow estimates.^11,13–18 These known and documented inaccuracies have partly prompted research into alternative measurement techniques to combat these sources of error.^19–25

Despite considerable promise to resolve a number of the known issues inherent to blood flow measurements obtained using standard duplex ultrasound, 3-dimensional measurement of Doppler blood flow systems is still in its infancy and under development. Until these prophecies are realized, the use of modern ultrasound systems to attain accurate quantitative blood flow measurements is still highly desirable for physicians and vascular sonographers and technologists. In this study, we assessed both accuracy and variability (intraobserver and interobserver) in volumetric blood flow measurements obtained using 5 different contemporary ultrasound systems and an in vitro experimental setup.

Materials and Methods

Flow Phantom and Control System

The experimental flow equipment setup used was a tissue-mimicking vascular flow phantom (peripheral vascular Doppler flow phantom; ATS Laboratories, Inc, Bridgeport, CT) and precision peristaltic pump (L/S digital console pump system; Cole-Parmer, Vernon Hills, IL) connected to a reservoir of blood-mimicking fluid (Doppler test fluid; ATS Laboratories, Inc) via 6.4-mm silicon tubing (L/S precision pump tubing; Cole-Parmer). A schematic diagram of the experimental setup is shown in Figure 1. The flow phantom vessel internal diameter was 6.0 mm (as verified by the manufacturer). The control system was calibrated by measuring the volume of fluid pumped into a graduated cylinder over a fixed interval (60 seconds). A mean value for a given pump flow rate was computed from 3 successive measurements. For true volumetric flow rates ranging from 100 to 1000 mL/min (15 increments), the average error of the pump flow rate was 0.1%.

Figure 1.

Schematic diagram of the in vitro experimental setup denoting the precision peristaltic pump with the electronic controller system in a series connection with the vascular flow phantom and tissue-mimicking fluid reservoir. The solid line and arrows indicate the direction of fluid flow; i.d. indicates inner diameter.

Ultrasound Systems

Five modern and commercially available duplex ultrasound systems were evaluated: (A) iU22 (Philips Healthcare, Bothell, WA) with an L17–5 probe, (B) HDI 5000 (Philips Healthcare) with an L7–4 probe, (C) Z.one Ultra (Zonare Medical Systems, Inc, Mountain View, CA) with an L10–5 probe, (D) LOGIQ E9 (GE Healthcare, Milwaukee, WI) with an ML6–15 probe, and (E) Acuson Sequoia (Siemens Medical Solutions, Mountain View, CA) with a 15L9 probe. For each ultrasound scanner tested, a brief training session (typically <60 minutes) conducted by an experienced operator preceded all experiments to ensure user familiarity.

Data Acquisition

The procedure for volumetric flow rate measurement was reviewed for each ultrasound scanner before experimentation because of the unique characteristics of each system. The following general imaging setup was chosen to maximize image resolution and visibility of the tubing lumen. The highest frequency available that gave optimal lumen definition was used. Image quality was also maximized using image depth, and the zoom feature was selected to maximize tubing visualization within the field of view. The focal zone was positioned to include the vessel, usually at its midpoint or near the posterior tubing wall. Beam steering or physical transducer angulation (heel and toeing using abundant gel when necessary) was performed so the angle between the ultrasound beam and blood vessel axis was 60°. The Doppler sample volume (gate) was adjusted to completely encompass the width of the vessel lumen, minimizing extension past the phantom tubing wall. On enabling the Doppler trace, the velocity scale was adjusted so the spectral waveform filled approximately 75% of the spectral window. A low wall filter setting was used, and the persistence was turned off when user adjustable. The baseline of the spectral velocity scale was set in the lower two-thirds of the spectral window where possible to avoid aliasing. With the same image for Doppler sample volume placement, caliper measurements of the vessel cross-sectional diameter were performed by the operator, and the area was computed automatically by scanner programs assuming vessel circularity. After adjustment of markers and analysis of at least 3 spectral flow cycles from the Doppler trace (ie, time-averaged mean velocity), the volumetric flow rate was calculated by the ultrasound system as the product of the time-averaged mean velocity and vessel area.

Data were acquired using the above protocol by 3 ultrasound users (1 sonologist and 2 sonographers, both with registered diagnostic medical sonographer credentials and 1 also with registered vascular technologist credentials) with an average of 15 years of experience (range, 9–22 years) in an American College of Radiology–accredited laboratory. In each session, the volumetric flow rates were presented in a predetermined randomized order. Blinded to the true volumetric flow rate, each user independently performed 3 consecutive measurements at a given flow rate, and data were recorded for future analysis. An example of different user measurements given a true volumetric flow rate of 500 mL/min is depicted in Figure 2.

Figure 2.

Representative duplex sonograms and user measurements given a true volumetric flow rate of 500 mL/min. As indicated on the images, the corresponding measured flow rates were 486.2 mL/min (A), 530.6 mL/min (B), and 491.2 mL/min (C).

To evaluate influences of flow profile variations on volumetric flow measurements, 1 system was reselected, and a single user repeated the set of measurements using a different flow phantom vessel size (4.0-mm internal diameter). Ultrasound scanner selection was determined by the system that had the greatest error trend. Otherwise, the experimental setup and data acquisition protocol were the same as detailed above.

Statistical Analysis

Experimental data were analyzed using both Microsoft Excel (Microsoft Corporation, Redmond, WA) and the statistical software package Minitab 15 (Minitab, Inc, State College, PA). The 3 volumetric flow measurements for each user in each ultrasound system were summarized as mean ± SD and represent a generalized metric for intraobserver variability. Using these summary statistics, the root mean square error (RMSE) was computed given knowledge of the true volumetric flow rate. Intraobserver and interobserver measurement variability was evaluated using 1-way and repeated measures analysis of variance, respectively. A linear regression analysis of individual and mean interobserver measurements was used to model data and introduces a representation of true error trends given both single and mean user measurements using modern duplex ultrasound systems. A 2-sample t test assessed statistical differences between the first and mean user measurements for the range of volumetric flow studied. The same statistical test also assessed differences between single-user measurements obtained from the 2 varying-size vessels. P < .05 was considered statistically significant.

Results

User measurements for each of the systems evaluated are depicted in Figure 3 as a function of the true volumetric flow rate. For the range of 100 to 1000 mL/min, the ideal measurement curve was plotted for comparison. A summary of correlation coefficients from the linear regression analysis of individual user measurements curves is presented in Table 1. An assessment of intraobserver measurements revealed no statistically significant differences for system A (P > .94), B (P > .96), C (P > .98), D (P > .99), or E (P > .96). Comparison of interobserver measurements indicated no statistically significant differences for system A (P = .51), B (P = .86), C (P = .79), D (P = .78), or E (P = .14). As flow varied from 100 to 1000 mL/min, the RMSE ranges for the 3 user measurements were as follows: system A, 38.8 to 79.7 mL/min; system B, 36.8 to 52.0 mL/min; system C, 73.0 to 85.3 mL/min; system D, 26.7 to 44.6 mL/min; and system E, 43.9 to 93.5 mL/min. Consequently, the average RMSEs for the ultrasound systems and the volumetric flow ranges studied were 60.3, 42.7, 81.1, 37.2, and 64.4 mL/min, respectively. Figure 4 summarizes the RMSE for each user and corresponding ultrasound system.

Figure 3.

Intraobserver volumetric flow rate measurements using the following duplex ultrasound systems: Philips iU22 (A), Philips HDI 5000 (B), Zonare Z.one Ultra (C), GE LOGIQ E9 (D), and Siemens Acuson Sequoia (E). The true volumetric flow rate ranged from 100 to 100 mL/min, and the ideal measurement curve is plotted for comparison.

Table 1.

Summary of Correlation Coefficients (Percent) From the Linear Regression Analysis of Individual User Measurement Curves and a Given Duplex Ultrasound System.

User	System A	System B	System C	System D	System E
1	>99.1	>95.5	>98.5	>99.1	>97.3
2	>95.7	>96.1	>98.8	>95.8	>95.6
3	>96.5	>95.2	>97.8	>98.7	>95.0

Figure 4.

Summary of the RMSE for each user and corresponding duplex ultrasound system tested.

Figure 5 depicts an overall average of user measurements for each ultrasound system. These results illustrate interobserver variability and permit a relative comparison of duplex ultrasound system performances for the range of flow rates evaluated. Linear regression analysis of measured volumetric flow rates (y) revealed the following trends for the ultrasound systems:

$$\begin{array}{ll}(\text{A})\hfill & y=1.02x+38.3(r^2=99.3\%)\hfill \\ (\text{B})\hfill & y=0.92x+17.9(r^2=99.6\%)\hfill \\ (\text{C})\hfill & y=0.81x+41.0(r^2=99.5\%)\hfill \\ (\text{D})\hfill & y=0.93x+31.4(r^2=99.4\%)\hfill \\ (\text{E})\hfill & y=0.86x+54.1(r^2=99.1\%),\hfill \end{array}$$

where x denotes the true flow rate. In general, slopes greater than unity indicate that the measured volumetric flow rates trended toward an overestimation, whereas slopes less than unity suggest an underestimation. Therefore, most of the systems (except A) underestimated true flow at high flow rates but to different degrees.

Figure 5.

Interobserver volumetric flow rate measurements for each duplex ultrasound system tested plotted as a function of the true volumetric flow.

For each duplex ultrasound system and for the entire volumetric flow rate range studied, the RMSE of the first user measurement was compared with the mean RMSE of the 3 repeated measurements. For systems B and C, differences between single and repeated measurement errors were not statistically significant (P > .14). Notwithstanding, repeated user measurements produced lower RMSE values 73% of the time (11 of 15), which shows that averaging multiple volumetric flow rate measurements can minimize intraobserver variability.

Using system C, volumetric flow rates were measured in a 4.0-mm phantom vessel by a single user. The corresponding RMSE of these averaged measurements was 81.5 mL/min, compared with an RMSE of 85.3 mL/min obtained from the larger- diameter vessel. Measurements from these 2 varying-size vessels were not statistically different (P = .98), indicating that flow profile variations did not influence duplex ultrasound measurements.

Discussion

In this study, we evaluated volumetric blood flow measurements by 3 blinded experienced users operating 5 commercially available duplex ultrasound scanners and an in vitro experimental design. For clinically relevant AVF blood flow ranging from 100 to 1000 mL/min, no statistically significant differences were found between intraobserver measurements from any of the duplex ultrasound systems tested (P > .94), which shows collective reproducibility of modern scanner flow measurements (Figure 3). This blood flow range was chosen to span clinically useful normal and abnormal blood flow rates for a hemodialysis AVF. This experiment was designed to determine the accuracy of a set of measurements of blood flow and to determine the precision of multiple measurements made by a single user as well as between users. Requiring users to repeat measurements multiple times minimized error associated with any cyclic variation in vessel diameter.^26,27

A statistical analysis of interobserver volumetric flow measurements for each system revealed no significant differences (P > .14), suggesting user precision regardless of the duplex ultrasound scanner used. A comparison of the graphs depicted in Figure 3 yields the conclusion that duplex ultrasound measurements using system C were the most reproducible and precise, albeit visibly inaccurate for flow rates exceeding 350 mL/min. In this system, underestimations in user measurements are marked by a progressive bias proportional to the true volumetric flow.

As flow varied from 100 to 1000 mL/min, the RMSEs for each user ranged from 38.8 to 79.7, 36.8 to 52.0, 73.0 to 85.3, 26.7 to 44.6, and 43.9 to 93.5 mL/min for systems A to E, respectively (Figure 4). The corresponding average RMSEs for these ultrasound systems and the volumetric flow ranges investigated were 60.3, 42.7, 81.1, 37.2, and 64.4 mL/min, respectively. With the exception of system D, all ultrasound systems tested required physical transducer angulation (rather than beam steering) to achieve a 60° angle between the ultrasound beam and blood vessel axis. Interestingly enough, system D proved to be the most accurate system tested throughout the rage of 100 to 1000 mL/min with a resultant average RMSE of 37.2 mL/min.

A relative comparison between the duplex ultrasound systems tested is possible given the results of interobserver measurements presented in Figure 5. It can be suggested that in vivo variability will be higher, and these results likely represent the upper limits of reproducibility because measurements were obtained in a controlled research setting using a flow phantom. For volumetric flow rates of less than 350 mL/min, all systems provide acceptable estimates. However, for flow rates exceeding 350 mL/min, there is a discernible difference in system measurements. At a typical fistula flow rate of 500 mL/min, which is considered the lower limit of flow for a mature fistula, the average volumetric flow rate varied as follows: system A, 563.7 ± 37.2 mL/min; system B, 471.4 ± 41.8 mL/min; system C, 434.6 ± 15.5 mL/min; system D, 514.7 ± 23.1 mL/min; and system E, 506.4 ± 66.0 mL/min. The clinical relevance is clear because a patient undergoing hemodialysis who would be considered to have a mature fistula on system X could be interpreted as having an immature fistula on system Y.

Regression analysis of both individual observer and mean interobserver measurement curves revealed an excellent correlation for all ultrasound systems (r ² > 95.0% and r ² > 99.1%, respectively). Regarding the latter, all measured volumetric flow rates trended toward an underestimation (ie, slopes less than unity) with the exception of system A. The value of this observation is that system C was shown to be the most precise of the systems tested for measuring volumetric flow rates, albeit discernibly inaccurate.

The discussion to this point has focused primarily on the user-dependent errors in volumetric flow rate measurements. However, we cannot overlook the considerable potential for large errors inherent to the duplex ultrasound-based approach for flow quantification such as those associated with measuring the mean Doppler shift (ie, mean blood velocity), vessel area, or Doppler angle.²⁸ Until more advanced and promising ultrasound-based techniques for quantifying volumetric flow are commercially available (eg, multidimensional techniques for blood flow velocity vector estimation), careful attention to the duplex ultrasound-based measurement technique can help avoid such errors.

Limitations of the study include the in vitro nature of the experimental setup used. However, performing the same study in an animal model would have introduced substantial complexity and measurement imprecision, which would have made data interpretation less straightforward. We did not investigate a range of diameters because of time constraints but chose 6.0 mm as both a reasonable estimate of the diameter of a mature fistula and an approximate diameter of the brachial artery after fistula creation.⁴ Our results represent the best one can expect when measuring volumetric flow using standard ultrasound techniques and equipment. Users should not be surprised if their results in clinical situations are actually much more variable.

In summary, we have found that modern duplex ultrasound scanners are reasonably accurate in measurement of blood flow in an experimental setup mimicking clinically relevant blood flow ranges in a hemodialysis fistula. Users need to be adequately trained and experienced, perform multiple measurements, and use the techniques detailed above to minimize controllable errors in sonographer blood flow measurement.

Abbreviations

AVF

arteriovenous fistula

RMSE

root mean square error