Evaluation of the Accuracy of Cuffless Blood Pressure Measurement Devices: Challenges and Proposals

Abstract

Several novel cuffless wearable devices and smartphone applications claiming that they can measure blood pressure (BP) are appearing on the market. These technologies are very attractive and promising, with increasing interest among healthcare professionals for their potential use. Moreover, they are becoming popular among patients with hypertension and healthy people. However, at the present time there are serious issues regarding BP measurement accuracy of cuffless devices and the 2021 European Society of Hypertension (ESH) Guidelines on BP measurement do not recommend them for clinical use. Cuffless devices have special validation issues which have been recently recognized. It is important to note that the 2018 Universal Standard for the validation of automated BP measurement devices developed by the American Association for the Advancement of Medical Instrumentation, the ESH, and the International Organization for Standardization (AAMI/ESH/ISO) is inappropriate for the validation of cuffless devices. Unfortunately, there is an increasing number of publications presenting data on the accuracy of novel cuffless BP measurement devices, with inadequate methodology and potentially misleading conclusions. The objective of this review is to facilitate understanding of the capabilities and limitations of emerging cuffless BP measurement devices. First, the potential and the types of these devices are described. Then, the unique challenges in evaluating the BP measurement accuracy of cuffless devices are explained. Studies from the literature and computer simulations are employed to illustrate these challenges. Finally, proposals are given on how to evaluate cuffless devices including presenting and interpreting relevant study results.

THE POTENTIAL OF CUFFLESS BLOOD PRESSURE MEASUREMENT DEVICES

Cuffless blood pressure (BP) measurement devices offer great promise in the field of hypertension awareness, management, and control. First, cuffless BP measurement technologies embedded in wearable devices and smartphones can improve hypertension awareness by providing numerous out-of-clinic measurements in the mass population, allowing thereby early diagnosis and intervention of this very common and largely asymptomatic condition. Second, cuffless BP monitoring can optimize the estimation of the true burden of BP over time, by providing a complete evaluation of the BP level and behavior during all daily circumstances and for long periods of time. Eventually, by continually revealing high BP in individual patients they can improve antihypertensive drug treatment compliance and hypertension control rates. Thus, cuffless BP measurement devices have the challenging potential to change the measurement of BP, the diagnosis of hypertension and its long-term management and control, mitigating thereby the burden of hypertension – the leading cause of disability-adjusted life years lost worldwide.¹

TYPES OF CUFFLESS BP MEASUREMENT DEVICES

There are two types of cuffless BP measurement devices: cuff-calibrated and calibration-free. The cuff-calibrated devices require periodic measurements with a conventional arm-cuff device, usually every few weeks for cuffless wearable devices, or few hours for devices used in anesthesia/surgery according to manufacturer specific instructions, aiming to yield cuffless measurements in units of mmHg in the time interval between the ‘cuff calibrations’. Calibration is usually performed using a validated automated oscillometric upper arm cuff device, as manual auscultatory BP measurement by users is impractical. However, different brands/models of oscillometric devices do not give identical readings, thus, influencing differently the consequent cuffless BP estimations. Hence, cuff-calibrated cuffless devices solely track BP changes relative to the preceding cuff BP measurement obtained for calibration (hereafter referred as calibration BP) in an individual in whom the device has been calibrated. Calibration-free devices do not require a cuff calibration procedure for each individual user, but accurate BP measurement may be more challenging to realize.

VALIDATION OF CUFFLESS BP MEASUREMENT DEVICES

Challenges

Studies of both cuff-calibrated and calibration-free cuffless BP measurement devices have been increasingly appearing in the literature.² More significantly, cuff-calibrated cuffless devices are increasingly emerging in the marketplace.^3-5 Thus, understanding how well these devices work is more important now than ever. However, the evaluation of the BP measurement accuracy of cuffless devices against standard cuff devices is not straightforward and has several methodological issues that need to be addressed. There are at least three challenges.

Firstly, in the last three decades established validation protocols, including the Association for Advancement of Medical Instrumentation (AAMI) protocol, the British Hypertension Society (BHS) protocol, and the European Society of Hypertension International Protocol (ESH-IP), have served as the standard methodology for testing the accuracy of BP measurement devices,⁶ but these protocols are not intended for cuffless BP measurement devices. The 2018 AAMI/ESH/International Organization for Standardization (ISO) Universal Standard⁷ is designed to assess automated cuff devices (mostly oscillometric but also others) against reference manual auscultatory BP measurements taken by two observers simultaneously. Three pairs of measurements of the test and reference devices must be obtained from at least 85 individuals of diverse characteristics (e.g., ≥5% of the reference systolic BP readings must be ≥160 mmHg, ≥20% must be ≥140 mmHg, and ≥5% must be ≤100 mmHg). The AAMI/ESH/ISO Universal Standard requires a device to pass specific criteria assessing individual BP readings (Criterion 1) and individual participant average BP readings (Criterion 2).⁷ The well-known Criterion 1 is bias and precision errors of individual BP readings (i.e., mean and standard deviation of the ≥255 BP errors) within 5 and 8 mmHg (for test-reference BP comparisons), respectively.⁷ Criterion 2 investigates the precision errors of individual participants (standard deviation of the ≥85 average BP differences of triplicate comparisons per individual).⁷ Since the Universal Standard does not include invoking BP changes within an individual, it is not at all applicable to cuff-calibrated devices. The Universal Standard may make sense for assessing the accuracy of calibration-free devices, yet again certain aspects need to be reconsidered as they operate differently from automated cuff devices. Unfortunately, there is an increasing number of publications presenting data on the accuracy of novel cuffless BP measurement devices, with inadequate methodology and potentially misleading conclusions.^8-11 More importantly, a recent search of self-home BP monitors which are commercially available in the Australian online market found that none of the wrist-band wearables BP monitors was validated in any standard way.¹² Thus, the 2021 ESH Practice Guidelines for BP measurement do not recommend cuffless devices for clinical use.¹³

Secondly, the inclusion of inter- and intra-individual BP variations are crucial for cuffless device evaluation but difficult to obtain. For cuff-calibrated devices, evaluation of accuracy in static conditions shortly after calibration is certainly not enough, and it is mandatory to evaluate the accuracy during BP changes within each individual. Interventions may be performed to invoke BP changes (exercise, cold pressor test, mental test, drug effects, and others^14,15), or naturally occurring variations during daily life (due to, e.g., stress, meals, and activity) may be leveraged. However, interventions for decreasing and even increasing BP may pose risk to some individuals, and it may require numerous reference cuff BP measurements to attain sufficiently large spontaneous BP variations in a person. Whilst exercise significantly increases BP and post-exercise BP can be lower than pre-exercise levels,¹⁶ it may be relatively easy to detect exercise-induced BP changes (e.g., heart rate and arterial pulse amplitude rise significantly). A battery of interventions that change BP via different physiologic mechanisms appears necessary. Most studies implementing such interventions aiming to assess whether cuffless devices can track BP changes are performed in the context of technical development within pilot trials and therefore usually are not published. For calibration-free devices, the participant cohort must exhibit a wide BP range such as that required by the Universal Standard. However, identifying such a cohort can be difficult and costly especially for laboratory investigations of cuffless devices that have become popular. This might be a barrier especially for independent scientific teams.

Thirdly, cuffless devices often employ a mathematical model that takes demographics (e.g., age, gender) in addition to a cuffless measurement (i.e., arterial pulse) from the individual as inputs to ‘predict’ BP (machine learning),^10,17-26 yet demographics alone are known to correlate with BP.²⁷ As a result, it is unclear how much of the attained BP measurement accuracy especially of calibration-free devices is due to the actual hemodynamic measurement. For example, if the BP measurement accuracy were mainly predicted by age and gender, then the device does not offer any added value and would be superfluous (example presented below in simulation study – third case). This challenge does not apply to automated cuff devices, as they typically do not take individual demographic information as input.

Critical review of data presentation in recent studies

The abovementioned three challenges are evident in recent literature, which makes interpretation difficult. Many studies of cuffless devices claim BP measurement accuracy on the basis that the AAMI/ESH/ISO Universal Standard criteria are satisfied (i.e., bias and precision error limits within 5 and 8 mmHg), yet they do not follow all the key aspects of the Universal Standard.^10,17-26

For cuff-calibrated devices, the intra-individual BP changes are typically induced by mild interventions (e.g., leg raise in seated posture), modest (e.g., one day or month after cuff calibration), or even minimal (e.g., immediately following the calibration).^{10,17,20,21,24,26} Thus, in these cases, using only the calibration BP would yield small errors in predicting the subsequent BP. Furthermore, the results of cuff-calibrated devices are sometimes pooled over the individual subjects, which typically yields remarkable correlations.^{10,17,23,24,26} However, because of the calibration, these results merely reflect the inter-individual differences in the reference BP levels. As a result, the accuracy of these devices in tracking short-term or long-term BP changes within an individual often remains unclear.

For calibration-free devices, the BP range of the participant cohort is often small.^18,19,22,25 In these cases, simply using the average BP of a similar population (i.e., the training data) to predict the BP of the cohort would result in low errors. Furthermore, the overall results of the calibration-free devices are often only shown without revealing the contribution of the demographic inputs versus the hemodynamic measurement itself to the BP measurement accuracy.^22,25 As a result, the ability of the actual measurement of these devices to predict BP across different people often remains unclear.

Even studies that do include interpretable results for cuff-calibrated or calibration-free devices often do not highlight them.^18,26

Simulation study

To concretely illustrate the difficulty of interpretation, a basic simulation was performed. The simulation involved generating ten pairs of cuffless and reference cuff BP measurements from 100 individuals wherein the cuffless measurements had zero correlation with the reference measurements outside of age and gender (details concerning simulation analysis methods are presented in the Data Supplement). Figures 1, 2 and 3 show typical ways of displaying the results. The displayed results suggest good accuracy even though there is no correlation. The reason is that the simulated reference BP variations were not large and partially dependent on age and gender.

Figure 1.

Graphic presentation of data evaluating the accuracy of cuff-calibrated cuffless versus reference BP measurements: Classic but potentially distracting versus more informative presentation revealing the important role of the calibration BP.

Figure derived from simulation analysis. (A) Classic presentation of correlation between cuffless and reference BP, despite poor correlation within each individual (see color datapoints); (B) Cuffless versus reference BP change relative to calibration BP for each individual; (C) Classic presentation of Bland-Altman scatterplot displaying cuffless (‘Test Device’)-reference BP difference versus their mean; (D) Additional informative scatterplot based solely on BP prediction via calibration BP, without including the hemodynamic cuffless measurement (‘Baseline Device’).

Figure 2.

Graphic presentation of data evaluating the accuracy of calibration-free cuffless device without demographic input versus reference BP measurements: Classic but potentially distracting versus more informative presentation revealing potentially better accuracy of the device when using simply the average BP of a similar population to predict BP.

Figure derived from simulation analysis. (A) Classic presentation of Bland-Altman scatterplot displaying cuffless (‘Test Device’)-reference BP difference versus their mean; (B) Additional informative presentation showing poor correlation between cuffless and reference BP; (C) Additional informative presentation of Bland-Altman scatterplot based solely on BP prediction via the average BP of a similar population, without including the hemodynamic cuffless measurement (‘Baseline Device’).

Figure 3.

Graphic presentation of data evaluating the accuracy of calibration-free cuffless device with demographic input versus reference BP measurements: Classic but potentially distracting versus more informative presentation revealing potentially better accuracy of the device when using only age and gender (which are known to correlate with BP) to predict BP.

Figure derived from simulation analysis. (A) Classic presentation of correlation between cuffless (‘Test Device’) and reference BP; (B) Classic presentation of Bland-Altman scatterplot displaying cuffless (‘Test Device’)-reference BP difference versus their mean; (C) Additional informative presentation of correlation between cuffless and reference BP, indicating better correlation when BP is predicted based solely on age and gender, without including the hemodynamic cuffless measurement (‘Baseline Device’) (D) Additional informative presentation of Bland-Altman scatterplot based solely on BP prediction via age and gender, without including the hemodynamic cuffless measurement (‘Baseline Device’).

Three types of cuffless devices are simulated: (i) cuff-calibrated device, (ii) calibration-free device without demographic input, and (iii) calibration-free device with demographic input.

In the first case (cuff-calibrated device), the correlation between cuffless BP and reference BP may appear strong (black datapoints in Figure 1A). However, the cuffless BP change relative to the calibration measurement does not follow the reference BP change for each individual (color datapoints in Figure 1A and Figure 1B). Moreover, the BP errors of the cuff-calibrated device (‘Test Device’ in Figure 1C) are not smaller than those obtained by simply using calibration BP to predict the ensuing BP in each individual (‘Baseline Device’ in Figure 1D). In the second case (calibration-free device without demographic input), the cuffless BP errors may seem acceptable (Figure 2A), however, there is no correlation between cuffless and reference BP (Figure 2B). Moreover, the BP errors of the cuffless device (‘Test Device’ in Figure 2A) are not smaller than those predicted by a population average BP (‘Baseline Device’ in Figure 2C). Thus, the average BP of a similar population to predict the BP of the cohort results in smaller errors than inserting the hemodynamic parameter (i.e., the cuffless BP measurement) in the BP calculation model. In the third case (calibration-free device with demographic input), the correlation between cuffless and reference BP may seem strong, with cuffless BP errors appearing to be small (‘Test Device’ in Figure 3A and 3B). However, the correlation and the BP errors predicted by demographic inputs alone (e.g., age and gender) appear to be superior (‘Baseline Device’ in Figures 3C and 3D). As in the previous cases, the inclusion of the hemodynamic parameter (i.e., the cuffless BP measurement) in the BP calculation model may give worse results.

These simple simulations demonstrate that the overall accuracy of cuffless devices can appear to be satisfactory, whereas at the same time the cuffless measurement itself actually has a negative impact on BP measurement accuracy. This paradox, where a cuffless device might ‘measure’ BP more accurately only by predicting it based on the calibration BP, average BP of a similar population, or demographics and less accurately when employing the hemodynamic measurement is at least surprising and needs to be carefully considered.

PROPOSALS

Due to the challenges of evaluating cuffless BP measurement devices, we make the following proposals.

1. We suggest that validation studies of cuffless devices should make every effort possible to be based on sufficient intra- or inter-individual BP variations.

   1. For cuff-calibrated devices, a standard has been developed by the Institute of Electrical and Electronics Engineers (IEEE) with requirements for increases and decreases in BP within an individual (in addition to the participant BP range).²⁷ However, the method for inducing these BP changes is not defined and arbitrary. We propose standardizing the approach to ensure that the validation includes BP changes induced by different physiologic mechanisms rather than only a single mechanism such as exercise. For example, such a standard could involve the requirement of at least three distinct BP interventions (e.g., dynamic exercise, cold pressor test, mental stress test, drug-induced BP change - rise or decline, Valsalva maneuver, etc.).
   2. For calibration-free devices, we currently advocate for a full BP range in accordance with the AAMI/ESH/ISO Universal Standard. However, we suggest that this topic needs to be revisited given that the devices often use demographics as input to predict BP and are likewise intended to track intra-individual BP changes.
2. In addition, or alternatively, we advise to present the results as shown in Figures 1, 2 and 3:

   1. We strongly suggest showing the BP errors and correlations of naïve devices or ‘baseline devices’ [e.g., device in which constant BP via the cuff BP for calibration (Figure 1) or a population average BP is used to predict BP (Figure 2) or, preferably, a device that also includes demographic inputs (Figure 3)] side-by-side with the cuffless device.
   2. It would also be helpful to statistically compare the cuffless device with the baseline devices (e.g., paired t-test for bias errors (μ); Pittman-Morgan test for precision errors (σ);²⁸ or bootstrapping for root-mean-squared-errors [= √(μ²+σ²)].²⁹
   3. For cuff-calibrated devices, we just as strongly suggest showing correlations in terms of changes (e.g., plot of cuffless BP minus calibration BP versus reference BP minus calibration BP; Figure 1B) rather than absolute BP (i.e., plot of cuff-calibrated cuffless BP versus reference BP; Figure 1A). The latter plot may largely and trivially reflect the inter-individual differences in the reference BP levels.

The results presented in these ways will clearly indicate whether the cuffless device provides added value or not in BP measurement accuracy and would be worthwhile to include even if the BP variations are extensive.

• 3.We propose to first evaluate the cuffless device in laboratory conditions ideally against a manual auscultatory cuff device and then, if successful, in field conditions against an ambulatory arm cuff device. The latter field testing will also indicate BP change tracking performance within individuals.
• 4.We advise that criteria for success as outlined in the IEEE standard for cuff-calibrated, cuffless devices or the AAMI/ESH/ISO Universal Standard should only be claimed in studies involving adequate BP variations. As argued elsewhere,³⁰ we also believe the intended use of the cuffless device (e.g., screening versus diagnosis versus titrating therapy) should influence the accuracy thresholds which need to be fulfilled in validation studies.

CONCLUSIONS

As alluded to in this article, more work is needed to establish a universal standard for assessing the performance and accuracy of cuffless BP measurement devices. In 2014 the IEEE presented the specific accuracy issues of cuffless devices and proposed procedures for their evaluation.²⁷ The ISO is currently developing a new standard specifically for the validation of cuffless BP measuring devices, which aims to address all the special issues of such technologies. In the meantime, we hope the article facilitates understanding of the capabilities and limitations of emerging cuffless devices in both the literature and marketplace.