A Universal Standard for the Validation of Blood Pressure Measuring Devices: Association for the Advancement of Medical Instrumentation/European Society of Hypertension/International Organization for Standardization (AAMI/ESH/ISO) Collaboration Statement

Abstract

In the last 30 years several organizations, such as the US Association for the Advancement of Medical Instrumentation (AAMI), the British Hypertension Society, the European Society of Hypertension (ESH) Working Group on Blood pressure (BP) Monitoring and the International Organization for Standardization (ISO) have developed protocols for clinical validation of BP measuring devices. However, it is recognized that science, as well as patients, consumers and manufacturers would be best served if all BP measuring devices were assessed for accuracy according to an agreed single validation protocol that had global acceptance. Therefore, an international initiative was taken by AAMI, ESH and ISO experts who agreed to develop a universal standard for device validation. This statement presents the key aspects of a validation procedure, which were agreed by the AAMI, ESH and ISO representatives as the basis for a single universal validation protocol. As soon as the AAMI/ESH/ISO standard is fully developed this will be regarded as the single universal standard, and will replace all other previous standards/protocols.

History of validation protocols

The accurate measurement of blood pressure (BP) is an important prerequisite for the reliable diagnosis and efficient management of hypertension and other medical conditions. Therefore, the evaluation of the accuracy of automated devices available on the market for BP measurement in the medical environment and the community is of paramount importance.

Validation of BP measuring devices began in the 1980s with a series of ad hoc validation protocols [1]. In 1987 the US Association for the Advancement of Medical Instrumentation (AAMI) standard for automated BP monitors included a clinical validation procedure [2]. In 1990 the British Hypertension Society (BHS) published a protocol dedicated to the validation of BP monitors in the clinical setting, which incorporated many of the features of the AAMI validation standard, but also had many important differences [3]. The AAMI standard was revised in 1992 and 2002 and the BHS protocol in 1993 [4,5]. In 1999 the German Hypertension League introduced its own validation protocol [6]. In 2002 the European Society of Hypertension (ESH) Working Group on BP Monitoring developed the ESH-International Protocol (ESH-IP), with the major difference that a smaller sample size was required (N=33 compared to N=85 in the AAMI and BHS protocols) [7]. A revised version of the ESH-IP with more stringent validation criteria was published in 2010 [8]. In 2009 the International Organization for Standardization (ISO) [9] developed another standard, which incorporated aspects of EN 1060-4 and the AAMI SP-10 (e.g. sample size and validation criteria) and has been adopted by the AAMI Sphygmomanometer Committee [10]. A revised version of the American National Standards Institute (ANSI)/AAMI/ISO standard was released in 2013 [11]. Despite whatever differences, all these protocols have major similarities and a common objective, namely the standardization of the validation procedures to establish minimum standards of accuracy and performance. The history and evolving progress of protocols to ensure BP monitors accuracy has recently been reviewed [12].

Objective

The authors of the different validation procedures appreciate that science, as well as patients, manufacturers and consumers would be best served if all BP measuring devices were assessed for accuracy according to an agreed single validation protocol that had global acceptance. The aim of this statement is to establish international willingness for a universally acceptable protocol, and having done so, to build on past experience to produce a single protocol for the validation of BP measuring devices that will replace all previous ones. It is not within the scope of the present work to provide a detailed comparison of the different validation protocols, which have indeed been the subject of extensive scientific discussion and debate in the last two decades [13–22].

AAMI/ESH/ISO collaboration

In acknowledgement of this objective, members of the AAMI, ESH and ISO committees agreed to meet and discuss all the aspects of validation that deserve to be reexamined, so as to be able to achieve a consensus on an optimal validation standard.

The ESH Working Group on BP Monitoring and Cardiovascular Variability, which consists of an international group of clinicians with expertise in BP monitoring with members from Europe, US, Canada, Japan, China, and Australia, appointed a committee (G Stergiou [chairman], R Asmar, N Atkins, JP Ioannidis [medical statistician], R McManus and P Lacy [also members of BHS Working Group on BP Monitoring], M Myers, P Palatini, G Parati, A Shennan, J Wang, E O’Brien), which met with representatives from AAMI and ISO (ISO/TC 121/SC 3/JWG 7, Non-invasive sphygmomanometers; B Alpert, S Mieke, D Quinn, S Eckert, G Frick, T Graßl, T Ichikawa, A Murray, J Sarkis, T Usuda, C Wu) on 7-8 March 2016 in Berlin, Germany.

A list of methodological-statistical and practical-clinical issues for the AAMI/ISO standard and the ESH-IP protocol was prepared by the ESH representatives and a medical statistician and presented for discussion with the AAMI/ISO representatives during the joint meeting in Berlin, Germany. A point-by-point discussion followed, aiming to identify areas of agreement, and also disagreement requiring further consideration and research. Another AAMI/ESH/ISO meeting took place in April 2017 in Athens, Greece.

This summary report presents nine key aspects of the validation procedure, which were agreed by all the AAMI, ESH and ISO representatives as the basis for developing a single universal protocol for the validation of BP monitors. Consensus was based on the evidence from previous validation studies using the AAMI, BHS, ESH-IP and ISO protocols, new statistical analyses on power of study sample and subgroups, and expert opinion. It is not the purpose of this preliminary document to present a detailed description of all the aspects of the validation procedure, which will, however, become available as the organizations involved develop the procedural detail of the universal protocol.

Methodological and clinical issues affecting the validation procedure

Validation study efficacy measure

A tolerable error of 10 mmHg or less (using an individual’s average of three BP readings versus a reference BP measurement method) and an estimated probability of that error of at least 85% is acceptable as a compromise, taking into account the performance of currently available BP monitors. This is compatible with the current ANSI/AAMI/ISO requirements [11] and those of the revised ESH-IP allowing for a 10 mmHg error with frequency of 12-18% [8].

This error does not reflect an acceptable level of inaccuracy for BP measurement, but takes into account the variability of the validation methodology and also leaves room for devices’ accuracy improvement. Setting this level of accuracy is expected to separate devices with ‘high’ or ‘moderate accuracy’ from those with ‘low accuracy’ (unacceptable). It is stressed that ’high accuracy’ does not mean necessarily ‘excellent’. Thus, passing these requirements does not equate to ideal accuracy, and some patients may still have inaccurate measurements. Clinicians need a higher level of accuracy in BP measurement and encourage the industry to continue efforts in technological improvement in order to develop more accurate devices.

CONSENSUS

• A device is considered acceptable if its estimated probability of a tolerable error (≤10 mmHg) is at least 85%.

Validation study sample size

As stated above, a standard is needed which acknowledges potential advances in technology, while still allowing for a number of contemporary devices to fulfil the protocol requirements. The standard should ensure that ‘high’ and ‘moderate accuracy’ devices will pass and ‘low accuracy’ devices will fail. In addition, it should be feasible for the standard to be implemented by many research centers.

The optimal sample size for a validation study has been a matter of debate and an important point of disagreement between the AAMI/ISO and the ESH-IP, requiring 85 and 33 subjects, respectively. A smaller sample size can reduce the cost of validation studies [14]. However, a smaller sample size also decreases the study power and accuracy and does not allow subgroup evaluation, e.g. for different cuff sizes, age groups, or other special populations [14].

A calculation of the power of studies with different sample sizes, which was performed specifically for this paper by Colin Wu, a US NIH biostatistician with extensive experience, showed that a study with sample size of N=35 is:

1. adequate for a ‘high accuracy’ device (defined as mean BP difference between reference and test device measurement and its associated standard deviation 0 ± 3-6 [mean±SD] mmHg), as it would have <14% chance to fail;

2. adequate for a ‘low accuracy’ device (difference 6-8 ± 5 mmHg, or 0 ± 10-12 mmHg, or 4-6 ± 8 mmHg), as it would have 94% chance to fail;

3. inadequate for a ‘moderate accuracy’ device (difference 4±5 mmHg), as it would have 28% chance to fail, which is unacceptably high.

On the other hand, with an N=80 study, a ‘moderate accuracy’ device (difference 4±5 mmHg) has 18% chance to fail, and this is only marginally improved with N=90 (17%).

Given that many of the BP monitors currently available are at the ‘moderate accuracy’ level, the N=85 sample size that has been utilized in previous versions of AAMI, ISO and BHS standards appears to be reasonable, and also necessary to allow any consideration of cuff-size stratified or other subgroups and special population evaluations.

CONSENSUS

• At least 85 subjects are required for an AAMI/ISO/ESH validation study

Cuff-sizes stratified subgroups

This is a necessity as devices often come with two or more cuffs. Although according to the formal sample size calculation described above an N=85 study is optimal for each cuff, in practice this requirement is unrealistic. It was agreed that a compromise would be to accept cuff-size stratified subgroups. These subgroups are not intended for separate analyses (per cuff size), but only to ensure an even representation of all cuffs with a minimum number of participants. Indeed, these stratified subgroups may hide or smooth the differences observed with one cuff. Thus, the mean test-reference BP difference and SD per cuff shall be reported.

For test devices that have multiple (n) cuffs, each cuff shall be tested on at least 1/(2xn) of the subjects, ≥40% of the subjects shall have arm circumference within the upper half of the specified range of use of the cuff, ≥40% within the lower half.

A proposal was considered for more controlled investigation of cuffs in validation studies, which might include the following: (i) a minimum of 22 subjects per cuff, which means that 4 cuffs could be evaluated in a 88-subject study, and (ii) for test devices that have a single cuff, ≥40% of the subjects shall have arm circumference within the upper half of the specified range of use of the cuff, ≥40% within the lower half, ≥20% within the upper quarter, ≥20 % within the lower quarter, ≥10% within the upper octile, and ≥10% within the lower octile.

CONSENSUS

• There is a minimum number of subjects to be tested per cuff depending on the number of the test device cuffs. Cuff subgroups are not intended for separate analyses.

• Requirements are set for the distribution of the participants’ arm circumference according to the specified range of use of the test device.

General population and special populations studies

The AAMI/ESH/ISO protocol should be applicable not only in general population samples with normal or high BP, but also in special populations, in which there is theoretical and clinical evidence of different accuracy of BP monitors.

Special population studies with smaller sample sizes should be performed only after a full general population study has been successfully completed. If the device is intended only for a special population, then a full 85-subject study is required. Special population study data should be analyzed and reported independently of the general population study data. There are no specific criteria (pass/fail requirements) defined for special populations, apart from pregnancy.

Definition of general population

Consensus was reached that a general population study should include only subjects older than 12 years, untreated or treated. The source of recruiting both hypertensive and normotensive subjects should be reported. An N=85 adults study shall include ≥30% males and ≥30% females, and shall have ≥5% of the reference systolic BP readings ≤100 mmHg, ≥5% with ≥160 mmHg, ≥20% with ≥140 mmHg, and ≥5% of reference diastolic BP readings ≤60 mmHg, ≥5% with ≥100 mmHg, and ≥20% with ≥85 mmHg [11].

Definition of special populations

The following are regarded as special populations: (i) age <3 years, (ii) pregnancy including pre-eclampsia, (iii) arm circumference >42 cm, (iv) atrial fibrillation. There is no agreed procedure for BP monitor validation in atrial fibrillation. Subjects aged 12-21 or >80 years and those with end-stage renal disease were considered as possible special groups, but there was uncertainty on the adequacy of existing data suggesting altered accuracy of the BP monitors in these groups.

Sample size for special population studies

According to the formal sample size calculation described above a sample of 85 subjects is desirable for each special population. In case that an independent general population 85-subject study has been completed successfully, a compromise was agreed to accept a minimum of 35 special population subjects (45 for pregnancy).

BP distribution criteria for special population studies

Those of general population studies cannot be applied but need to be defined for each special population because of their different usual BP levels (e.g. children, pregnancy, atrial fibrillation).

Pediatric studies

For devices intended for general population and children, 35 subjects aged 3-12 years can be included together with 50 subjects aged >12 years, and the BP distribution criteria apply to the total 85-subject study. In such studies, further to the formal analysis of the total 85-subject sample, the mean systolic and diastolic BP difference (test versus reference device) and their SD (Criterion 1) shall also be reported separately for subgroups aged 3-12 and >12 years. For devices with a special BP measurement mode for children, 35 subjects aged 3-12 years shall be included and these are exempt of BP distribution requirements. Korotkoff K5 shall be used for reference diastolic BP. If K1 or K5 are not audible the child shall be excluded.

Pregnancy and pre-eclampsia

Include 45 women in 2^nd and 3^rd trimesters of pregnancy, of whom 15 with pre-eclampsia, defined as elevated systolic BP ≥140 mmHg and/or diastolic ≥90 mmHg with proteinuria, 15 with gestational hypertension (new onset in pregnancy with systolic BP ≥140 mmHg and/or diastolic ≥90 mmHg without proteinuria), and 15 normotensives. Chronic hypertension is not included as an additional group because (i) the hemodynamics are similar to gestational hypertension and (ii) its diagnosis is often retrospective as it presents as gestational hypertension. Korotkoff K5 shall be used for reference diastolic BP. Age criteria and BP distribution criteria will not be applied. The pass/fail criterion 1 of the ANSI/AAMI/ISO 81060-2:2013 (mean difference of test versus reference BP measurements ≤5 mmHg with SD ≤8 mmHg for systolic and diastolic BP) [11] will be applied in the 45 women sample. Data from pre-eclamptics (mean difference and SD) shall be reported separately to allow comparison across studies.

CONSENSUS

• A general population study should include only subjects older than 12 years.

• Special populations include at least: (i) age <3 years, (ii) pregnancy including pre-eclampsia, (iii arm circumference >42 cm), (iv) atrial fibrillation. Other special populations may be added as special groups.

• Special population studies to include ≥35 subjects, provided that a general population study has been completed successfully. For special populations BP distribution criteria to differ from those of general population studies. Data to be analyzed independently of general population study data.

• Studies in pregnancy to include 45 women of whom 15 with pre-eclampsia, 15 with gestational hypertension, 15 normotensive. Korotkoff K5 shall be used for reference diastolic BP.

• For devices intended for adults and children, 35 subjects aged 3-12 years can be included and analyzed together with 50 subjects aged >12 years. Mean BP difference and SD shall also be reported separately for age 3-12 and >12 years groups. Korotkoff K5 shall be used for reference diastolic BP.

Method for BP data collection

The same arm sequential method (table 1) is by far the most well-studied and supported by all protocols [1–11]. Thus, this is the preferred method for BP data collection. As the test device cuff may not fulfill the requirements for reference auscultatory BP measurement and some devices have fast deflation rate or measure BP during inflation, it was agreed that the same arm simultaneous method is no longer included as a possibility. The opposite arm simultaneous method will be retained as presented in the ANSI/AAMI/ISO protocol [11].

Table 1.

Procedure for reference and test device BP measurements in same arm sequential validation method.

INITIAL BP MEASUREMENTS*
1.	Take reference BP measurement by the 2 observers	R0
2.	Take test device BP measurement	T0
VALIDATION BP MEASUREMENTS FOR ACCURACY EVALUATION
3.	Take 1st reference BP measurement by the 2 observers	R1
4.	Take 1st test device BP measurement	T1
5.	Take 2nd reference BP measurement by the 2 observers	R2
6.	Take 2nd test device BP measurement	T2
7.	Take 3rd reference BP measurement by the 2 observers	R3
8.	Take 3rd test device BP measurement	T3
9.	Take 4th reference BP measurement by the 2 observers	R4

^*Measurement R₀ shall not be used in the evaluation of reference BP distribution and variability criteria. Measurements R₀ and T₀ shall not be used in the evaluation of the test device accuracy.

CONSENSUS

• The same arm sequential BP measurement is the preferred method for validation.

• The same arm simultaneous method has been eliminated.

Reference BP measurement and validation procedure

The auscultatory standard is retained for reference BP measurement with measurements taken simultaneously by two trained observers blinded to each-other’s readings and to the measurements taken with the test device (Y-tube connected double stethoscope; observers qualified for their agreement according to the BHS protocol criteria within 12 months before the validation [3]; baseline and repeat audiogram every 3 years). Korotkoff K1 shall be used for reference systolic BP and K5 for diastolic BP in all subjects, including adults, adolescents, children aged ≥3 years, and pregnancy. If K1 or K5 is not audible the subject shall be excluded. A third observer (supervisor) is necessary to supervise the validation procedure, the adequacy of reference and test device BP measurements, the agreement between the two observers who should be unaware of the magnitude or direction of their disagreement, and any other issue during the validation procedure.

The validation procedure starts with the subject seated comfortably and relaxed for at least 5 min, her/his back and arm supported with the middle of the upper arm at heart level, legs uncrossed and feet flat on the floor. Talking and any other interference needs to be avoided throughout the entire validation procedure. The sequential method (Table 1) requires a reference BP measurement (R₀) taken by the two observers, followed by a test device measurement (T₀) to confirm the device function. Then 4 reference BP measurements follow, alternated by 3 test device measurements (R₁-T₁-R₂-T₂-R₃-T₃-R₄). Measurements will be performed with at least 60-second intervals.

The supervisor will review each pair of test/reference BP measurements. If one of them (test or reference BP) has to be excluded (due to test device failure or observers’ disagreement >4 mmHg in systolic or diastolic BP), then another pair of BP readings (test and reference) is taken. A maximum of 8 pairs is allowed (4 additional pairs). Observers’ BP comparison during the validation study shall be reported (mean difference, SD and range), together with the number of repeated measurements. Subjects with systolic BP difference >12 mmHg and/or diastolic >8 mmHg in any 2 of the 4 reference (average of two observers) BP measurements (R₁ to R₄) shall be excluded.

Because of concern with mercury toxicity, in many countries mercury devices are not available, or the maintenance of mercury sphygmomanometers is very difficult. Therefore, reference BP measurements can be obtained using mercury sphygmomanometers, or non-mercury manometers (aneroid or other) that fulfill the ISO 81060-1 requirements for accuracy (maximum permissible error shall be ±1 mmHg) [9]. The accuracy of non-mercury devices shall be evaluated against a mercury sphygmomanometer or a calibrated and certified pressure device at the beginning of each validation study. National metrology institutes and other institutions might provide such calibration services.

The cuffs used for reference auscultatory BP measurement shall have an inflatable bladder length that covers 75-100% of the arm circumference of each subject and width that covers 37-50% of the arm circumference. The test device cuffs shall not be used for reference BP determination. Detailed description of cuffs used for reference BP measurement shall be reported in each study (manufacturer, construction, bladder dimensions). An example of cuff use (inflatable bladder dimensions) for reference auscultatory BP measurement in a general population validation study including children is shown in Table 2.

Table 2.

Example of cuff use (inflatable bladder dimensions) for reference auscultatory BP measurement in a general population validation study including children.

Bladder	Participant’s mid-arm circumference (cm)
dimensions	12-15	15-18	18-23	23-28	28-35	33-42
Length (cm)	12	15	18	23	28	33
Width (cm)	6	7	9	12	14	16

CONSENSUS

• Reference BP measurement to be performed with mercury sphygmomanometers or accurate non-mercury devices. The accuracy of non-mercury devices shall be evaluated at the beginning of each study.

• Detailed description of cuffs used for reference BP measurement shall be provided.

• The test device cuffs shall not be used for reference BP determination.

Validation criteria and reporting

Each of the reference BP measurements (R₀-R₄) is the average of the simultaneous readings of the two observers. Each of the test device measurements is compared against the average of the previous and next reference BP reading (e.g. T₁ versus the average of R₁-R₂, T₂ versus average of R₂-R₃, T₃ versus average of R₃-R₄). Differences are calculated by subtracting the reference BP measurement from the test device measurement. The mean BP difference (test versus reference device) and its SD, i.e., Criteria 1 and 2 of the ANSI/AAMI/ISO 81060-2:2013 [11] will be applied for the AAMI/ESH/ISO validation data evaluation. The same criteria will be used for systolic and diastolic BP measurements. In studies including children, further to the total 85-subject analysis, the mean systolic and diastolic BP difference and their SD (Criterion 1) shall also be reported separately for subgroups aged 3-12 years and >12 years.

The number of absolute BP differences (test BP minus average of previous and next reference BP readings) within 5, 10 and 15 mmHg used by the ESH-IP [8] shall also be reported. This categorization is important so that the validation data can be understood by a variety of potential users, i.e. clinicians, public, industry, etc. Standardized Bland-Altman scatterplots as shown in Figure 1 [8] will be presented in the AAMI/ESH/ISO validation study report.

Figure 1.

Standardized Bland-Altman scatterplots as shown below will be presented in the AAMI/ESH/ISO validation study report (with permission from ref. [8] modified).

CONSENSUS

• The mean BP difference (test versus reference) and its SD, Criteria 1 and 2 of the ANSI/AAMI/ISO 81060-2 [11], to be applied for systolic and diastolic BP.

• The number of absolute BP differences within 5, 10 and 15 mmHg and standardized Bland-Altman scatterplots will be presented.

Validation of other BP monitors

• Separate validation protocols need to be developed for specific functions of certain BP monitors, including continuous, cuffless and central BP monitors. Task groups have been established to explore the methodology of such protocols.

• Devices that have more than one BP measurement mode (e.g. auscultatory and oscillometric) require separate validation of each mode in an 85-subject study. Devices that have peripheral (brachial) and other BP measurement mode (e.g. central BP) should first be assessed for peripheral BP measurement accuracy using the main 85-subject study.

CONSENSUS

• Separate validation protocols will be developed for continuous, cuffless and central BP monitors.

Quality and reliability of validation study reports

Violations of the validation protocols and incomplete reporting (missing and unclear data) are particularly common and are likely to be missed by the peer-review process of scientific journals [23–26]. Tools and forms (printed and electronic) for reporting complete data from validation studies similar to those in the revised ESH-IP [8] should be developed in order to prevent protocol violations and incomplete reporting.

Measures to ensure transparency in selecting data for inclusion into analysis should be applied i.e. providing reasons for excluding data and stipulating inclusion in chronological order to fill BP ranges (e.g. electronic online patient report forms). This provides safeguards against potential exclusion of inaccurate data when actual sample size exceeds specified sample size and is necessary to fill BP ranges.

CONSENSUS

• Tools need to be developed to prevent protocol violations and incomplete reporting and to secure appropriate and transparent patient and data selection.

• Detailed forms should be developed to fill in all the data from validation studies that need to be reported.

Further procedures.

The objective of this initiative is to satisfy the existing need for a single standard for BP monitor validation to be internationally accepted and used. The AAMI, ESH and ISO experts agreed to develop a universal standard for device validation, as described in this statement. This preliminary document does not present a detailed description of all the aspects of the validation procedure, which will become available as the organizations involved develop the universal protocol in more detail. As soon as the AAMI/ESH/ISO standard is fully developed this will be regarded as the single universal standard, and will replace all other previous standards/protocols.