Test–retest reliability of pulse amplitude tonometry measures of vascular endothelial function: Implications for clinical trial design

Abstract

Endothelial dysfunction is an important outcome for assessing vascular health in intervention studies. However, reliability of the standard non-invasive method (flow-mediated dilation) is a significant challenge for clinical applications and multicenter trials. We evaluated the repeatability of pulse amplitude tonometry (PAT) to measure change in pulse wave amplitude during reactive hyperemia (Itamar Medical Ltd, Caesarea, Israel). Twenty healthy adults completed two PAT tests (mean interval = 19.5 days) under standardized conditions. PAT-derived measures of endothelial function (reactive hyperemia index, RHI) and arterial stiffness (augmentation index, AI) showed strong repeatability (intra-class correlations = 0.74 and 0.83, respectively). To guide future research, we also analyzed sample size requirements for a range of effect sizes. A crossover design powered at 0.90 requires 28 participants to detect a 15% change in RHI. Our study is the first to show that PAT measurements are repeatable in adults over an interval greater than 1 week.

Introduction

It is now well established that vascular endothelial dysfunction is positively associated with traditional cardiovascular disease (CVD) risk factors,¹ and independently predicts cardiovascular events over intervals of 1–6 years.^2–7 Vascular dysfunction is evident in high-risk populations as early as adolescence,⁸ and intervention studies have shown that it is responsive to dietary,^9,10 behavioral,¹¹ pharmacological¹² and biomechanical interventions.^13,14 The gold standard, non-invasive measurement of endothelial function (flow-mediated dilation, FMD) uses ultrasound to measure changes in brachial artery diameter in response to reactive hyperemia.¹⁵ Many laboratories (including our own) have shown robust test–retest reliability for FMD when conditions are rigorously standardized.^16–18 However, the technique requires specialized training and high-resolution sonography equipment. Thus, it is expensive and highly operator-dependent. Many consider FMD testing to be impractical for either large-scale clinical trials or clinical use.^19,20

Pulse amplitude tonometry (PAT), a recently FDA-approved method which is relatively inexpensive and operator-independent, is increasingly being used as an alternative measure of endothelium-dependent dilation in response to reactive hyperemia.² The PAT device records digital pulse wave amplitude (PWA) using fingertip plethysmography (EndoPAT; Itamar Medical Ltd, Caesarea, Israel).²¹ PWA is measured continuously during three phases: a quiet baseline period, 5-min forearm occlusion, and reactive hyperemia following cuff release. Unlike FMD, PAT testing is not dependent on a highly skilled technician and post-test analysis is largely automated. Most importantly, one longitudinal study has shown that PAT measures of endothelial function predict CVD events over a 6-year follow-up period.² These significant advantages may make PAT testing suitable for clinical practice if prognostic significance and reliability can be verified. In the text below, we define and discuss each of the PAT-derived measures of vascular function before considering their reliability under controlled laboratory conditions.

Reactive hyperemia scores

The reactive hyperemia index (RHI) measures nitric-oxide dependent changes in vascular tone.²² The continuous monitoring of blood volume reaching the finger tips allows the hyperemic response to be quantified as a ratio, comparing PWA from pre to post occlusion. As seen in Figure 1, RHI is calculated as follows: the ratio of the occluded arm's mean PWA 90–150 seconds post occlusion (A) to the mean PWA from baseline readings of the same arm (B). The result is further divided by the same ratio from the control arm (C/D), which allows the device to account for systemic vascular changes during testing. The final ratio is then multiplied by a proprietary baseline correction factor (Itamar Medical Ltd). An alternative reactive hyperemia score, proposed by Framingham Heart Study researchers (fRHI), uses the natural logarithmic transformation of the RHI ratio, does not include the baseline correction factor, and utilizes only the readings from 90 to 120 seconds for post occlusion.¹ RHI and fRHI are highly correlated (r's = 0.92 to 0.97 in this sample); however, their relationships to CVD risk factors may differ in important ways,¹ and researchers should be encouraged to present both measures in future studies.

Figure 1.

PAT control and occlusion arm pulse amplitude tonometry recordings and calculation of the reactive hyperemia index (RHI).

Augmentation index scores

As a pressure wave moves through the arterial tree, it encounters impedance resulting in a reflected wave that moves back toward the heart and may augment peak systolic pressure.²³ Arterial stiffness increases pulse wave velocity, causing early reflection of this waveform.²⁴ The EndoPAT-derived augmentation index (AI) provides a measure of arterial stiffness by considering the timing and magnitude of this wave reflection in the digital pulse.²⁵ Calculated from baseline resting pulse waves, AI represents the relative contribution of augmented pressure due to wave reflection to the pressure wave form. Proprietary software automatically identifies inflection points distinguishing the systolic peak (P₁) and the reflected peak (P₂) for the calculation of this ratio and converts it into a percentage (P₁–P₂/P₁*100).²⁴ Because AI is inversely related to heart rate,²⁶ values are sometimes mathematically adjusted to represent arterial stiffness at a standard heart rate of 75 beats per minute (AI@75). PAT arterial stiffness measures are associated with abnormal ventricular–vascular coupling²⁴ and correlate well with AI measures from other devices.²⁷

Reliability of PAT scores

Although these tests have been widely adopted by investigators, relatively few studies have examined the test–retest reliability of PAT-derived indices of vascular health.^8,14,28 Importantly for clinical researchers, no previous studies have used these data to provide sample size calculations for a wide variety of study designs and hypothesized treatment effect sizes. For example, one of the most widely cited papers on repeatability of RHI utilizes a Bland–Altman plot to demonstrate test–retest consistency.¹⁴ While these plots provide a clear visual representation of reliability of paired measurements, they allow neither sample size calculations nor direct comparison of reliability metrics across studies. Precision of the sample size estimate is particularly important for crossover studies, in part because the standard deviation of change from one test to the next (SD_within) is not often provided in published manuscripts. Therefore, the current study was designed to provide reliability data on PAT measures of endothelial function and arterial stiffness, including newly proposed metrics derived from the PAT signal. Our goal is to inform the design of future clinical trials by providing sample size calculations for a wide range of experimental conditions.

Methods

Participants

Characteristics of the sample are shown in Table 1. The sample was comprised of 20 disease-free but overweight, normotensive participants (14 men and six women). Participants were recruited for a study of antioxidant effects on vascular function, and only the fasting vascular tests are reported here. Age ranged from 31 to 63 years with a mean of 41.2 ± 2.4. The average BMI was 30.5 ± 0.86. Exclusion criteria included: tobacco use, fasting blood glucose > 126 mg/dl, blood pressure > 160/100 mmHg, clinically significant arrhythmia, history of heart attack, stroke, renal or hepatic disease. Use of the following also resulted in exclusion: anti-inflammatory medication, statins, anti-hypertensive medications, hormones, daily aspirin, non-SSRI psychotropic medication and dietary supplements/vitamins. Participants were excluded if they were unable to tolerate wheat, gluten or certain spices due to dietary intervention demands that were required as part of the larger clinical trial.

Table 1.

EndoPAT scores and metabolic parameters measured in a healthy sample across days

	Visit 1	Visit 2	Average CV (%)	Mean variability |t1–t2|	Pearson's r	ICC
RHI	1.81 ± 0.12	1.82 ± 0.12	12.22 ± 2.19	0.31 ± 0.06	0.68	0.74
fRHI	0.29 ± 0.09	0.34 ± 0.1	–	0.23 ± 0.04	0.77	0.77
AI	–4.23 ± 3.18	–2.53 ± 4.17	–	5.40 ± 1.85	0.89	0.83
AI@75	–11.16 ± 3.29	–8.81 ± 4.25	–	5.60 ± 2.04	0.88	0.81
Heart rate (beats/min)	63.9 ± 1.5	64.9 ± 1.7	3.6 ± 0.7	3.3 ± 0.6	0.85a	–
Systolic BP (mmHg)	121.9 ± 2.3	121.3 ± 2.0	4.3 ± 0.8	7.4 ± 1.3	0.54a	–
Diastolic BP (mmHg)	80.2 ± 1.5	81 ± 1.2	3.2 ± 0.8	3.6 ± 0.8	0.79a	–
C-reactive protein (nmol/l)	16.0 ± 3.5	13.7 ± 2.3	22.2 ± 5.5	5.7 ± 1.9	0.77a	–
Insulin (pmol/l)	57.9 ± 7.9	64.1 ± 14.0	29.9 ± 5.8	32.6 ± 10.4	0.70	–
Glucose (mmol/l)	5.01 ± 0.14	4.99 ± 0.19	6.9 ± 1.3	0.51 ± 0.09	0.19a, ns	–
Total cholesterol (mmol/l)	4.54 ± 0.15	4.62 ± 0.15	5.5 ± 1.0	0.36 ± 0.06	0.78	–
HDL cholesterol (mmol/l)	1.11 ± 0.05	1.10 ± 0.06	4.7 ± 1.0	0.07 ± 0.02	0.92	–
LDL cholesterol (mmol/l)	2.68 ± 0.13	2.67 ± 0.13	9.9 ± 1.5	0.37 ± 0.06	0.71	–
Total/HDL cholesterol (mmol/l)	0.11 ± 0.005	0.11 ± 0.005	5.5 ± 1.1	0.009 ± 0.002	0.90	–
Triglycerides (mmol/l)	1.65 ± 0.11	1.86 ± 0.17	12.6 ± 2.2	0.34 ± 0.08	0.89	–

Data given as mean ± SEM.

^aSpearman rank order correlation. All correlations were statistically significant (p < 0.05), except for glucose.

CV = (100 × SD)/mean; ICC, intra-class correlation, calculated as S_b² – S_w² / S_b² + S_w²; RHI, reactive hyperemia index; fRHI, Framingham reactive hyperemia index; AI, augmentation index; AI@75, augmentation index at a heart rate of 75 beats per minute; BP, blood pressure; HDL, high-density lipoprotein; LDL, low-density lipoprotein.

Procedures

Subjects underwent two EndoPAT tests with a mean interval of 19.5 ± 6.2 days, but not less than 7 days. Tests occurred at approximately 11 a.m., after a standardized low-fat, low-antioxidant breakfast (low-fat cream cheese and a white bagel) had been consumed between 7 and 8 a.m. Subjects consumed the breakfast after completing a 12-hour overnight fast. Premenopausal females (n = 2) were tested during the first 7 days of their menstrual cycle. The protocol from which these fasting data were taken required that participants limit their consumption of high-antioxidant foods 48 hours prior to testing (a list was provided). Food and beverage records were reviewed by study personnel to assure compliance. The study protocol was reviewed and approved by The Pennsylvania State University Institutional Review Board and all participants gave written informed consent.

PAT tests were performed in a supine position, in a dimly lit and temperature-controlled room (70–75°F [21–24°C]). After application of the occlusion cuff to the right forearm and finger tip probes to the index fingers of each hand, the study began with a 10-min quiet rest period. Measures of PWA and heart rate were captured continuously by pneumatic finger probes throughout testing, as outlined above. Using PWA recordings, proprietary EndoPAT software calculated the RHI, fRHI, AI and AI@75. Seated blood pressure measurements were obtained using a Dinamap oscillometric device (GE Healthcare, Fairfield, CT, USA) before each PAT test and blood samples were taken immediately following PAT testing.

Statistical analysis

Analyses were conducted with SAS v. 9.2 (Cary, NC, USA). Reported variability measures include: (1) Pearson's correlations for measures across visits; (2) mean variability expressed as the absolute value of the difference between visit 1 and 2 scores; and (3) the intra-class correlation (ICC) calculated using a conventional formula: S_b² – S_w²/ S_b² + S_w², where ‘S_b²’ represents between-subjects variance and ‘S_w²’ represents within-subjects variance.²⁹ As in previous studies,^28,30,31 the ICC was computed using raw, non-normalized data for each individual and a mean ICC is reported for each variable. The coefficient of variation (CV) was also appropriate for RHI given its non-negative values³² and was therefore calculated using the following equation: CV = (100 × SD)/mean. Variables were tested for normality and transformed when appropriate. Of the four PAT scores, all but fRHI required transformation. For normally distributed variables, the Pearson correlation was used. When distributions remained non-normal post transformation, the Spearman rank order correlation was used on the original data.

Sample size calculations were computed using two-tailed tests: α = 0.05 (CI = 95%), β = 0.80 or 0.90. Consistent with previous reports,^33,34 sample size calculations for the crossover design were performed using SD = √2 × SD_within subject, where SD_within subject = √MSE as provided by a one-way ANOVA with subject as a main effect. All values are reported as mean ± SEM unless otherwise indicated.

Results

Reliability of EndoPAT measures

Table 1 lists blood chemistry, blood pressure and heart rate measures of participants and reliability calculations for EndoPAT measurements taken on two separate days using carefully standardized procedures. No statistically significant differences between days were observed for any of these variables. Similarly, all measures correlated well across visits with the exception of glucose (p = 0.43). Variability in glucose was not correlated with change in any PAT score (data not shown). Values for RHI and fRHI were similar to those from a recent sample with similar BMI, age, and lipid profiles.³⁵ All four EndoPAT scores were repeatable across days. RHI scores were correlated across the two testing days (r = 0.68, p < 0.001; Figure 2). Reliability estimates for RHI (ICC = 0.74 and CV = 12.2 ± 2.2) were within acceptable ranges. The fRHI appeared to be slightly more reliable (ICC = 0.77) and, as seen in Figure 2, the correlation between visit 1 and visit 2 scores for fRHI was stronger (r = 0.77) than that of RHI (r = 0.68). Of the measures of arterial stiffness, both AI (ICC = 0.83) and AI@75 (ICC = 0.81) were found to be highly reliable. Correlations for AI and AI@75 collected across the two testing days were high (0.89 and 0.88, respectively).

Figure 2.

PAT reactive hyperemia index (RHI) scores and Framingham reactive hyperemia index (fRHI) scores plotted by visit. (A) Plot of paired measurements of log RHI illustrating a significant correlation (r = 0.68, p < 0.001. (B) Plot of paired measurements of fRHI also demonstrate a significant correlation (r = 0.77, p < 0.0001).

Table 2 compares our reliability statistics to the small number of published studies that have assessed test–retest reliability of EndoPAT measures. The studies that we selected met two criteria: (1) they were specifically designed to test reliability, and (2) they presented adequate data to allow comparison to the present study. Previous studies reported ICCs of 0.78 and 0.73 for repeated RHI measurements across days in healthy participants (Table 2). The ICC from our sample was 0.74, indicating robust reliability, in keeping with previous studies. We found that our estimate of test–retest reliability for fRHI (0.77) was also similar to that found in a previous study (0.88; Table 2).

Table 2.

Comparison studies reporting test–retest analysis for the reactive hyperemia index (RHI) and Framingham reactive hyperemia index in healthy subjects

	Current studya	Selamet Tierney et al., 200930,b	Tomfohr et al., 200828,b
Sample size	20	30	12
Age, years	41.2 ± 2.4	17.3c	26.8
RHI
Visit 1, mean ± SEM	1.81 ± 0.12	1.91 ± 0.10	–
Visit 2, mean ± SEM	1.82 ± 0.12	1.78 ± 0.09	–
Mean CV	12.2 ± 2.2	–	–
Mean variability |t1–t2|	0.31 ± 0.06	0.43	–
r-value t1 vs t2	0.68	–	0.76
Intra-class correlation	0.74	0.78	0.73
fRHI
Visit 1, mean ± SEM	0.29 ± 0.09	0.57 ± 0.08	–
Visit 2, mean ± SEM	0.34 ± 0.10	0.49 ± 0.07	–
Mean CV	1.59 ± 0.27	–	–
Mean variability |t1–t2|	0.23 ± 0.04	0.34	–
r-value t1 vs t2	0.77	–	–
Intra-class correlation	0.77	0.88	–

Comparison literature standard deviations converted to SEM, SEM = SD/√n.

^a3 hours post standardized breakfast preceded by 12-hour fast

^bpost 12-hour fast

^cage expressed as a median.

–, indicates that information was not reported in manuscript.

Sample size for experimental design

Table 3 contains sample size calculations for both crossover and parallel design studies, using varying magnitudes of treatment effects for RHI, fRHI and AI@75, with α = 0.05 (confidence interval fixed at 95%), and β = 0.80 or 0.90. For example, a crossover design powered at 0.90 would require 28 participants to detect an absolute change in RHI of 0.25 units (corresponding to ~15% increase in RHI from pre to post treatment). In a parallel arm design, 96 participants would be required to detect the same effect size. For AI@75, a treatment effect of 9 units would require 24 participants to detect at 0.90 power in a crossover design and 95 in a parallel arm design.

Table 3.

Sample size required to detect treatment effects for the RHI, fRHI and AI@75 EndoPAT endpoints in both parallel arm and crossover designs

Magnitude of treatment effect	RHI				Magnitude of treatment effect	fRHI				Magnitude of treatment effect	AI@75
	Parallel arma		Crossoverb			Parallel armc		Crossoverd			Parallel arme		Crossoverf
	Power					Power					Power
	0.80	0.90	0.80	0.90		0.80	0.90	0.80	0.90		0.80	0.90	0.80	0.90
0.05	1765	2363	485	648	0.04	1732	2318	406	543	1.8	1367	1830	271	361
0.1	442	592	123	164	0.08	434	581	103	138	3.6	343	459	69	92
0.15	197	264	56	74	0.12	194	259	47	63	5.4	153	205	32	42
0.2	112	149	33	43	0.16	110	146	28	36	7.2	87	116	19	25
0.25	72	96	22	28	0.20	71	94	19	24	9	56	75	13	17
0.3	50	67	16	20	0.24	50	66	14	18	10.8	39	52	10	13
0.35	37	50	12	16	0.28	37	49	11	14	12.6	29	39	8	10
0.4	29	38	10	13	0.32	29	38	9	11	14.4	23	30	7	8
0.45	23	31	9	11	0.36	23	30	8	9	16.2	18	24	6	7
0.5	19	25	7	9	0.40	19	25	7	8	18	15	20	5	6
0.55	16	21	7	8	0.44	16	21	6	7	19.8	13	17	5	6
0.6	14	18	6	7	0.48	14	18	6	6	20.6	11	14	5	5

RHI, reactive hyperemia index; fRHI, Framingham reactive hyperemia index; AI@75, augmentation index at heart rate of 75 beats/min; calculated using:

^amean ± SD = 1.82 ± 0.53

^bmean = 1.82, SD = √2 × SD_within = 0.392

^cmean ± SD = 0.313 ± 0.42

^dmean = 0.313, SD = √2 × SD_within = 0.287

^emean ± SD = –9.99 ± 16.79

^fmean = –9.99, SD = √2 × SD_within = 10.52.

All α = 0.05, all confidence intervals fixed at 95%.

Discussion

In this sample of 20 overweight (but otherwise healthy) adults, we showed robust reliability of measures collected during PAT testing conducted in the same individuals under controlled conditions. Despite the long list of factors that are known to cause short-term fluctuations in vascular endothelial function, intra-class correlations ranged from 0.74 to 0.83, indicating strong repeatability in this relatively small sample of healthy adults. We provide preliminary evidence that PAT measures may be more repeatable measures than FMD.^16,33,34 Few published studies have directly addressed test–retest reliability of these measures, and there is little consistency in the metrics reported (Table 2). Existing studies of adults have assessed repeatability within a single day,³¹ or across intervals up to 7 days.^14,28,31,36 The current findings illustrate the robust repeatability of these variables across a mean follow-up interval of nearly 3 weeks. This is especially useful for intervention researchers, who attempt to alter and measure changes in endothelial function within the same individual over intervals longer than a few hours or days.

Given the relatively recent introduction of RHI and fRHI as measures of endothelial function, few studies have examined their clinical utility or prognostic significance. After an exhaustive analysis in which many different components of the PAT signal were evaluated, Hamburg and colleagues determined that the fRHI ratio (referred to as the ‘PAT ratio’) had the strongest relationship to known CVD risk factors such as smoking, BMI, diabetes mellitus and ratio of total-to-high-density lipoprotein (HDL) cholesterol.¹ More recent work suggests that RHI is similarly useful; the natural log of RHI joined age and prior coronary bypass as significant independent predictors of late cardiovascular adverse events in 270 outpatients followed over 6 years.² As this field has developed, investigators have proposed new scoring equations for calculating indices of vascular health from the PAT signal. It is critically important that future publications provide complete information about how the vascular measurements were derived from the PAT signal.

PAT measures of augmentation index (AI and AI@75), a commonly reported indicator of vascular stiffness,^24,37 also displayed robust repeatability in this sample. While little work has considered the relationship between this PAT device's arterial stiffness measures and CVD risk, previous literature has found that AI measured by a different device (SphygmoCor) correlates well with CVD risk.^38,39 One study of men referred for coronary angiography found that when compared to the lowest quartile of SphygmoCor AI scores, individuals whose AI score fell within the fourth quartile were four times more likely to have coronary artery disease.⁴⁰ To our knowledge, our study provides the first published data analysis of the repeatability of augmentation index from the EndoPAT device. Power and sample size calculations for these measures should assist future research evaluating new interventions for reducing arterial stiffness.

The present study reports a wide range of test–retest statistics that are particularly useful for the design of crossover studies. The within-subject standard deviation, presented in Table 3, is required for accurate sample size calculations for crossover studies. However, this variable is not typically reported in the existing literature. Given the potential for day-to-day fluctuations in measures of endothelial function, future studies should directly assess statistical power before reporting intervention data. Effect sizes were chosen to reflect the magnitude of change reported in previous intervention studies using the EndoPAT device. For example, one study of mandibular advancement splint use for obstructive sleep apnea resulted in a significant change in RHI of 0.33 units.⁴¹ A separate intervention utilizing a standard 35-hour course of enhanced external counterpulsation in patients with symptomatic coronary artery disease reported a 0.26-unit change in RHI.¹⁴

Study strengths and limitations

Many variables known to acutely influence endothelial function were controlled for in this protocol. These variables include testing during the follicular menstrual cycle phase⁴² (for premenopausal women), standardized time of day⁴³ and room temperature,⁴⁴ limiting postprandial effects,⁴⁵ limiting recent exercise,⁴⁶ and disallowing certain medication use.^47,48 The testing procedure also calls for the standardization of posture, probe placement and a timed resting period to eliminate sympathetic stimulation prior to testing.⁴⁹ As per findings from previous literature, alcohol,⁵⁰ caffeine^51,52 and antioxidant intake⁵³ were also limited in the 48 hours prior to testing. Future studies should consider including these controls for reliable measurement of endothelial function via PAT. However, recent work suggests that even in the absence of stringent controls, RHI scores are a valuable tool for discriminating between patients with and without coronary artery disease.⁵⁴

Limitations of this study include the small sample size, the small number of women (n = 6), the healthy nature of the participants and the limited number of retests. It should be noted that sample size calculations for FMD studies are commonly based on a classic paper which reported reliability over four testing occasions, with a sample size double that of our own.⁵⁵ Finally, many CVD risk interventions require treatment periods of longer than 3 weeks. Future investigations should consider the stability of these EndoPAT measures over longer periods of time.

Sample characteristics are also an important consideration because intervention studies are typically targeted toward individuals at high risk of CVD, diabetes, or other chronic disease. While these participants were overweight and had less than ideal HDL and triglyceride levels, future research should consider the reliability of PAT measures in other populations, particularly in subjects at higher risk of CVD. For example, we previously studied day-to-day variation in FMD in a sample of adults with type 2 diabetes. We found that higher day-to-day variability in glucose and insulin was associated with greater variability in endothelial function.³³ This pattern was not observed in the present study (data not shown), perhaps because of the narrow range of fasting glucose values in these participants.

Conclusion

As one of the precursors to atherosclerosis, endothelial dys-function is a novel target for CVD risk reduction and therefore is of great interest to interventionists. This study is the first to show that PAT measures of endothelial function are highly repeatable across intervals greater than 1 week in healthy adults. In addition, the present study provides some guidance for sample size and power calculations for future intervention design. Specifically, it appears that well-controlled crossover designs of relatively small size (n = 15–30) can detect treatment effects for RHI and fRHI that are plausible given the current literature.