Reliability of RT3 Accelerometers Among Overweight and Obese Adults

Abstract

Purpose

Accurate and reliable measurement of physical activity plays an important role in assessing effective lifestyle interventions for obesity. This study examined reliability of accelerometer-based estimates of physical activity levels of overweight and obese adults before and after a lifestyle weight loss program.

Methods

Participants were overweight and obese (BMI 25–45 kg/m²) members (n=1592; 67% female, 42% African American) of the multi-center Weight Loss Maintenance trial. They wore RT3 accelerometers during waking hours for 7 days at baseline and after a 6-month weight loss intervention that included diet and physical activity recommendations. Moderate to vigorous physical activity (MVPA) and MVPA occurring in bouts ≥ 10 minutes (bout MVPA) were assessed.

Results

At baseline, wear time minimums of 10 and 6 hours/day resulted in similar average minutes·day⁻¹ of MVPA (18.3 and 18.0 minutes) and MVPA bout minutes·day⁻¹ (6.9 and 6.7 minutes). Similar wear times occurred after the weight loss intervention for MVPA (27.0 and 26.8 minutes) and bout MVPA (15.1 and 15.0 minutes). Reliability measurements by Intra Class Correlation (ICC) were larger for 4 versus 2 days·week⁻¹ minimum wear time for both MVPA and bout MVPA (4 day ICCs .27–.44 and 2 day ICCs .19–.38), but there was little increase in ICC comparing 4 (ICCs.27–.44) and 7 days·week⁻¹ (ICCs .30–.46).

Conclusions

Longer wear time requirements did not result in significant increases in reliability. Using 4 days of data with ≥ 6 hours·day⁻¹ of wear time optimized the balance between ICC and participant burden in overweight and obese adults before and after a weight loss intervention. Future investigations using accelerometers to estimate MVPA in overweight and obese samples can consider requiring less monitor wear time.

Introduction

Accurate and reliable measurement of physical activity is needed to assess risk status and compliance with physical activity recommendations, monitor participant progress during behavioral physical activity and weight loss interventions, and determine the effectiveness of such programs. Accelerometers provide an objective measure of physical activity and are increasingly being used in research and population surveillance (12). However, despite the widespread use of accelerometers, the impact of varying amounts of wear time on physical activity estimates has not been well studied.

An important methodological concern is the amount of wear time necessary to reliably estimate physical activity levels. A handful of studies have examined the number of days of data collection needed for reliable estimates, but many of these efforts have focused on children (13). Matthews reported that in adults, fewer days are needed to reliably estimate moderate to vigorous physical activity (MVPA) than to measure inactivity (7).

It is not known whether the required wear time for a reliable estimate varies with the activity level of the sample. Limited evidence suggests that low-active samples have lower reliabilities than more-active samples. For example, a sample of 55 adults wore accelerometers for 2 days on 14 different occasions across a calendar year. As a group, those with lower activity levels had lower reliability, as measured by Intra Class Correlations (ICCs), than those with higher activity levels (5). Because this finding could have implications for population surveillance as well as evaluations of lifestyle interventions, further study is needed.

The purpose of this study was to examine reliability of wear time in a large sample of overweight and obese adults. It is one of the first papers to examine the reliability of accelerometer estimates of both minutes·day⁻¹ of MVPA and estimates of MVPA performed in bouts of 10 minutes or more (bout MVPA). In addition, this paper examined the impact of days·week⁻¹and hours·day⁻¹ wear time criteria on accelerometer-based estimates of MVPA both before and after a weight loss intervention. Finally, we estimated the number of days of wear time needed to reach specific reliability levels.

Methods

Sample

We report on all participants with accelerometer data who underwent screening in the multi-site Weight Loss Maintenance trial (pre-Phase 1) (11). Participants were at least 25 years old and had body mass index (BMI) 25 – 45 kg/m², classifying them as either overweight (BMI 25–29.9 kg/m²) or obese (BMI ≥ 30 kg/m²). They were also at high risk for developing cardiovascular disease, evidenced by taking antihypertensive or cholesterol-lowering medication. Those who lost at least 4 kg by the end of the initial intervention, Phase 1, were invited to participate in Phase 2, a 30-month weight maintenance program that compared two intervention conditions and a self-directed control condition on success in sustaining weight loss.

Phase 1 Weight Loss Intervention

Weight Loss Maintenance - Phase 1 was a 6-month intensive weight-reduction intervention that incorporated recommendations for diet and physical activity. The intervention included weekly group meetings and supplemental individual sessions. Interventionists encouraged participants to gradually increase to 180 minutes·week⁻¹ of moderate or greater intensity physical activity. It was suggested that participants accumulate physical activity in bouts.

The trial was reviewed and approved by an independent Protocol Review Committee appointed by the NHLBI and by each site’s Institutional Review Board. All participants provided written informed consent.

Assessment of Physical Activity

Physical activity was assessed at both time points using the RT3 (Stayhealthy, Inc.), a triaxial accelerometer that provides an objective measure of physical activity. Accelerations are detected from vertical, horizontal, and anterior-posterior planes and converted to counts. Greater acceleration over a given period of time results in higher counts. The instrument has adequate reliability and validity against treadmill walking at different speeds and non-regulated physical activity (8,9).

During a screening visit prior to the start of Phase 1, participants were given an RT3 accelerometer programmed to capture data in one-minute increments. Participants were shown how to wear the monitor above the left hip and asked to wear it during all waking hours for 7 consecutive days. They returned the monitor to the clinic, where it was downloaded and output was checked for wear time. Whenever possible, a full 7 days of data were collected. The minimum requirement was ≥ 10 hours·day⁻¹ for 4 days, including at least one weekend day. When possible, those who did not meet this requirement were asked to wear the accelerometer again. This procedure was repeated at the end of the 6-month Phase 1 weight loss program for those participants who lost 4 kg and were eligible for the Phase 2 weight maintenance program. MVPA was defined as ≥ 3 metabolic equivalents (1). Previously published RT3 cutpoints (> 1316.5 counts/min) were used to identify total minutes of MVPA and minutes of bout MVPA (9).

Statistical Analyses

We limited this analysis to individuals with at least 4 days of data (including at least one weekend day), with at least 6 hours of wear time each day. Descriptive statistics were used to describe these samples at screening (Pre Phase 1) and after the 6-month intervention (Post Phase 1). Average daily MVPA minutes (both total and 10-minute bout minutes) were calculated by averaging daily data within individuals and then computing an unweighted average of these individual participant means. Similarly, we report between-person standard deviations of individual averages. The reliability of average daily MVPA estimates was estimated using Intra Class Correlations (ICC), which represents the between-person variance/total variance. The ICC was derived from repeated measures analyses of variance (SAS PROC MIXED) using daily activity counts for each individual and controlling for the class variables of clinical center/site.

We examined ICC for the first 4 days (including at least 1 weekend day) using different wear time minimums (≥ 10 hours ·day⁻¹ and ≥ 6 hours ·day⁻¹) both Pre and Post Phase 1. Next we examined ICC for those participants with 7 days of data with a minimum of 10 hours·day⁻¹. These analyses were repeated examining only the first 4 days (including at least 1 weekend day) and again for the first 2 days of monitor wear. The latter ICC was used for the Spearman-Brown prophecy formula (3, 10) to determine number of days to achieve specific reliabilities. The Spearman-Brown prophecy formula was used to determine the number of days need for desired reliabilities of .70, .80, and .90, where [days of monitoring needed for desired reliability] = [ICC_desired/(1-ICC_desired)][(1-ICC_estimated)/ICC_estimated] (2, 5, 14).

Results

Demographics

Characteristics of the participants in the Pre Phase 1 (n = 1592) and Post Phase 1 (n=1070) samples are presented in Table 1. Both Pre and Post Phase 1 samples included at least 47% over the age of 55, 60% females, and 33% African Americans.

Table 1.

Baseline demographic characteristics of sample pre and post lifestyle intervention

Total	Pre Phase 1 N=1592 (%)	Post Phase 1 N=1070 (%)
Age (years)
≤ 45	13.8	10.9
46–55	38.8	38.1
56–64	32.5	35.0
≥ 65	14.8	16.1
Gender
Male	33.3	35.9
Female	66.7	64.1
Race
African American	42.3	36.6
Non- African American	57.7	63.4
BMI
Normal	0.0	6.9
Overweight	21.2	39.4
Obese	78.8	53.7

Note: Pre Phase 1 was screening data. Post Phase 1 was after 6 months of weight loss intervention.

Wear Time: Hours·day⁻¹

Physical activity estimates and the reliability of these estimates were similar for both the 10-hour and 6-hour minimum wear times. All calculations were performed using the first 4 days (ensuring at least 1 weekend day) that met the daily wear time criteria. In Pre Phase 1, mean MVPA estimates using a 10-hour minimum were similar to those using a 6-hour minimum (18.3 minutes·day⁻¹ and 18.0 minutes·day⁻¹), and both methods resulted in an ICC of 0.40. Table 2 shows that estimates of bout MVPA also were similar across wear time minimums (6.9 minutes·day⁻¹ and 6.7 minutes·day⁻¹), and both methods resulted in an ICC of 0.33.

Table 2.

Mean moderate to vigorous physical activity and inter class correlations for different accelerometer wear times pre and post Phase 1

	4 days ≥ 10 hours·day−1			4 days ≥ 6 hours·day−1
	N	M (SD)	ICC	N	M (SD)	ICC
Pre-Phase 1
MVPA	1351	18.3 (25.5)	.40	1592	18.0 (25.3)	.40
MVPA in bouts	1351	6.9 (18.7)	.33	1592	6.7 (18.6)	.33
Post-Phase 1
MVPA	964	27.0 (32.0)	.46	1070	26.8 (32.8)	.47
MVPA in bouts	964	15.1 (26.4)	.43	1070	15.0 (27.1)	.42

Note: Models were adjusted for site. ICC = Intra Class Correlation. MVPA = Moderate to Vigorous Physical Activity. Pre Phase 1 was screening data. Post Phase 1 was after 6 months of weight loss intervention.

Post Phase 1, the 10-hour and 6-hour minimums resulted in similar MVPA estimates and reliabilities (27.0 and 26.8 minutes·day⁻¹; 0.46 and 0.47 ICC). Similar results were found for bout MVPA (15.1 and 15.0 minutes·day⁻¹; 0.43 and 0.42 ICC). We used the 6-hour minimum wear time for the final analyses.

Wear Time: Days needed

We conducted this set of analyses as a subset of the original sample: participants who had 7 days with ≥ 6 hours·day⁻¹ of wear time. Table 3 shows that there were 721 participants meeting this requirement from Pre Phase 1 and 637 from Post Phase 1. ICCs were calculated using 2, 4 (must include 1 weekend day), and all 7 days of wear time. The ICC for the first 2 days of wear time was lower across time points and outcomes (MVPA and bout MVPA). In the Pre Phase 1 sample, the MVPA ICCs were .36 and .37 for 4 days and 7 days, and the bout MVPA ICCs were .29 and .30. In the Post Phase 1 sample, the MVPA ICCs were .44 and .46 for 4 and 7 days, and the bout MVPA ICCs were .39 and .41. A sensitivity analysis was conducted using the 304 participants who had 7 days of accelerometry data at both Pre and Post Phase 1. Results from this sensitivity analysis indicated similar ICCs Pre Phase 1; however, compared to the larger sample, the subsample of 304 individuals had a lower ICC Post Phase 1.

Table 3.

Estimated accelerometer wear time needed to achieve desired levels of reliability for Mean Moderate to Vigorous Physical Activity

	Moderate to Vigorous Physical Activity						Moderate to Vigorous Physical Activity in bouts
	ICC			Days of monitoring required to achieve reliability of			ICC			Days of monitoring required to achieve reliability of
	2 days	4 days	7 days	.70	.80	.90	2 days	4 days	7 days	.70	.80	.90
All Participants
Pre Phase 1* (N=721)	.26	.36	.37	13	23	51	.25	.29	.30	11	20	44
Post Phase 1 (N=637)	.35	.44	.46	9	16	35	.34	.39	.41	9	16	35
Participants with both pre and post values
Pre Phase1* (N=304)	.32	.40	.40	10	17	37	.38	.29	.32	8	13	30
Post Phase1 (N=304)	.19	.30	.36	20	35	78	.19	.27	.32	20	35	78

Note: Models were adjusted for site. Pre Phase 1 was screening data. Post Phase 1 was after 6 months of weight loss intervention

Days of Monitoring Needed to Achieve a Given Level of Reliability

Table 3 also presents estimates, based on the Spearman-Brown prophecy formula, of the number of days needed to achieve reliabilities of .70, .80, and .90 based on the ICC corresponding with 2 days of wear time. The results for MVPA indicate that between 9 and 13 days of data collection are needed for a reliability of .70 and between 16 and 23 days for a reliability of .80 in both the Pre and Post Phase 1 samples. In the Pre Phase 1 sample, bout MVPA reliabilities of .70 and .80 required 11 and 20 days of data collection. In the Post Phase 1 sample, bout MVPA reliabilities of .70 and .80 required 9 and 16 days.

Discussion

This is one of the largest, most diverse samples used to examine reliability in accelerometry data. We found little support for using the more-stringent wear time minimum of ≥ 10 hours·day⁻¹ when estimating MVPA. The 6-hour wear time afforded a larger sample size than the more stringent minimum and resulted in similar physical activity estimates and corresponding reliability. We also found similar results using a 4-day versus a 7-day criterion for wear time. Thus, shorter wear time minimums may provide useful MVPA estimations in populations of sedentary overweight and obese adults.

Studies designed to estimate either total energy expenditure or daily physical activity often require stringent wear time criteria in order to capture all pertinent behavior. (4) Although some studies have required up to 12 hours of wear time (15), there have been few published examinations of the impact of wear time on reliability in estimating minutes·day⁻¹ of MVPA or bout MVPA. It has been suggested that participants are not likely to be engaged in MVPA and even less likely to be engaged in bout MVPA during periods when accelerometers are removed (6). Consequently, wearing the accelerometers during all waking hours may not be necessary when making MVPA estimates. Additionally, requiring long hours (e.g., 10 or 12 hours) of minimum wear time increases participant burden and is not always feasible in large-scale studies. We did not examine minimum wear time needed for reliability estimates of sedentary activities, which is likely to be more stringent than wear time required for reliable MVPA estimates.

It was not surprising to find low levels of activity in this sample of overweight and obese adults in Pre Phase 1. As a result of low activity levels, the between-person variability in activity levels expressed as a percentage of total variance was low, resulting in low ICCs for the Pre Phase 1 sample. In contrast, the between-person variability would account for a larger portion of the overall variability in a sample including both sedentary and active individuals, resulting in a larger ICC. The higher ICCs Pre compared to Post Phase 1 could be due to an increase in between-person variance rather than an indicator of more-reliable MVPA and bout MVPA measurement. The results from the sensitivity analyses do not support this hypothesis, however, as the Post Phase 1 ICCs are lower than the Pre Phase 1 ICCs for this group.

Currently there is no gold standard for capturing intensity, duration, and frequency of daily MVPA. Accelerometry is one of the most promising methods for objective recording of daily physical activity. As with other physical activity measurements, there is a degree of measurement error in accelerometer estimates. Although ICC is widely used to describe reliability of measurement, it may not be the best indicator. To date there have been few attempts to develop standardized methods or identify a battery of tests for assessing measurement reliability. Such a collection might include ICC, inter-monitor reliability, and criterion references.

The Spearman-Brown prophecy formula (3, 10) is one of the most widely used techniques for estimating days of measurement needed to achieve a specific reliability. Different studies have operationalized the procedure slightly differently, but the general methods are similar. A common length of time to construct a repeated measure is used to determine an ICC, we used 2 days of daily MVPA or bout MVPA. The Spearman-Brown prophecy formula uses this ICC to determine how many “lengths of time” are required to achieve a specific reliability. Previously reported Spearman-Brown estimates of the number of days needed to achieve reliability of .80 range from 3 to 10 days (5, 13). We found that between 16 and 23 days of monitoring would be needed to achieve reliability of .80. Reliability increased when comparing 2 to 4 days of monitoring, yet there was negligible increase in ICC when wear time was increased to 7 days. As such, 4 days of monitoring appeared to be an optimal tradeoff between reliability and participant burden for wearing the monitor on multiple days. Because the Spearman-Brown prophecy formula relies on between-person variance, there is concern when using it to estimate the number of days needed for reliable activity estimates (5). Additionally, the formula assumes that reliability increases with increased “length of time”. In this study we found that the reliability did not increase in the same proportion as predicted by the formula, thus highlighting another limitation of using Spearman-Brown prophecy formula for determining reliability of accelerometer data. Generalizing Spearman-Brown estimates across populations should be done with caution and should be secondary to finding actual reliability estimates to determine needed wear time for specific reliabilities.

There are known limitations to accelerometer-based activity estimates. In general, accelerometers capture walking and jogging better than other activities (4), and the nature of the activity may affect reliabilities. Accelerometers need to be taken off before swimming, and they do not accurately estimate bicycling because of lack of hip movement while cycling. However, based on self report there were low rates of these activities in this sample (data not shown). Reliability of estimates could also be influenced by the accuracy of the monitor (17) and completeness of staff training, both of which received substantial attention in this trial. While compliance rates were high across all data collection sites, some of our analyses were conducted on a subsample with the highest compliance. This subsample was larger (n=637) than most samples previously used to examine reliability with adults and is one of the largest reported samples of overweight and obese adults with accelerometer data. While this paper did not test the impact of different wear time instructions, we did examine how data processing decisions affect reliability and sample size. As such, we believe these results are applicable to future study designs.

This paper examined the relationship between wear time requirements and optimal use of data collected when developing reliable physical activity estimates. More stringent (longer) wear time requirements resulted in lower sample sizes without a significant increase in reliability. It has been suggested that researchers seek a balance between feasibility and what is needed to answer the research question at hand (16). Using 4 days of data with ≥ 6 hours·day⁻¹ of wear time optimized both ICC and sample size. Hence, we believe it resulted in the most reasonable estimation of MVPA and bout MVPA in both the Pre and Post Phase 1 samples. These results should help guide future efforts to use accelerometers in estimating MVPA in diverse overweight and obese samples.