Cross-validation of Waist-Worn GENEA Accelerometer Cut-Points

Abstract

Purpose

The purpose of this study was to determine the classification accuracy of the waist GENEA cut-points developed by Esliger et al. for predicting intensity categories across a range of lifestyle activities.

Methods

Each participant performed one of two routines, consisting of seven lifestyle activities (home/office, ambulatory, and sport). The GENEA was worn on the right waist and oxygen uptake was continuously measured using the Oxycon mobile. A one-way chi-square was used to determine the classification accuracy of the GENEA cut-points. Cross tabulation tables provided information on under- and over-estimations, and sensitivity and specificity analyses of the waist cut-points were also performed.

Results

Spearman’s rank order correlation for the GENEA SVM_gs and Oxycon mobile MET values was 0.73. For all activities combined, the GENEA accurately predicted intensity classification 55.3% of the time, and increased to 58.3% when stationary cycling was removed from the analysis. The sensitivity of the cut-points for the four intensity categories ranged from 0.244 to 0.958 and the specificity ranged from 0.576 to 0.943.

Conclusion

In this cross-validation study, the proposed GENEA cut-points had a low overall accuracy rate for classifying intensity (55.3%) when engaging in 14 different lifestyle activities.

INTRODUCTION

To measure population compliance with the 2008 national physical activity guidelines and progress towards meeting physical activity goals (e.g., as identified in Healthy People 2020), researchers have developed a number of monitoring tools that can be used to determine activity levels within large populations (4). Most large cohort studies use physical activity questionnaires to estimate activity levels, due to their low participant burden and ease of administration. However, a study using data from the U.S. National Health and Nutrition Examination Survey, has found that moderate to vigorous physical activity measured by accelerometer-based activity monitors yields a stronger association with physiological biomarkers (2). This provides strong evidence that objectively-measured physical activity has superior validity, compared to subjective questionnaires. Accelerometer-based physical activity monitors enable researchers to estimate the intensity of activities, towards the goal of developing a more complete picture of physical activity habits.

A number of factors influence the accuracy of activity monitors in classifying physical activity intensities, including the number of monitoring axes, the placement site of the monitor on the body, and the type of activities an individual performs (3). Activity monitor use has increased in physical activity research, and it is therefore important that these devices are accurately calibrated and that researchers analyze the data correctly.

The Gravity Estimator of Normal Everyday Activity (GENEA) is a tri-axial accelerometer-based activity monitor that can be worn on the waist, wrist, or ankle. Recent research has shown the GENEA, whether worn on the waist or wrist, has excellent accuracy for classifying intensity of physical activity (8). Based on data from 16 structured activity bouts, Esliger et al. (8) developed intensity classification (sedentary, light, moderate, vigorous) cut-points for the GENEA worn at the waist. Upon further analysis of the accuracy of their cut-points, Esliger and colleagues (8) reported a criterion validity of r = 0.87 and a classification accuracy of 95%. However, Esliger et al. (8) determined classification accuracy on the same sample used to develop the cut-points, which may have artificially inflated the accuracy rate (10). Thus, the purpose of this study was to determine whether the cut-points developed by Esliger and colleagues for the waist-worn GENEA accelerometer are accurate for predicting intensity categories across a range of lifestyle activities (home/office, ambulatory, and sport), in an independent sample.

METHODS

Adult participants (N=139, age range = 20 to 60 years) from the University of Tennessee, Knoxville (UTENN), the University of Massachusetts, Amherst (UMASS), and surrounding communities were recruited to participate in this study. Upon final analysis, nine participants were excluded for being left-side dominant and 24 participants were excluded due to incomplete data, therefore the final sample totaled 106 participants (n= 64 UTENN; n = 42 UMASS). Inclusion criteria for participation included: 20–60 years of age, apparently healthy, and free from chronic disease or any musculoskeletal injuries that may affect activity. Each participant signed an informed consent form approved by the institutional review boards of the University of Tennessee, Knoxville and University of Massachusetts, Amherst.

Protocol

The study sample consisted of 48% males and 51.9% females (n=106, mean age 39.1 ± 11.2 years old; mean BMI 25.5 ± 4.6 kg/m²). Prior to testing, participants filled out a Physical Activity Readiness Questionnaire (PARQ), Healthy History Questionnaire, and Physical Activity Status questionnaire (PAS). The PARQ was used to assess participants’ current health status to engage in physical activity. The PAS questionnaire was used to determine the types of activities the participants engaged in during their leisure time. This information was used when assigning activity routines. For example, someone who had never played basketball would not be asked to play basketball during their routine as to avoid inaccurate metabolic responses as a result of the new movements and activity. Participants’ height (stadiometer) and weight (Tanita BC-418, Tanita Corporation of America, Inc.; Arlington Heights, IL [UTENN]; physicians’ scale, Detecto; Webb City, MO [UMASS]) were measured and body mass index was calculated (kg/m²). The study protocol has been previously described (17), although, the following section will briefly review the methods.

Participants were instructed not to consume nicotine, caffeine, other stimulants or food for four hours prior to reporting to the lab, and to avoid any planned exercise for 24 hours prior to arrival. Each individual was randomized to perform one of two routines, which consisted of seven activities each. Activities in routine one included: filing papers, vacuuming, self-paced walking, treadmill walking at 6.4 km.hr⁻¹, cycling 49 watts, basketball, and treadmill running at 9.6 km.hr⁻¹. Activities in routine two included: computer work, treadmill walking at 4.8 km.hr⁻¹, cycling 98 watts, moving a box, treadmill walking at 4.8 and 6.4 km.hr⁻¹ with a 5% grade, and tennis. Each activity lasted seven minutes with at least a four-minute break in between activities.

Oxygen uptake (VO₂) was measured using the Oxycon Mobile portable metabolic unit (CareFusion; San Diego, CA). VO₂ data were collected breath-by-breath and averaged over minutes four through six of the collection period. Due to variations between the Oxycon systems at the two testing sites VO₂ values were increased by 7.8% at UTENN and decreased 7.8% at UMASS. Further information on this correction can be found elsewhere (17). The GENEA was worn on the dominant waist (right waist) of all participants and was initialized to collect data at 80 Hz. Following data collection, all data were downloaded onto a laboratory computer.

Measured VO₂ values were converted to METs (1 MET = 3.5 ml.kg^−1.min⁻¹)(1). These MET values were then classified into an intensity category using the same thresholds as Esliger et al. (8) (sedentary (<1.5 MET), light (1.5–3.99 METs), moderate (4.0–6.99 METs), or vigorous (7+ METs)).

The GENEA data were analyzed using the GENEA post-processing software (version 1.2.1) to convert the raw data into a gravity-subtracted Signal Vector Magnitude (SVM_gs) for each minute of activity. Three minutes (minutes 4–6) of each activity were averaged and classified into an intensity category using the hip cut-points of Esliger et al. (8)(sedentary (<77 counts/min), light (77–219 counts/min), moderate (220–2056 counts/min), or vigorous (>2056 counts/min).

Statistical analysis

Due to non-normal distribution of the data, Spearman’s rank correlation coefficients between the METs and GENEA SVMgs values were computed. However, Pearson’s correlation coefficients were also computed in order to allow comparisons with previously published studies. Accuracy of intensity classification for each activity when using the cut-points of Esliger et al. (8) was assessed by a one-way chi-square test. Cross-tabulation tables were constructed to provide GENEA under- and over-estimations and accuracy of cut-points by intensity category. Sensitivity and specificity of the waist cut-points were computed to determine accuracy rate by intensity classification.

RESULTS

Table 1 shows the accuracy of intensity category agreement within each activity (one-way chi-square analysis). The waist-worn GENEA accurately predicted intensity category classification 55.3% of the time. Two activities (walking at 4.8 km.hr⁻¹ at a 5% incline and running at 9.6 km.hr⁻¹) had perfect agreement between the GENEA estimated intensity categories and those based upon measured METs, and one activity (stationary cycling at 98 watts) had 0% agreement. Thus, a chi-square analysis could not be performed for these three activities.

Table 1.

Mean GENEA SVMgs, MET values, and intensity classification accuracy by activity

Variable	GENEA SVMgs (g.min)	METs	Intensity Classification Accuracy	p-value (80% accuracy)
Home/Office
Filing Papers (n=54)	44.97	1.46	53.7%	<0.001
Vacuuming (n=54)	244.25	3.2	46.3%	<0.001
Computer Work (n=49)	31.88	1.17	87.8%	0.175
Moving a Box (n=50)	694.45	4.52	62.0%	0.001
Self-Paced Walking (n−53)	1085.76	3.57	17.0%	<0.001
Walking/Running on TM
TM 4.8 km/hr 0% grade (n=49)	1130.91	3.68	20.4%	<0.001
TM 4.8 km/hr 5% grade (n=49)	1155.34	5.11	100.0%	*
TM 6.4 km/hr 0% grade (n=51)	2063.04	5.48	54.9%	<0.001
TM 6.4 km/hr 5% grade (n=41)	2007.24	7.08	65.9%	0.024
TM 9.6 km/hr 0% grade (n=33)	3687.5	9.67	100%	*
Sports
Cycle 49 watts (n=48)	126.66	3.8	52.1%	<0.001
Cycle 98 watts (n=49)	199.17	6.12	0.0%	*
Basketball (n=31)	2171.27	8.48	77.4%	0.719
Tennis (n=37)	1164.88	7.77	35.1%	<0.001
Average for Combined Activities:		55.3%

^*Unable to perform chi-square

TM = Treadmill

Spearman’s rank order correlation coefficient between the GENEA SVM_gs values and the measured MET values when all activities were combined was 0.730 (p<0.001). The Pearson’s correlation coefficient was 0.734 (p<0.001). Figure 1 shows the relationship between energy expenditure (METs) and GENEA SVM_gs values. Vertical lines indicate the proposed GENEA waist cut-points of Esliger et al. (8) and horizontal lines indicate the corresponding cut-points based on measured energy expenditure.

Figure 1.

Relationship between measured energy expenditure (METs) and GENEA SVM_gs (N=106).

Table 2 shows the cross-tabulation tables by actual intensity category as determined by the GENEA-estimated intensity category and the criterion intensity category. Intensity classification accuracy was calculated from the cross-tabulation tables. Shaded boxes represent accurate intensity classification. Overall intensity classification accuracy for all activities was 55.3% (Table 2a). When cycling was removed, intensity classification accuracy slightly increased to 58.3% (Table 2b). Light intensity activities were overestimated by the GENEA 60.6% of the time, and vigorous intensity activities were underestimated 43.2% of the time.

Table 2.

Cross tabulation tables showing the classification of physical activities into one of four categories, based upon GENEA waist-worn tri-axial accelerometer and measured energy expenditure

a. all activities
		GENEA
		Sedentary	Light	Moderate	Vigorous
Actual	Sedentary	69	3	0	0
	Light	32	52	129	0
	Moderate	4	53	158	30
	Vigorous	0	9	42	67

b. cycling removed
		GENEA
		Sedentary	Light	Moderate	Vigorous
Actual	Sedentary	69	3	0	0
	Light	29	27	126	0
	Moderate	0	3	158	30
	Vigorous	0	0	39	67

For all activities combined, the sensitivity of the cut-points for the four intensity categories ranged from 0.244 to 0.958 and specificity ranged from 0.576 to 0.943 (Table 3). Due to the frequent misclassification of cycling, when the two stationary cycling activities are removed from the analysis sensitivity for the four intensity categories ranged from 0.148 to 0.958 and specificity ranged from 0.542 to 0.984 (Table 3).

Table 3.

Sensitivity and specificity comparison of the GENEA tri-axial accelerometer (waist)

	Sensitivity		Specificity
	All Activities	Cycling removed	All Activities	Cycling removed
Sedentary	0.958	0.958	0.937	0.939
Light	0.244	0.148	0.850	0.984
Moderate	0.645	0.827	0.576	0.542
Vigorous	0.568	0.632	0.943	0.933
MVPA	0.818	0.990	0.547	0.504

DISCUSSION

Traditionally, accelerometer-based activity monitors have been worn on the waist to measure a person’s activity level (16). Results of the present study indicate that when the Esliger et al. (8) cut-points for the waist-worn GENEA activity monitor are used, it correctly classifies intensity category 55.3% of the time. In contrast, Esliger et al. (8) reported an intensity classification accuracy of 95% at the waist site. Both studies included a large range of MET values, ranging from 0.94 METs to 13.61 METs for Esliger et al. (8) and 1.46 METs to 9.67 METs in the current study. However, the Pearson’s correlation coefficient between activity counts and energy expenditure in the current study (r=0.73) was less than that seen in the study by Esliger et al. (r=0.87). One factor that could have contributed to these differences is that the study of Esliger et al. (8) was a calibration study with a subsequent validation, whereas ours was a cross-validation study in a different group of individuals. Another factor that could have contributed to the higher rate of misclassification in the current study was the addition of sporting activities, which were not included in the calibration study of Esliger et al. Although basketball showed an accuracy rate of 77.4% (p=0.719), the remaining sports activities (tennis and stationary cycling) were significantly below an 80% accuracy rate, with 0% agreement for cycling at 98 watts.

Waist vs. Wrist Site Comparison

The positioning of the activity monitor on the body is an important consideration for physical activity researchers. The waist and wrist positions both have associated strengths and limitations. While the waist-mounted accelerometers tend to underestimate upper body activities, the wrist-site can underestimate lower body activities that do not involve upper body movement (13). Therefore, it is of interest to compare the current waist site placement with the left wrist cut-points developed by Esliger et al. (8), which were previously cross-validated by our group in the same group of subjects, performing the same activities (17). Previous research comparing accelerometer outputs between tri-axial accelerometers at the wrist and waist have reported similar accuracy for these two wear locations (8). In addition, the use of tri-axial accelerometers vastly increases the relationship between energy expenditure and activity counts, whether accelerometers are worn at the wrist or waist sites (15). As previously discussed, Esliger et al. (8) reported high levels of explained variance at both the waist (r=0.87) and left wrist (r=0.86). While our data showed a weaker relationship between energy expenditure and activity counts than Esliger’s study, the relationship between energy expenditure and activity counts for these two sites was similar in the current study (waist r=0.73; left wrist r=0.74) (17).

Rosenberger et al. (14) placed triaxial accelerometers (Wockets) on the dominant waist and dominant wrist during 20 different activities, including sedentary activities, cycling, ambulation, and lifestyle activities. In contrast to the two studies described above, their results demonstrated large differences in the variance in energy expenditure explained by an accelerometer at the waist site (R²=0.52) and wrist site (R²=0.13). However, this discrepancy could be due to differences in accelerometers’ dynamic range (Wockets: ±2g; GENEA ±6g), or activities performed during validation. Chen et al. (5) combined the Tritrac R3D worn on the right waist and the ActiWatch worn on the dominant wrist during a 24-hour stay in a room calorimeter. The Tritrac R3D reported a 90% accuracy rate and the ActiWatch an 86% accuracy rate, when compared to the measured energy expenditure. However, because their study employed different monitor types at the wrist and the waist sites, it does not allow for a true comparison between these two different sites.

In addition, two studies have compared activity type (as opposed to intensity) classification accuracy at the waist and wrist site using machine-learning algorithms. These studies found that the waist and wrist sites showed close agreement, and both sites had high accuracy for classifying activity type (7, 18). It should also be noted that the wrist site offers some important advantages compared to the waist site, including the ability to estimate sleep variables and increased participant compliance in wearing the monitors (3, 9).

Recent research has highlighted the need for activity monitors that accurately classify sedentary behavior (3, 11, 12). The GENEA waist cut-points correctly classified sedentary activity 95.8% of the time, while the left wrist-worn GENEA only correctly classified sedentary activity 69.8% of the time (Table 4). However, at the opposite end of the intensity spectrum, the waist-worn GENEA activity monitor correctly classified vigorous intensity activity 56.8% of the time, but the wrist-worn activity monitor was able to correctly classify vigorous activity 70.7% of the time. Interestingly, basketball (mean METs: 8.5), was correctly classified 77.4% of the time with the waist-mounted GENEA, while the left-wrist worn GENEA correctly classified it 77.6% of the time (17). We had previously considered this to be a potential limitation to wrist-worn accelerometers due to the fact that most dribbling and shooting was performed with the dominant (right) arm, but the GENEA was worn on the non-dominant wrist (6). Overall, the wrist did a better job of classifying the intensity of sports activities (i.e. tennis and basketball), while the waist-worn GENEA cut-points were better able to detect activities such as inclined walking.

Table 4.

Accuracy of the Esliger et al. (8) cut-points (for waist and wrist-worn GENEA) by physical activity intensity level

	Waist	Wrist (18)
Sedentary	95.8%	69.8%
Light	24.4%	44.9%
Moderate	64.5%	46.2%
Vigorous	56.8%	70.7%

Sensitivity and specificity measure the ability of the cut-points to correctly identify a particular activity classification (i.e. sedentary/not sedentary) by calculating the amount of true positives and true negatives. Esliger et al. (8) reported a range of sensitivity and specificity for the waist site from 0.73–0.99 and 0.80–0.99, respectively. Our waist site analysis showed a much lower overall range for both sensitivity (0.24–0.95) and specificity (0.58–0.94). When stationary cycling was removed from the analysis, the sensitivity of the light intensity cut-point actually decreased from 0.244 to 0.148 (Table 3). The decrease in sensitivity can be explained by a decrease in true positives when stationary cycling data were removed. With regards to the left wrist site, Esliger et al. (8) reported a sensitivity range of 0.78–0.97 and specificity range of 0.72–0.98. Our previous study which cross-validated the Esliger et al. GENEA left wrist cut-points, resulted in a lower range of sensitivity (0.45–0.71) and specificity (0.74–0.91) when compared to the Esliger and colleagues reported values (17).

This study had several limitations. The participants’ age range in the current study was slightly younger (mean age: 39.1 ± 11.2 years) than Esliger and colleagues (mean age: 49.4 ± 6.5 years) (8). Also, we used the MET cut-points of 4 METs for moderate intensity activity and 7 METs for vigorous intensity activity. These cut-points were chosen in order to exactly replicate the method used by Esliger et al. (8). However, the use of these cut-points may limit comparability to other studies using other MET cut-points.

This study had several strengths worth noting. Participants with a wide range of ages (20–60 years) were recruited and we examined a wide array of activities, within the different domains of physical activity. We were also able to compare intensity category classification accuracy for both the non-dominant wrist-site using our previously reported data and the dominant waist-site, providing evidence relevant to positioning an activity monitor on the wrist versus the waist.

CONCLUSION

The intensity cut-points developed by Esliger et al. (8) for the waist-worn GENEA accelerometer have previously been shown to accurately estimate the classification of physical activity intensity. However, cross-validation of the waist cut-points in an independent sample, performing structured lifestyle activities, resulted in a 55.3% accuracy rate. In addition, the present study reported similar correlation coefficients between energy expenditure and activity counts for the waist-worn and wrist-worn devices (17), although these correlations were lower than those reported by Esliger et al. (8). This information is important to consider when deciding on which monitor to use and the placement of activity monitors in future studies.