Validity and Reliability of the Exercise Vital Sign Questionnaire in an Ethnically Diverse Group: A Pilot Study

Abstract

The purpose of this study was to determine the validity and reliability of the Exercise Vital Sign (EVS) questionnaire in an ethnically diverse sample. Participants (N = 39) were asked to wear an accelerometer at the hip for at least 7 days and to complete the EVS at the beginning (T1) and end (T2) of the wear period. The EVS questionnaire validity was determined against accelerometry, and bias was calculated as the mean difference between measures. The sensitivity and specificity of the EVS questionnaire were also evaluated. The reliability of the questionnaire was calculated using intraclass correlation coefficient (ICC) between EVS responses at T1 and T2. The mean difference in EVS- and accelerometer-determined time in MVPA was 24 min/wk. The reliability for the questionnaire was excellent (ICC = 0.98). The EVS specificity and sensitivity at T2 were 56% and 78%, respectively. The EVS questionnaire may be an acceptable measure of weekly MVPA time compared to accelerometry in an ethnically diverse sample; however, further research is needed to confirm these findings.

Introduction

Insufficient physical activity (PA) is one of the leading risk factors for all-cause mortality in the world and is a major risk factor for a number of non-communicable diseases.¹ Evidence suggests that regular participation in PA is effective at decreasing risk of cardiovascular disease, diabetes, certain cancers and premature mortality.² However, approximately 49.1% of Americans report insufficient PA and do not meet current government PA recommendations.³

The Exercise Vital Signs (EVS) questionnaire is a brief PA questionnaire that was developed to measure PA as a vital sign during a patient’s visit to a physician’s office and to further determine if a patient meets current PA recommendations.⁴ The initial EVS validation was conducted in 2012 and showed good face validity against PA data from large national surveys.⁴ However, few studies have attempted to determine the validity of the EVS against device-measured estimates of PA, such as with accelerometry.⁵

Research suggests that self-reported PA volumes may vary between ethnic groups.^6-8 For example, nationally representative data have shown no significant differences in accelerometer measured PA between adult non-Hispanic whites and non-Hispanic blacks.⁹ However, when looking at data on self-reported PA some studies have shown that non-Hispanic black adults report significantly lower levels of PA than non-Hispanic white adults.^6,7 Furthermore, a study comparing PA between Whites and South Asians showed that while objectively measured PA was similar between ethnicities, self-reported PA was 40% lower in South Asians when compared with whites.⁸ Thus, it is important that the measurement properties of PA questionnaires be evaluated in samples comprised of ethnically diverse individuals. With regard to the EVS, only one study has validated the questionnaire against accelerometry in a minority group of African American women,¹⁰ and another was conducted in a group of predominantly White adults.¹¹ However, to date, no studies have evaluated the validity and reliability of the EVS in an ethnically diverse group of adults against an objective measure of PA.⁵ The purpose of this pilot study, therefore, was to determine the validity and reliability of the EVS questionnaire in an ethnically diverse sample.

Methods

Participants

A convenience sample of 39 participants was recruited from the Queens College campus in Queens, New York and the community. The study took place in the Queens College Applied Physiology Lab in New York. All study procedures were verbally explained and written informed consent was obtained from all participants during the first visit prior to participation. The City University of New York Institutional Review Board approved the study (#2016-1378). Eligibility criteria included (1) being an adult between the ages of 18 to 65 years and (2) willingness to wear an accelerometer on the hip. Exclusion criteria included (1) any musculoskeletal disease that would limit PA participation and (2) any unmanaged chronic disease. Participants were asked to visit the lab on 2 occasions 9 days apart. On the first visit participants were administered a health history questionnaire and the EVS questionnaire. Height, weight, and body fat percentage (BF%) were measured, then participants received an accelerometer with instructions on how to properly wear the device. Participants were contacted every morning by either text or phone call to remind them to wear the accelerometer. When participants returned for the second visit, the accelerometers were collected and the EVS questionnaire was administered a second time by the same person.

Measures

Self-Reported Physical Activity

Self-reported PA was recorded using the EVS questionnaire. In short, the EVS is a brief PA questionnaire that requires less than 60 seconds to complete. The questionnaire asks 2 questions to assess habitual PA: (1) the average number of days/week the respondent engages in moderate to strenuous PA and (2) the minutes/day the respondent engages in such activities. The questionnaire score is determined by multiplying the number of days/week by the minutes/day of PA the respondent reports, resulting in an estimate of the total minutes of weekly PA. The initial validation of the EVS showed good face validity against PA data from the National Health and Nutrition Examination Survey (NHANES).⁴ The validity of the EVS has also been evaluated against accelerometry in African American women and Whites adults.^10,11 The reliability of the EVS has yet to be determined.

Body Composition

Body mass index (BMI) was assessed with weight and height measurements, respectively, taken with a calibrated scale and stadiometer (Detecto; Webb City, MO). Weight was measured to the nearest kilogram and height to the nearest centimeter. BF% was measured using the whole-body bioelectrical impedance analysis method (Body Stat 1500; BodyStat Ltd, Douglas, Isle of Man) following standardized procedures provided by the device manufacturer. Normal BMI is 18.5 to 24.9 kg/m², overweight is considered as 25.0 to 29.9 kg/m², and obesity is considered as 30.0 kg/m² or greater.¹² A BF% range considered satisfactory for health is 10% to 22% in males, and 20% to 32% in females.¹²

Accelerometry

PA was measured using an ActiGraph GT9X (Actigraph Corp, Pensacola, FL) over the course of nine days. The accelerometer was placed on the right hip in line with the anterior axillary line,¹³ and raw triaxial accelerometer data were recorded at 30 Hz. Participants were asked to wear the accelerometer during all waking hours except while engaging in water-based activities. Vertical axis activity counts per minutes (cpm) were evaluated over a 60-second epoch length. The last 7 consecutive monitored days before the day the accelerometers were returned were used for all PA analyses. Non-wear time was calculated following procedures by Choi et al.¹⁴ A valid day was considered as one with ≥10 hours of wear time, and participant data sets were only conserved if ≥4 valid wear days of accelerometer data were available—including one weekend day.^13,15 The Freedson et al¹⁶ thresholds were used to classify PA intensity using activity counts from the vertical axis. We use the term “MVPA bout minutes” to refer to time spent in moderate to vigorous PA (MVPA) episodes that lasted ≥10 minutes. The MVPA bout minutes were used to facilitate comparison with prior accelerometer-based studies on the EVS.¹¹

Physical Activity Recommendations

The US PA recommendation that adults engage in ≥150 minutes of aerobic MVPA/week was used to classify participants as sufficiently or insufficiently active.¹⁷ This criterion was applied to both accelerometer-derived MVPA bout minutes from the 7-day wear period and the EVS-reported MVPA minutes/week at T2.

Statistical Analyses

Data were analyzed using SPSS v24 (IBM Corp; Armonk, NY) and MATLAB R2017a (The Mathworks, Inc, Natick, MA). Descriptive statistics are presented as mean (standard deviation) or frequencies [% (n)]. A Bland-Altman plot with 95% limits of agreement (LOA) was used to determine agreement between the EVS questionnaire MVPA minutes/week and accelerometer-derived MVPA bout minutes/week.¹⁸ The 95% confidence interval for the mean difference (ie, bias), as well as upper and lower 95% LOA were, respectively, calculated following the methods of Bland and Altman.¹⁸ The relationship between EVS-determined MVPA minutes/week and accelerometer-determined MVPA bout minutes/week was also evaluated using Spearman’s rho (ρ). Using the dichotomous outcomes of meeting/not meeting PA recommendations as determined by both the EVS and accelerometer-derived MVPA minutes/week, the sensitivity and specificity of the EVS to identify PA recommendation compliance was evaluated against accelerometer data as the criterion. The EVS reliability was determined using absolute agreement intraclass correlation coefficients (ICCs) between questionnaire responses from T1 and T2. The significance level was established a priori as α = .05.

Results

The age range of our participants was 19 to 62 years. Approximately half of our participants were female (n = 20) and the rest were male (n = 19). Our participant pool included whites or Caucasians, Hispanic or Latinos, Black or African Americans, and Asians with at least some college education. One participant was excluded due to knee pain caused by arthritis. Participant characteristics are presented in Table 1. Participants wore the device for 6.9 (o.5) days and 13.8 (2.6) h/d, on average. EVS-determined mean MVPA minutes/week were 114 (127) minutes at T1 and 120 (130) minutes at T2. The accelerometer-determined mean MVPA minutes/week was 144 (108). A moderate, positive correlation was observed between the accelerometer-determined MVPA bout minutes/week and the EVS MVPA minutes/week at T2 (ρ = 0.60, P < .01). Using accelerometer-determined MVPA bout minutes/week, 41% (16) of the sample met PA recommendations. Using EVS-determined minutes/week at T2, 36% (14) of the sample met PA recommendations.

Table 1.

Participant Characteristics.

Variable	Males, n (%)	Females, n (%)	Total, n (%)
Age, years, mean ± SD	30.9 ± 9.6	31.0 ± 11.4	31.0 ±10.4
Ethnicity
White or Caucasian	7 (36.8)	6 (30.0)	13 (33.3)
Hispanic or Latino	5 (26.3)	7 (35.0)	12 (30.8)
Black or African American	3 (15.8)	3 (15.0)	6 (15.4)
Asian	4 (21.1)	4 (20.0)	8 (20.5)
Education
Some college	10 (52.6)	8 (40.0)	18 (46.2)
College graduate	9 (47.4)	12 (60.0)	21 (53.8)
Body mass index, kg/m2, mean ± SD	29.2 ± 4.0	24.3 ± 3.3	26.8 ± 4.4
Body fat percentage, mean ± SD	20.7 ± 5.5	30.1 ± 6.4	25.4 ± 7.6

The Bland-Altman plot in Figure 1 shows that the mean difference between the EVS- and accelerometer-determined MVPA time was 24 (95% CI: −13 to 60) min/wk. Figure 1 also shows that the upper and lower 95% LOA were 250 (95% CI: 186 to 313) and −202 (95% CI: −265 to −139) MVPA minutes/week, respectively. The sensitivity and specificity of the EVS at T2 were 78% and 56%, respectively. The EVS questionnaire showed excellent reliability (ICC = 0.98, P < .01) between T1 and T2.

Figure 1.

Bland-Altman plot for MVPA in minutes per week as determined by accelerometers versus the EVS questionnaire.

Abbreviations: EVS, Exercise Vital Sign questionnaire; MVPA, moderate to vigorous physical activity; GT9X_10min, accelerometer-derived MVPA in ≥10-minute bouts.

Discussion

Results from our pilot study indicate that the EVS may be a valid PA questionnaire within a diverse sample. Additionally, the EVS test-retest reliability was excellent when assessed at multiple time points. To our knowledge, this is the first study to evaluate the validity and reliability of the EVS questionnaire within a racially/ethnically diverse sample. As such, our study provides preliminary evidence that the EVS questionnaire can be used as a measure of habitual MVPA in racially/ethnically diverse populations.

Results from the Bland-Altman plot showed that while participants underestimated MVPA minutes/week by 24 minutes using the EVS, when averaged over the course of a week this means that EVS responses were biased by ~3.3 min/d when compared with accelerometer-determined MVPA minutes. Interestingly, we also found the 95% LOA in our study to be narrower than a prior report on agreement between the EVS questionnaire and accelerometry.¹¹ It is worth noting that previous research has shown differences in self-reported versus objectively measured MVPA volumes and walking activity between racial/ethnic groups.⁸ This may in part explain the differences between our results and those of previous studies on the EVS. In addition, it has also been reported that cultural differences may influence perceptions of exercise.^19,20 Taken together, our pilot results and prior studies suggest that additional research is needed to determine if there are any biases in the EVS questionnaire across diverse groups. Nevertheless, our promising finding showing a low bias in the EVS of 24 MVPA minutes/week preliminarily suggests that the EVS questionnaire is a useful proxy for accelerometer-determined levels of moderate to strenuous physical activity in free-living contexts.

The EVS sensitivity in our study was 78%, which suggests that the EVS is able to identify a fair proportion of respondents who were insufficiently active and are in need of interventions to increase daily PA participation. Our finding on the EVS sensitivity is similar to that reported by Joseph et al¹⁰ who found that the EVS was able to correctly identify 74% of participants who were insufficiently active in a group of African American women. Comparatively, Fitzgerald et al¹¹ reported an EVS sensitivity of 59% in a group of White adults. It is worth noting, however, that Fitzgerald et al¹¹ used different accelerometer data processing and classification methods, which may have contributed to the differences in results. The specificity of the EVS in our pilot study was 56%, and fell within the range of values reported by Fitzgerald et al¹¹ (77%) and Joseph et al (33%).¹⁰ These divergent results on the EVS specificity appear to suggest that the questionnaire may identify those meeting PA recommendations with modest accuracy at best. Thus, we found that while the sensitivity of the EVS in a diverse sample was greater than those reported in prior studies that have compared self-reported and accelerometer-determined MVPA volumes, the EVS specificity is relatively low and inconsistent among the available studies. Further work is required to confirm the sensitivity and specificity of the EVS in a larger cohort of racially/ethnically diverse adults.

This is the first study to formally report on the test-retest reliability of the EVS questionnaire. The intraclass correlation coefficient between the EVS at T1 and T2 was excellent (0.98), which is not uncommon among PA questionnaires.^21,22 Additionally, all participants had some college education or were college graduates, which may have played a role in these results. Previous research suggests that higher education levels are associated with better ability to recall PA participation.²³ Nevertheless, results from our study suggest that the EVS is a reliable PA questionnaire for estimating weekly time spent in MVPA.

It is worth noting that the EVS does not add any specific weights for vigorous-intensity PA or moderate-intensity PA. Therefore, it is possible that if someone does only 75 minutes of vigorous PA they can be erroneously misclassified as insufficiently active because there is no way to distinguish between moderate and vigorous PA from the overall EVS score. However, vigorous intensity physical activity is rare in the population, as studies using samples from the NHANES have shown that adults typically perform between 0.4 and 18.6 minutes of vigorous-intensity PA per week.^9,24 Another potential limitation of our study is that participants in our sample were well-educated which, as noted previously, may have improved the ability to respond to the questionnaire with accuracy.²³ Finally, in closely following the methods of prior PA questionnaire validation studies,^10,11 we delimited our study to compare the EVS questionnaire and another proxy of free-living MVPA (ie, accelerometry).

In summary, in a racially/ethnically diverse sample the EVS questionnaire appears to have acceptable validity and high test-retest reliability. Given its short length and acceptable measurement properties the EVS appears to be a useful tool for physicians and other clinicians to conveniently assess MVPA participation, given the health-related benefits of MVPA. However, further research is needed in a larger cohort of participants that reflect greater racial/ethnic, cultural, socioeconomic diversity.

Author Biographies

Norberto N. Quiles is an assistant professor in the department of Family, Nutrition and Exercise Sciences, Queens College of the City University of New York. His research focuses on physical activity and cardiovascular disease risk in healthy individuals and people living with HIV.

Aston K. McCullough received his PhD and MPhil degrees in Kinesiology from Columbia University in the City of New York; MS, Applied Statistics, Columbia University Teachers College; MA, Dance Education, New York University; and BA, Liberal Arts, Sarah Lawrence College. He is currently a Postdoctoral Research Fellow in the School of Public Health & Health Sciences at the University of Massachusetts Amherst, as well as an Adjunct Assistant Professor of Applied Physiology at Columbia University Teachers College.

Lin Piao received her MS degree in Exercise Sciences from Queens College of the City University of New York.