Development and Reliability Testing of the Worksite and Energy Balance Survey (WEBS)

Abstract

Context

Worksites represent important venues for health promotion. Development of psychometrically-sound measures of worksite environments and policy supports for physical activity and healthy eating are needed for use in public health research and practice.

Objective

Assess the test-retest reliability of the Worksite and Energy Balance Survey (WEBS), a self-report instrument for assessing perceptions of worksite supports for physical activity and healthy eating.

Design

The WEBS included items adapted from existing surveys or new items based on a review of the literature and expert review. Cognitive interviews among 12 individuals were used to test the clarity of items and further refine the instrument. A targeted random-digit-dial telephone survey was administered on two occasions to assess test-retest reliability (mean days between time periods=8; min=5; max=14).

Setting

Five Missouri census tracts that varied by racial-ethnic composition and walkability

Participants

Respondents included 104 employed adults (67% White, 64% female, mean age =48.6 years). Sixty-three percent were employed at worksites with <100 employees, approximately one-third supervised other people, and the majority worked a regular daytime shift (75%).

Main Outcome Measures

Test-retest reliability was assessed using Spearman correlations for continuous variables, Cohen's kappa statistics for non-ordinal categorical variables, and one-way random intra-class correlation coefficients for ordinal categorical variables.

Results

Test-retest coefficients ranged from 0.41 to 0.97, with 80% of items having reliability coefficients >0.6. Items that assessed participation in or use of worksite programs/facilities tended to have lower reliability. Reliability of some items varied by gender, obesity status, and worksite size. Test-retest reliability and internal consistency for the five scales ranged from 0.84 to 0.94 and 0.63 to 0.84, respectively.

Conclusions

The WEBS items and scales exhibited sound test-retest reliability and may be useful for research and surveillance. Further evaluation is needed to document the validity of the WEBS and associations with energy balance outcomes.

Introduction

Worksite wellness programs and policies represent promising strategies for making a positive impact on the physical and psychological health of working adults, with the added benefit of reducing absenteeism, injuries, and healthcare costs.^1-3 The worksite represents a particularly important setting for promoting physical activity, healthy eating, and healthy weight. For example, Healthy People 2020 includes an objective to increase the number of worksites that provide classes or counseling on nutrition and weight management.⁴ Moreover, in the Guide for Community Preventive Services, the Centers for Disease Control and Prevention highlights worksite wellness programs as evidence-based community interventions for controlling employee weight.⁵

Supportive worksite environments and policies assist employees in making healthful choices during and after work.^5-8 A recent meta-analysis found that worksite physical activity and dietary behavior interventions contributed to a greater reduction in body weight among employees if the interventions contained an environmental component.⁹ Another review observed that multi-component strategies including education, employee and peer support for physical activity, incentives, and access to places to exercise have strong evidence for influencing physical activity and good nutrition.¹⁰ Other reviews have been less conclusive about the unique contribution of environmental and policy approaches to influence healthy eating, physical activity and obesity, noting the limited number of studies that modified the work environment.^{5, 11} Several cross-sectional studies have observed associations between self-reported access to specific environments and policies and employee physical activity.^{7-8, 12-14} Moreover, having multiple worksite policies has been associated with recreational physical activity^7-8 and active commuting.¹³

Worksite environments and policies can be assessed by direct observation, manager interviews and employee surveys.¹⁵ Although several tools have been tested for assessing worksite-level environmental supports for physical activity and healthy eating,^16-20 the majority of surveys assessing employee perceptions in the general population have not undergone psychometric testing,^{7-8, 12, 14, 21} or are limited to assessing the worksite environment as it relates to physical activity only²² or nutrition offered through vending machines at the workplace.²³ While direct observations of worksite environments and policies are useful for intervention planning, they are expensive for assessing population-level exposures. A survey instrument that reliably and comprehensively assesses employees’ perceptions of worksite environments and policies would provide a valuable research tool for examining individual-level relationships with health outcomes and could be incorporated into surveillance systems. Altogether, improving the metrics for assessing worksite health promotion strategies will contribute to a better understanding about what strategies are most effective in promoting energy balance (caloric intake and energy expenditure via physical activity) and, thus, will inform best practices. The purpose of the current study was to develop and test a survey instrument for assessing perceived worksite environmental and policy supports for healthy eating and physical activity.

Methods

Tool Contents

The Worksite and Environment Balance Survey (WEBS) was developed following a literature review of existing instruments in which items were classified by the following worksite domains relevant for physical activity and healthy eating: promotions and programs (e.g., exercise classes, diet counseling), organizational policies (e.g., paid time for physical activity), social environment (e.g., role models for making healthy food choices), internal physical environment (e.g., exercise equipment, healthy vending options) and external environment (e.g., walkable streets, fast food restaurants). In addition, items were identified that captured important contextual information about worksite and occupational characteristics, as well as behaviors occurring during work hours (e.g., where meals eaten at work). The final WEBS included seven sections (Table 1) with 84 items that were new, or derived or adapted from existing instruments. The sections for worksite environmental and policy supports, and associated scales described later, were organized by specific worksite settings (i.e., neighborhood outside the worksite, within the worksite as a whole, in the cafeteria, and at vending machines) or as part of the social environment/culture of the workplace. The sections are ordered by their appearance in the survey. Section 1 consisted of 12 descriptive open-ended and categorical items about the respondent's primary job and worksite characteristics. Items in this section were new (e.g., supervise others, day/hours per week work) or derived from existing surveys.^24-27 Section 2 included 12 four-point Likert scale (strongly disagree to strongly agree) items about the neighborhood around the respondent's primary workplace, adapted from the Physical Activity Neighborhood Environment Scale,²⁸ Moore et al. (2009)²⁹ or new items. Section 3 included 32 yes/no/don't know response items about programs, facilities and policies the respondent's primary employer offers. Most were adapted from previous surveys, for example from the Neighborhood Quality of Life Study^{7, 14} or others.^{8, 16-17, 30} Fourteen of these items assessed participation or use of worksite supports among participants who reported having the specified program/facility/policy. Section 4 was made up of 12 yes/no/don't know response items about the cafeteria, snack bar, or food service, many of which were new or adapted from existing instruments.^{16, 20, 31} These items were only asked among employees who reported having a cafeteria, snack bar, or employee food service. Section 5 included five questions pertaining to vending machines asked among employees who reported having vending machines at their workplace. These items were either new or adapted.³¹ Section 6 consisted of five open-ended questions about where the respondent obtained his/her meals during lunch breaks or other breaks while at work during the past week, and were either new or adapted from existing instruments.³² Section 7 included six four-point Likert scale (n=3 with strongly disagree to strongly agree and n=3 with never to often response choices) questions about the organizational and social environment. Most were new items with one item from an existing instrument.³¹

Table 1.

Worksite and Energy Balance Survey Items by Section

Section 1. Job and Worksite Characteristics (n=12 items)	Section 4. Cafeteria (n=12 items; yes/no)
Number of employees	Cafeteria, snack bar, or food service employees
Supervise others	Purchased food or beverage
Number of years at main job	Fresh fruits and vegetables
Number of days per week typically work	Non-fried meat entrees
Number of hours per week typically work	Low fat snack items
Type of shift typically work	Low fat dairy products
Flexibility	Sandwiches made with whole grain bread
Number of work days missed due to health	Food in smaller portions
Mode of transportation to work	Symbols/signs to identify healthy food alternatives
Number of minutes to get to work	Calorie information
Another job	Posters or signs encouraging healthy eating
Number of hours per week typically at other job	Affordable prices for fresh fruits and vegetables
Section 2. Neighborhood Around Workplace (n=12 items; strongly disagree to strongly agree)	Section 5. Vending (n=5; yes/no)
Fast food	Vending machines for employees
Healthy restaurants	Purchased food or beverage ≥ once/week
Stores within walking distance	Low-fat snacks
Transit stop within 10-15 min. walk	Low-fat or fat free milk, or water
Sidewalks	Symbols/signs to identify healthy food alternatives
Bicycle facilities
Free/low cost recreation facilities	Section 6. Meals (n=5; open-ended number)
Crime rate makes it unsafe to walk
High traffic	Number of times brought lunch last week
Others are physically active	Number of times purchased cafeteria meal last week
Interesting things to look at	Number of times purchase fast food meal last week
Walk in workplace neighborhood ≥ once/week	Number of times purchased meal from sit-down restaurant last week
Section 3. Programs, Facilities, and Policies (n=32; yes/no)*	Number of times purchased meal from another type of restaurant last week
Personal services
Health fairs	Section 7. Organizational and Social Environment (n=6 items; strongly disagree to strongly agree)
Worksite challenge events
Regular exercise programs	Company values healthy lifestyle
Indoor exercise facilities	Co-workers are good role models for healthy food choices
Outdoor exercise facilities
Shower facilities	Co-workers are good role models for being physically active
Place to lock bike
Flexible time for physical activity during work	Co-workers eat fruits and vegetables
Physical activity breaks	Co-workers use alternative transportation to get to work
Free/reduced membership to offsite exercise facility	Co-workers are physically active during work breaks
Incentives to walk or bike to work
Incentives to use public transit
Maps or signs of walking routes
Signs encouraging stair use
Information encouraging physical activity
Clean water fountain
Free water bottles or water cooler

^*Missing items from the list were whether the respondent used the facility or participated in the program.

Survey Development and Cognitive Testing

The WEBS was part of a larger questionnaire being used for the Supports at Home and Work for Maintaining Energy balance (SHOW-ME) study. The entire survey was reviewed by six external researchers and two practitioners with expertise in physical activity, diet, worksite wellness and health promotion. Edits were made following their review. Next, the survey underwent cognitive response testing (CRT). Volunteers from a research registry who were representative of the SHOW-ME participant population were recruited by telephone to participate in the CRT session. Additional volunteers (n=4) were sought to add diversity of occupations. Twelve participants consented and participated in telephone interviews with two trained project staff (mean administration time, 53 min; range, 37-66 min). Interviews consisted of asking participants each question from the WEBS, followed by specific questions to determine information retrieval, question comprehension, decision processing, and confidence in the answer selected.^33-34 Interviews were tape-recorded to facilitate note-taking. Participants were sent a $40 gift card after interview completion. Findings from the interviews were summarized and reviewed by the project team, resulting in the rewording of several questions to improve clarity. Additional CRT findings led to changes in the interviewer script and instructions to enhance participants’ understanding of question intent.

Study Population

Participants for the reliability study were sampled from Missouri regions within race-ethnicity and walkability strata being used for the larger SHOW-ME study. For the current study, five Missouri census tracts were randomly sampled from the St. Louis and Kansas City metropolitan areas. The tracts varied with respect to race-ethnicity (n=2 tracts in high racial-ethnic minority stratum; n=3 tracts in low racial-ethnic minority stratum) and walkability (n=2 tracts in low walkability stratum, n=2 tracts in moderate walkability stratum, n=1 tract in high walkability stratum). High racial-ethnic minority was defined as ≤50% White or ≥50% Hispanic; and low as >50% White and <50% Hispanic. Walkability was based on an index created from the sum of the z-scores for intersection density, retail density, and household density, modified from a similar index.³⁵ Adults within each of the five selected census tracts were recruited using list-assisted telephone random-digit-dialing methods with a target sample size of 20 respondents per tract. Inclusion criteria for participation included: (1) 21-65 years old; (2) employed at least 20 hours/week; (3) works at one primary location; (4) primary workplace has ≥5 employees; (5) not pregnant; and (6) no physical limitation to prevent walking or bicycling in the past week. Participants completed the survey at two time points. Response rate for the baseline interview was 49%; 82% of participants at baseline participated in the re-test survey. Mean time between test and re-test surveys was 8 days (range 5-14). Participants received a $20 incentive for participating. This study was approved by the Institutional Review Boards of Washington University in St. Louis and University of Missouri-Columbia.

Data Analysis

Data were analyzed using Statistical Package for the Social Sciences (SPSS) Version 20. To examine test-retest reliability, Spearman correlations were calculated for all continuous variables due to non-normality of their distributions, Cohen's kappa statistics for non-ordinal, categorical variables, and one-way model intra-class correlation coefficients (ICC) for the ordinal, categorical variables derived from one-way analysis of variance. ‘Refused’ and ‘don't know’ responses were included when calculating kappa statistics because ‘don't know’ was considered an important response to understand for questions about worksite supports; however, participants who answered ‘don't know’ or ‘refused’ were set to missing when calculating ICC and correlation coefficients for ordinal or continuous variables, respectively. Notably, the maximum number of participants with missing data for any of these ordinal or continuous variables was six.

Scales were developed for all sections except the job and worksite characteristics (section 1) and meals (section 6) items. The items about the neighborhood around the workplace and about the organizational and social environment were dichotomized and coded so that higher values represented positive supports for physical activity and healthy eating, similar to what has been done previously for items pertaining to the neighborhood around the home.^{28, 36} Next, items within each section of the WEBS were summed to yield five scales. The scales can be interpreted as the number of perceived healthy eating and physical activity supports within specific worksite settings, or as part of the social environment/culture of the workplace. Items related to use of programs, facilities and policies were excluded from the scales because they measured behaviors, not a worksite environmental or policy support. Test-retest was calculated with the ICC and internal consistency with Cronbach alpha.³⁷

Differences in test-retest reliability of the items were examined by obesity status (<30 and ≥30 kg/m²) and worksite size (<100 employees and ≥100 employees) to test whether consistency in reporting of worksite supports varied by these factors. For example, employees in smaller worksites may more reliably report certain facilities and policies compared with employees in larger, and potentially more dispersed, worksites. As a rough guide in interpreting results, we followed the adjectival ratings suggested by Landis and Koch³⁸ in the following categories: 1.0–0.8 (almost perfect agreement), 0.8–0.6 (substantial agreement), 0.6–0.4 (moderate agreement), 0.4–0.2 (fair agreement), 0.2–0.0 (poor agreement).

Results

Sample Descriptives

Table 2 shows the demographic characteristics of the sample of 104 respondents at first completion (test portion) of the survey. Approximately two-thirds of participants were between 45 and 65 years old, female, non-Hispanic white, or married. The majority lived with one or more other people. About half of the respondents had a household income of ≥$70,000, and most (62.5%) had at least a college degree. Years employed at primary workplace varied. Approximately half of participants were employed at worksites with <100 employees, approximately one-third supervised other people, and 75% worked a regular daytime shift. Overall, more than half reported very good to excellent health, most did not smoke at all, and over one-fourth were obese.

Table 2.

Descriptive characteristics of the study sample at baseline (“test”) data collection, n=104

Variable	% (n)
Age Group (years)
≤ 29	5.9 (6)
30-44	26.7 (27)
45-65	67.3 (68)
Missing (n=3)
Gender
Male	35.6 (37)
Female	64.4 (67)
Race
White	67.3 (70)
Black	26.0 (27)
Other	6.7 (7)
Marital Status
Married or living with partner	71.2 (74)
Widowed	1.9 (2)
Divorced or separated	14.4 (15)
Never married	12.5 (13)
Number people in household
1	15.4 (16)
2	26.9 (28)
3	24.0 (25)
≥4	33.7 (35)
Household Income ($)
<30,000	8.2 (8)
30,000-49,000	20.4 (20)
50,000-69,000	19.4 (19)
≥70,000	52.0 (51)
Missing (n=6)
Education
12th grade or completed high school	11.5 (12)
Post high school training other than college	1.9 (2)
Some college	24.0 (25)
College graduate	37.5 (39)
Postgraduate	25.0 (26)
Years employed at workplace
≤1	15.4 (16)
2-5	22.1 (23)
6-10	22.1 (23)
11-20	24.0 (25)
>20	16.3 (17)
Number of employees
<25	18.6 (19)
25-49	6.9 (7)
50-99	20.6 (21)
100-199	16.7 (17)
200-499	9.8 (10)
500-999	4.9 (5)
≥1000	22.5 (23)
Missing (n=2)
Supervise other people	35.6 (37)
Type of shift
Regular daytime schedule	74.8 (77)
Regular evening shift	3.9 (4)
Regular night shift	5.8 (6)
Rotating shift	6.8 (7)
Another schedule	8.7 (9)
Missing (n=1)
Smoking Frequency
Every day	5.8 (6)
Some days	1.9 (2)
Not at all	92.3 (96)
Body mass index
Underweight (<18.5 kg/m2)	0.0 (0)
Normal weight (18.5-24.9 kg/m2)	35.6 (37)
Overweight (25.0-29.9 kg/m2)	33.7 (35)
Obese (≥30.0 kg/m2)	27.9 (29)

Overall Reliability

Table 3 summarizes the mean and distribution of test-retest reliability for the WEBS items by each section of the tool. Most items (80.5%) demonstrated substantial to nearly perfect reliability, and no items had less than moderate reliability. Mean test-retest reliability was 0.70 (SD, 0.13) among all items and ranged from 0.41 to 1.0. In addition, there were 34 more ‘don't know’ responses at baseline than at retest (data not shown).

Table 3.

Test-retest reliability range for items by section within the Worksite and Environment Balance Survey (n=104)

	WEBS items								WEBS scales
	Test-retest reliability coefficients			Number of items in test-retest reliability range*						Test- retest	Internal consistency
	N Items	Mean (SD)	Min, Max	0.00- 0.19 (Poor)	0.20- 0.39 (Fair)	0.40-0.59 (Moderate)	0.60-0.79 (Substantial)	0.80-1.00 (Nearly Perfect)	N Items	ICC	Cronbach α
1. Job and worksite characteristics	12	.81 (.13)‡§II	.55,.97	0	0	1	4	7	--	--	--
2. Neighborhood around workplace	12	.71 (.04)‡§	.65, .79	0	0	0	12	0	11	.90	.72
3. Programs, facilities, and policies	30†	.69 (.13)§	.41, 1.0	0	0	6	18	6	18	.93	.84
4. Cafeteria	12	.61 (.11)§	.47, .83	0	0	5	6	1	10	.87	.79
5. Vending	5	.74 (.10)§	.63, .84	0	0	0	3	2	3	.84	.63
6. Meals	5	.59 (.11)II	.46, .78	0	0	4	1	0	--	--	--
7. Organizational and social environment	6	.71 (.03)‡	.68, .76	0	0	0	6	0	6	.88	.77
Total	82	.70 (.13)	.41, 1.0	0	0	16	50	16

^*Spearman Correlation, Kappa, and ICC were used to calculate reliability coefficients

^†Two items related to use of worksite support could not be calculated due to an insufficient number of participants reporting presence of worksite support

^‡ICCs were used in this calculation

^§Kappas were used in this calculation

^IISpearman correlations were used in this calculation

Of the 12 items within the job and worksite characteristics section (section 1), seven had almost perfect reliability, four had substantial reliability, and one had moderate reliability (namely, item about the number of days missed because of problems with physical or mental health). All of the neighborhood around workplace items (section 2) and organizational and social environment supports items (section 7) had substantial reliability. Of the 30 items related to programs, facilities, and policies (section 3), six items had almost perfect reliability, 18 had substantial reliability, and six had moderate reliability. The reliability of two items which assessed use/participation in physical activity breaks during meetings and incentives to walk/bike to work could not be calculated due to an insufficient number of respondents reporting these supports. Both of the items that assessed the presence of these supports had moderate reliability. The four other items with moderate reliability were those that assessed use of or participation in other worksite supports (e.g., use of outdoor exercise facilities or flexible time for physical activity during the work day). The cafeteria and vending items were asked among the respondents who reported having a cafeteria, snack bar, or food service (n=47) or vending machine (n=84). Of the cafeteria items (section 4), one had almost perfect, six had substantial and five had moderate reliability. Items with moderate reliability included ones assessing availability of the following in the cafeteria: non-fried meat entrees, low fat snack items, low fat dairy products, food in smaller or half-sized portions, and posters/signs that encourage healthy eating. Section 5 had two items with almost perfect reliability and three with substantial reliability. The meals section (section 6) had one item with substantial reliability (frequency of bringing lunch to work) and four with moderate reliability (frequency of purchasing meals at a cafeteria/food service, sit-down restaurant, fast food restaurant, or other type of restaurant). Test-retest reliability and internal consistency of the WEBS scales were strong. Deletion of the seven items with moderate reliability did not alter the internal consistency of their respective scales (i.e., cafeteria scale and program, facilities, and policies scale).

Reliability by Obesity Status

When stratified by obesity status (obese vs. non-obese), three items had an insufficient number of respondents to calculate stratum-specific reliability coefficients. Thirty items had reliability coefficients that differed by >|0.20| between obese and non-obese participants, and half of the differences were higher among obese participants with the other half higher among non-obese participants. Fourteen of the notable differences in reliability coefficients were in the workplace programs, facilities, and policies section. In general, most of the differences were among items that were asked among respondents who had reported a specified worksite support (i.e., with small n's) and/or items that asked about a behavior that may have changed between assessment periods (e.g., frequency of fast-food consumption).

Reliability by Size of Worksite

Three items had an insufficient number of respondents to calculate reliability coefficients by size of worksite (<100 employees or ≥100 employees). There were 26 instances where the coefficients for an item were > 0.2 different by size of worksite (data available upon request). Nine coefficients were higher among respondents from larger worksites, and 17 coefficients were higher among participants from smaller worksites. The program, facilities, and policies section had 11 of the 26 reliability differences, and nine of those showed a higher reliability among the smaller worksites compared to the larger worksites. Eight of the differences in reliability coefficients in this section were for questions that asked about participation/use of a program or facility. Nine additional differences were found for items about food and beverage offerings among respondents with a cafeteria or food service. All but two of these had reliability coefficients that were higher among respondents employed in smaller worksites.

Discussion

This study demonstrated high reliability of the WEBS among a diverse sample of working adults. Relationships between some of these worksite support measures and physical activity behavior have been observed in previous cross-sectional studies,^{7-8, 12-14} but the psychometric properties of the measures were unknown. To our knowledge, no other self-report instruments for assessing worksite supports for healthy eating have been tested; thus, this study contributes to a growing literature on measurement of the food environment in different settings.¹⁵

All WEBS items had reliability coefficients that exceeded 0.40 with 80% of items exhibiting test-retest reliability coefficients above 0.60. The test-retest reliability coefficients were similar to findings from other studies of surveys used to assess worksite environments for physical activity^{22, 39} as well as neighborhood environments. For example, the worksite neighborhood items and scale in the WEBS showed similar test-retest reliability as the home neighborhood environment items from which they were adapted.²⁸ Altogether, of the 16 items with coefficients in the moderate range (0.40-0.59) across all sections, nine items assessed behaviors or conditions, which had the potential to vary between assessment periods; thus the lower reliability may reflect true changes in behavior or health condition rather than measurement error. Moreover, we observed a reduction in responses of ‘don't know’ at re-test. Taking the survey could have been enough to raise the awareness of some respondents about their worksite environment and policies, and in the time between the test and retest, respondents may have sought an answer to a survey question. This testing effect is only a concern if the tool were used as part of a pre- and post-test assessment for a worksite intervention.

Reliability differed for some items by obesity status and size of worksite. There were more items for which reliability was higher for respondents at smaller worksites (17 of 26 items) than for respondents at larger worksites. No clear patterns in reliability emerged by obesity status. Overall, the strength of coefficients within the various strata supports the reliability of the WEBS for diverse populations in worksites of various sizes.

A few limitations deserve mention. First, further evaluation is needed to document the validity of the WEBS and associations with physical activity and diet outcomes. Second, the relatively long length of the WEBS may preclude its adoption into surveillance systems. Yet, the WEBS can be shortened in the future when more evidence is available about associations with physical activity, diet and obesity. Third, use or adaptation of existing items was a priority, resulting in some inconsistencies in response options. Specifically, factual items in the WEBS typically used a yes/no response format (e.g., programs, facilities, and policies) with the exception of items pertaining to the neighborhood around the workplace, which used a Likert scale response because they were derived from an existing survey.^{7, 14} Fourth, the sampling methods may limit generalizability in that participants were required to work at one primary location and possess a landline telephone; thus, wireless-only households were not represented.⁴⁰

In summary, worksites are excellent venues for health promotion.⁸ Due to the rising costs of healthcare associated with obesity-related illness and disability, employers are increasingly interested in offering programs or benefits to assist employees in making healthful decisions.⁴¹ The WEBS showed promise as a useful tool for assessing workers’ perceptions and knowledge about a comprehensive set of worksite facilities, programs, information, and policies for encouraging physical activity and healthy food options, both inside and outside the worksite. Moreover, it assessed coworker supports and norms concerning healthy lifestyle. WEBS items that are later shown to be associated with energy balance outcomes could be incorporated into ongoing surveys and surveillance systems to assist the public health and business community in understanding priorities for worksites, as well as uptake of available worksite programs and policies for activity and healthy eating.