Test-retest reliability of a questionnaire measuring perceptions of neighborhood food environment

Abstract

There is a lack of validated and reliable instruments on perception of the food environment, in particular for rural environments. We estimated the test-retest reliability of a questionnaire assessing perceptions of the food environment. A total of 101 primary food shoppers in South Carolina were interviewed by phone to assess their perceptions of the food environment and presence of different food outlet types in their neighborhood. The survey was repeated approximately one month after the initial administration. The intra-class correlation (ICC) and Phi coefficient are reported as measures of reliability. The majority of questions on perceptions of the neighborhood food environment appear highly reliable (ICCs range from 0.55 to 0.71), including the 3-item scale on healthy food availability (ICC 0.71). Compared to participants in rural areas, those in urban areas demonstrated better reliability for questions on opportunities to purchase fast food and perceived presence of a supercenter. More research is needed to evaluate potential rural-urban differences in reliability.

INTRODUCTION

Recent research suggests that an individual’s perception of the food environment is associated with dietary intake (Hearst et al., 2012; Keita et al., 2011; Sharkey et al., 2010; Zenk et al., 2005). Several measures have been developed to characterize the food environment including observations of local neighborhoods, geographic information system (GIS) - based measurements, and self-reported perceptions of the food environment. Studies have shown that perceptions of food environment are reliable but not identical compared to GIS-based measurements (Echeverria et al. 2004; Moore et al. 2008a; Freedman et al. 2009). Subjective reports may provide information on the foods actually available and of interest to residents which are not captured by data on the locations of facilities.

The most well-known self-report instrument on the food environment was developed for the Multi-Ethnic Study of Atherosclerosis (MESA) to measure the perceived availability of healthy food options and lack of access to adequate food shopping within a person’s neighborhood (Echeverria et al., 2004; Mujahid et al., 2007). Test-retest reliability of a 6-item healthy food access scale was assessed by Echeverria et al. (2004) in a pilot study of 48 volunteers living in New York City. The scale was subsequently refined to include only three items by Mujahid et al. (2007) in a subsample of the MESA study, which included 120 individuals in three study sites (Maryland, North Carolina, and New York).

Both aforementioned studies (Echeverria et al., 2004; Mujahid et al., 2007), as well as several others (Moore et al., 2008a; Moore et al., 2008b; Freedman et al., 2009; Keita et al., 2011) have been conducted in urban environments. To the best of our knowledge, only one has compared the perceptions of food environments of rural, suburban, and urban food pantry clients in Iowa and found that rural clients were significantly more likely to perceive their community as having an inadequate number of grocery stores or supermarkets (Garasky et al., 2004).

Several studies have collected data on perceived presence of specific food store outlet types in the neighborhood to evaluate the food environment (Gustafson et al., 2011; Zenk et al., 2009). However, no studies have been performed to assess the reliability of questions on perceived presence of specific food store outlet types.

This study sought to estimate the test-retest reliability of a questionnaire assessing self-reported perceptions of the food environment including the access, availability, and quality of healthy food options, and the perceived presence of specific food outlets in the neighborhood. We examined the test-retest reliability overall and examined whether there were any differences by urban-rural classification.

METHODS

Study Population and Sub-study Sample

In 2010, we conducted a cross-sectional study of residents of an eight-county region in South Carolina. Using a geographically-based sampling scheme, we randomly sampled 2,477 residential listed landline phone numbers and sent out introductory letters. Recruitment calls were made between April and July 2010 and a total of 968 adults participated in the telephone survey. For the reliability sub-study, we randomly selected 155 respondents from the main sample and repeated the phone interview about one month (Mean=35.4 days, SD=8.2 days) after the initial survey. In the end, 101 persons completed the reliability study, 7 refused, 42 could not be reached, and 5 were ineligible (not in service or no longer living at the number), yielding a response rate of 67.3%. All the interviewers were from the Survey Research Laboratory (SRL) at University of South Carolina and all were highly trained. The interviewer conducting the second interview may have differed from the interviewer conducting the initial interview. In present study, we define urban and rural residents using the 2010 Census-based designation of urban and rural area (2010 Census urban and rural classification and urban area criteria, 2010). The urbanized areas (of 50,000 or more people) were considered as urban areas. Urban clusters (of at least 2,500 and less than 50,000 people) and rural areas were considered as rural areas in this study. The protocol was reviewed and approved by the Institutional Review Board (IRB) at University of South Carolina, and respondents gave verbal consent.

Questionnaire Administration

Five questions on perceptions of the food environment previously developed for the MESA Neighborhood Study (Mujahid et al., 2007) were used and included (a) availability of fresh fruit and vegetables, (b) quality of fresh fruit and vegetables, (c) availability of low fat products, (d) opportunities to purchase fast food in the neighborhood, and (e) access problems for food shopping (Table 1). As previously used in other food environment research, the neighborhood was defined as within a 20-minute walk or one mile (1.6km) from home (Echeverria et al., 2004; Mujahid et al., 2007). The responses to above questions were coded on a Likert scale ranging from 1 to 5 (see Table 1). To create a composite healthy food availability scale in accordance with Mujahid et al. (2007), we computed the average of three of the five items (i.e. availability of fresh fruit and vegetables, quality of fresh fruit and vegetables, and availability of low fat products).

Table 1.

Questions on perceptions of food environment in the telephone survey questionnaire^*

Perceptions of the food environment†

1. A large selection of fresh fruits and vegetables is available in my neighborhood

2. The fresh fruits and vegetables in my neighborhood are of high quality

3. A large selection of low fat products is available in my neighborhood

4. There are many opportunities to purchase fast foods in my neighborhood such as McDonald’s, Taco Bell, KFC and takeout pizza places etc.

5. How much of a problem would you say that lack of access to adequate food shopping is in your neighborhood?

Perceived presence of food retail outlet‡

Which of the following stores, if any, are located in Your Neighborhood:

1. A supercenter such as Wal-Mart or Target

2. A supermarket such as Food Lion, Kroger, Publix, or Piggly Wiggly

3. A smaller grocery store

4. A convenience store with or without a gas station attached

5. A specialty store such as ethnic specialty store, meat market, seafood market, green grocer, or bakeries

6. A freestanding drug store or pharmacy Store such as CVS, Rite-Aid, Eckerd’s, or Walgreen’s

7. A dollar variety, dollar general, dollar store, or dollar tree

8. A franchised fast food restaurant including places like McDonalds, Subway, or Taco Bell

9. A sit down restaurant or buffet restaurant

^*For each of the following statements, please think of your neighborhood as the area within a 20 minute walk or about a mile from your home;

^†The responses to above questions were coded on a Likert scale ranging from 1 to 5 (1=strongly agree, 2=agree, 3=neutral, 4, disagree, 5=strongly disagree) for question #1–#4 and ranging from 1 to 4 (1=very serious problem, 2=somewhat serious problem, 3=minor problem, 4=not really a problem) for question #5;

^‡Response options were simply “yes” or “no”.

We also asked the respondents whether they had certain types of food retail outlets available within their neighborhood (Table 1). This set of nine questions was newly developed for our survey. The same definition of the neighborhood was used. Response options were simply “yes” or “no”. The list of outlet types included supercenter, supermarket, smaller grocery store, convenience store, specialty store, drug store or pharmacy, dollar variety, franchised fast food restaurant, and sit down restaurant.

Statistical Analysis

The test-retest reliability on questions of perceptions of food environment was estimated using ICC described by Shrout and Fleiss (Shrout and Fleiss, 1979). ICC values range between 0 and 1, > 0.8 is considered excellent, 0.6–0.8 good, 0.4–0.6 moderate, and < 0.4 as poor agreement (Landis and Koch, 1977). The test-retest reliability on questions of perceived presence of food outlet was assessed using Phi coefficients (Cramer, 1946) and interpreted using the same qualitative categories as for the ICC. Because of likely differences in the foodscape for persons living in urban vs. rural areas, the reliability analysis was repeated after stratification. All analyses were performed in SAS 9.2 (Cary, NC).

RESULTS

Sample characteristics are presented in Table 2. The mean age of our sample was about 60 years and approximately 80% of the respondents were female. Two-thirds were non-Hispanic Whites and one third of respondents were African Americans and other minority race/ethnic groups. Approximately 75.5% of the respondents lived in rural areas. Difference on demographic variables was examined using the T-test for age and Chi Square analysis for the sex and race by urban-rural classification. The p values were 0.634 for age, 0.689 for sex and 0.820 for race, which meant there was no statistical difference on demographics between rural and urban residents in this sample.

Table 2.

Characteristics of study sample

Variables	N	Mean (SD) or Percentage (%)
Age	101	60.3 (12.5)
30–45	13	12.9
46–55	22	21.8
56–65	31	30.7
>65	35	34.7
Sex	101	100.0
Male	21	20.8
Female	80	79.2
Race*	100	100.0
Non-Hispanic White	67	67.0
African American and Other†	33	33.0
Area*	98	100.0
Urban	24	24.5
Rural	74	75.5

^*There are missing data on race (1 in 101) and urban/rural area (3 in 101). The proportions were calculated based non-missing data.

^†Other race group includes Asian, Native Hawaiian or other Pacific Islander, and American Indian or Alaska Native.

The reliability was good for questions assessing the availability of fresh fruit and vegetables in the neighborhood (0.60), availability of low fat products (0.62), opportunities to purchase fast food (0.66), the lack of access to food shopping (0.71) and the healthy food availability composite scale (0.71) (Table 3). Reliability was excellent for the perceived presence of a supercenter (0.96) and drug stores or pharmacies (0.83). Reliability was good for the presence of a supermarket (0.77), specialty store (0.65), dollar store (0.71), fast food restaurant (0.79) and sit-down restaurant (0.65). Reliability was moderate for questions on the availability of high quality fresh fruit and vegetables (0.55), and the presence of a small grocery (0.51) and convenience stores (0.58). Table 4 shows the consistency of the answers to the questions on perceived presence of food outlets between the first and second interview. Over 90% of respondents answered consistently on the presence of supercenter, supermarket, specialty store, drug store or pharmacy, and franchised fast food restaurant in both interviews (yes-yes and no-no). For other types of food outlets, approximately 80% of the respondents provided the same answer.

Table 3.

Reliability of perception of neighborhood food environment questions (N=101)

Question	Call 1	Call 2	Reliability
Perceptions of food environment	Mean (SD)	Mean (SD)	ICC (95%CI)
Selection of fresh fruit and vegetables†	2.9 (1.5)	2.7 (1.3)	0.60 (0.46, 0.71)
High quality fresh fruit and vegetables†	3.0 (1.4)	2.9 (1.3)	0.55 (0.40, 0.67)
Large selection of low fat products†	2.8 (1.4)	2.5 (1.2)	0.62 (0.48, 0.73)
Average of healthy food availability scale*	2.9 (1.3)	2.7 (1.2)	0.71 (0.60, 0.80)
Opportunities to purchase fast food†	2.6 (1.5)	2.5 (1.4)	0.66 (0.54, 0.76)
Lack of access is a problemΔ	2.9 (1.2)	2.9 (1.2)	0.71 (0.60, 0.79)
Perceived presence of food outlet	Yes (%)	Yes (%)	Phi Coefficient (95%CI)
Supercenter‡	12.9	11.9	0.96 (0.93, 0.97)
Supermarket‡	30.7	30.7	0.77 (0.67, 0.84)
Smaller grocery store‡	26.7	28.7	0.51 (0.35, 0.64)
Convenience store‡	50.5	54.5	0.58 (0.43, 0.69)
Specialty store‡	13.9	11.9	0.65 (0.52, 0.75)
Drug store or Pharmacy‡	26.7	28.7	0.83 (0.76, 0.88)
Dollar store‡	39.6	37.6	0.71 (0.59, 0.79)
Franchised fast food restaurant‡	25.7	23.8	0.79 (0.70, 0.85)
Sit down restaurant‡	33.7	35.6	0.65 (0.52, 0.75)

^†Range of scores: 1=strongly agree, 2=agree, 3=neutral, 4, disagree, 5=strongly disagree;

^Δ Range of scores: 1=very serious problem, 2=somewhat serious problem, 3=minor problem, 4=not really a problem;

^‡Categories: 1=yes, 2=no;

^*The average scores were calculated by taking the average across three items for perceptions of food environment (availability of fresh fruit and vegetables, quality of fresh fruit and vegetables, and availability of low fat products) (see Mujahid et al., 2007), and the overall reliability was estimated by the average scores of call 1 and call 2.

Table 4.

Answers of perceived presence of food outlets for first and second interview (N=101)

Perceived presence of food outlet	Answers for 1st and 2nd Call, Number (Percentage)
	1st=Yes,	1st=No,	1st=Yes,	1st=No,
	2nd=Yes	2nd=No	2nd=No	2nd=Yes
Supercenter	12 (11.9%)	88 (87.1%)	1 (1.0%)	0
Supermarket	26 (25.7%)	65 (64.5%)	5 (5.0%)	5 (5.0%)
Smaller grocery store	18 (17.8%)	63 (62.4%)	9 (8.9%)	11 (10.9%)
Convenience store*	42 (42.0%)	36 (36.0%)	9 (9.0%)	13 (13.0%)
Specialty store	9 (8.9%)	84 (83.2%)	5 (5.0%)	3 (3.0%)
Drug store or Pharmacy	25 (24.8%)	69 (68.3%)	2 (2.0%)	5 (5.0%)
Dollar store*	32 (32.0%)	54 (54.0%)	8 (8.0%)	6 (6.0%)
Franchised fast food restaurant	21 (20.8%)	72 (71.3%)	5 (5.0%)	3 (3.0%)
Sit down restaurant	27 (26.7%)	58 (57.4%)	7 (6.9%)	9 (8.9%)

^*Only 100 respondents answered this question in both interviews.

Urban residents demonstrated better reliability on questions pertaining to opportunities to purchase fast food and perceived presence of a supercenter than rural residents (both p<.05) (Table 5). Though no other differences reached statistical significance, almost all the ICCs for the other perceptions questions including healthy food availability composite scale were consistently higher for urban than rural residents.

Table 5.

Reliability of perception of neighborhood food environment questions by urban and rural (N=98)

Questions	Urban (N=24)			Rural (N=74)
	Call 1	Call 2	Reliability	Call 1	Call 2	Reliability
Perceptions of food environment	Mean (SD)	Mean (SD)	ICC (95%CI)	Mean (SD)	Mean (SD)	ICC (95%CI)
Selection of fresh fruit and vegetables†	3.9(1.4)	3.8(1.2)	0.68 (0.40,0.84)	2.6(1.4)	2.4(1.2)	0.49 (0.30,0.64)
High quality fresh fruit and vegetables†	3.8(1.2)	3.8(1.1)	0.63 (0.32,0.82)	2.7(1.3)	2.6(1.3)	0.48 (0.28,0.64)
Large selection of low fat products†	3.5(1.2)	3.6(1.1)	0.73 (0.47,0.87)	2.6(1.3)	2.2(1.1)	0.49 (0.30,0.65)
Average of healthy food availability scale§	3.8(1.2)	3.7(1.0)	0.77 (0.54,0.89)	2.6(1.2)	2.4(1.1)	0.61 (0.44,0.74)
Opportunities to purchase fast food†	3.4(1.4)	3.5(1.5)	0.90 (0.79,0.95)*	2.3(1.4)	2.1(1.2)	0.47 (0.27,0.63)*
Lack of access is a problemΔ	3.8(0.5)	3.7(0.7)	0.61 (0.30,0.81)	2.6(1.2)	2.4(1.1)	0.65 (0.50,0.76)
Perceived presence of food outlet	Yes (%)	Yes (%)	Phi Coefficient (95%CI)	Yes (%)	Yes (%)	Phi Coefficient (95%CI)
Supercenter‡	33.3	33.3	1.00*	5.4	4.1	0.86 (0.79,0.91)*
Supermarket‡	62.5	66.7	0.73 (0.46,0.88)	17.6	17.6	0.72 (0.59,0.81)
Smaller grocery store‡	29.2	37.5	0.35 (0.06,0.72)	25.7	27.0	0.55 (0.37,0.69)
Convenience store‡	73.9	78.3	0.41 (0.00,0.70)	43.2	47.3	0.54 (0.35,0.68)
Specialty store‡	50.0	41.7	0.51 (0.13,0.76)	2.7	1.4	0.70 (0.56,0.80)
Drug store or Pharmacy‡	66.7	70.8	0.91 (0.80,0.96)	14.9	16.2	0.74 (0.62,0.83)
Dollar store‡	58.3	56.5	0.55 (0.19,0.78)	33.8	32.4	0.79 (0.68,0.86)
Franchised fast food restaurant‡	58.3	54.2	0.75 (0.50,0.89)	13.5	12.2	0.70 (0.56,0.80)
Sit down restaurant‡	50.0	54.2	0.59 (0.24,0.80)	28.4	29.7	0.64 (0.48,0.76)

^†Range of scores: 1=strongly agree, 2=agree, 3=neutral, 4, disagree, 5=strongly disagree;

^Δ Range of scores: 1=very serious problem, 2=somewhat serious problem, 3=minor problem, 4=not really a problem;

^‡Categories: 1=yes, 2=no;

^*The difference between urban and rural is significant. Non-overlapping confidence intervals is indication that the ICCs and Phi values were different;

^§The average scores were calculated by taking the average across three items for perceptions of food environment (availability of fresh fruit and vegetables, quality of fresh fruit and vegetables, and availability of low fat products) (see Mujahid et al., 2007), and the overall reliability was estimated by the average scores of call 1 and call 2.

DISCUSSION

Our study builds on two previous publications on psychometric characteristics of perceptions of food environment questionnaires (Echeverria et al., 2004; Mujahid et al., 2007). The validity of the questionnaire compared to direct measures of food environment has been reported (Moore et al., 2012). The sensitivity and specificity were reported as 79.6% and 46.8%, respectively. When it comes to reliability, in the study by Echeverria et al. (2004), excellent test-retest reliability of a 6-item scale was reported with an ICC of 0.88 (95% CI 0.79–0.93). However, it was a pilot study based on only 48 volunteers living in one urban location and questions focused exclusively on ease of purchase, available selection and quality of fruit and vegetables and low-fat products. Mujahid et al. (2007) reported on the reliability of a 3-item healthy food availability scale based on a study of 120 residents sampled in multiple locations. They found good reliability with an overall ICC of 0.69 (95% CI 0.57–0.77). In our study, the reliability of the five individual food environment perceptions questions ranged from 0.55 to 0.66. When we applied the same approach as Mujahid et al. (2007) and calculated the ICC of the healthy food availability scale based on three questions, we found the ICC to be substantially higher at 0.71 (95% CI 0.60–0.80), consistent with Mujahid et al. (2007).

Our study adds to the existing literature by additionally examining the psychometric properties of questions on perceived presence of specific food outlet types. In the present study, we found the reliabilities were good or excellent for most types of food outlets except smaller grocery stores and convenience stores, for which the reliability was moderate. Supercenters comprised the most reliable outlet category, possibly due to their limited number in the area, followed by drug stores or pharmacies. Smaller grocery stores and convenience stores were less reliable than supercenters; this may be caused by the high neighborhood density of such stores and misclassification with other types of food outlets.

To the best of our knowledge, no previous studies have compared the test-retest reliability of perceptions of food environment between urban and rural residents. We found that urban residents reported more reliably on all questions on perceptions of food environment, including the healthy food availability composite scale, than rural residents, with one question reaching statistical significance. Reliability was higher for urban residents on perceived presence of supercenters than for rural residents. Discrepancies on supercenters may be because supercenters are sparse in the area and the distance to the supercenter is shorter in urban areas. We used the same buffer (1 mile or 20 minutes walk) for both urban and rural residents. It is conceivable that this buffer may not be entirely appropriate for rural residents and that some rural residents may mentally extend the buffer as they answer the questions. Given that, the literature on perceptions of food environment in rural areas is still limited (Garasky et al., 2004; Sharkey et al., 2010; Smith and Morton, 2009), our findings suggest the need for additional research.

Our study has several limitations and strengths. Because we selected the household food shopper as the respondent, most of the participants were women, which limits the generalizability of our findings. Second, our landline-based telephone sample yielded an age distribution with an average age in the middle-to-older age category, which does not represent all rural residents. However, the larger study aimed to achieve good geographic coverage of the study area which precluded the use of a dual sampling scheme including cell phone numbers, because the latter are not linked to geography. Third, the average time between the two survey administrations was 35.4 days (SD=8.2days) which is more than double than in the study by Mujahid et al. (2 weeks). Fourth, due to the limited sample size, we coded urbanicity into a binary variable as urban and rural, which may result in some misclassification of the rural continuum. In addition, due to the zip code-based sampling scheme in the main study, only 24 urban residents (24.5%) were included in the analysis. The small sample size may weaken the statistical power. Moreover, the response rate (67.3%) was reasonable but not high. Studies with lower response rates provide more opportunity for bias, but a low response cannot be equated with bias. However, we compared the demographic variables between participants of the reliability study (N=101) and participants of the larger study who were not invited to the reliability study (N=818), and no significant difference was found for age, gender, and race. Lastly, the lack of statistically significant differences between urban and rural residents for most of the items should be interpreted with caution given the small sample. Among the strengths of this study is that we found high test-retest reliability on a newly developed set of questions on the perceived presence of specific food outlet types. Our findings furthermore suggest that there may be different levels of reliability between urban and rural residents.

In conclusion, the majority of questions on perceptions of the neighborhood food environment appear highly reliable, including a set of questions on the perceived presence of specific food outlet types. With respect to potential differences in reliability between rural and urban residents, our findings suggest the need for more research.