Development and validation of a farmers’ market audit tool in rural and urban communities

Abstract

The number of farmers’ markets in the United States is growing. Although there are tools to analyze food availability at grocery stores, corner stores, and convenience stores, little research exists about the availability of food types at farmers’ markets. This research developed an audit tool to measure the food environment at farmers’ markets in rural and urban food environments and examined its psychometric properties, including face validity, inter-rater reliability, and discriminant validity. The Farmers’ Market Audit Tool (F-MAT) was reviewed by content experts, revised, and then tested in six farmers’ markets by researchers across three states in 2013, including Kentucky, North Carolina, and Montana. Seven food categories were developed, including vegetables, fruits, meats, cheeses, eggs, grains, and samples. Inter-rater reliability was high within farmers’ market across states. As expected, discriminant validity indicated a systematic disagreement within and between states due to seasonality and ability to grow different types of food across different farmers’ markets. The total scores assessing the healthfulness of each farmers’ market was 38 (range = 28 – 50). Utilizing the F-MAT at farmers’ markets is a reliable and valid method to capture the availability of food offerings.

Introduction

The United States population has consistently consumed a low-quality diet, with lower consumption of fruits, vegetables, and whole grains, and higher consumption of sugar-sweetened beverages (Popkin, 2010) and processed foods (Krebs-Smith, Guenther, Subar, Kirkpatrick & Dodd, 2010) over time. These dietary patterns have led some researchers and policy makers to focus on more upstream, distal determinants of diet and weight status, including the food environment (Story, Kaphings, Robinson-O’Brien, & Glanz, 2008). The food environment is broadly operationalized as the community food environment (defined as access to food venues) and the consumer food environment (defined as what consumers encounter in each food venue) Glanz, Salli, Saelens, & Frank, 2005). Neighborhoods with a higher proportion of minority, low-income residents have fewer healthy food retailers (Bodor, Rice, Farley, Swalm, & Rose, 2010) and fewer healthy foods within stores (Izumi, Zenk, Schulz, Mentz, & Wilson, 2011). The lack of healthy food options is associated with consumption of fewer fruits and vegetables and higher body mass index (Gustafson, Hankins, & Jilcott, 2012). Thus, ameliorating food environment disparities, such as inadequate access to healthy food retailers, is paramount to achieving national nutrition goals (Voss, Masuoka, Webber, Scher, & Atkinson, 2013). However, the food environment is a rather complex construct and, to date, standard measures for measuring the healthfulness of venues has not been established.

Background

To begin to address disparities in access to healthy foods, standard assessment tools are needed. To assess the consumer food environment, food store audit tools measure healthy and unhealthy food availability, price, promotion, placement, and quality (Cummins & Macintyre, 2006; Gustafson et al., 2012; Kelly, Flood, & Yeatman, 2011). Such tools have been used in a wide variety of traditional and non-traditional food venues (McKinnon et al., 2009). Desirable features of a tool include an established scoring protocol, adequate characterization of healthy versus unhealthy foods, assessment of price, availability, quality, and promotion of foods (as these are likely to influence consumer behavior), reliability and validity. To date, most tools have accessed the consumer food environment within supermarkets, grocery stores, or convenience stores (Kelly et al., 2011) but have not been able to capture non-traditional food venues such as farmers’ markets.

Between 2011 and 2012, there has been a 9.6% increase in the number of farmers’ markets operating in the United States; 7,175 in 2011 and 7,864 in 2012 (United States Department of Agriculture, 2013). While the total number of farmers’ markets is relatively low compared to the number of supermarkets in the United States (FMI, 2013), there has been consistent growth in operations among farmers’ markets relative to the slow or stagnant growth among supermarkets and grocery stores (Ibis World, 2014). Given the growth in number of farmers’ markets and the federal food aid programs that promote use of farmers’ markets (Kim, 2011), a tool to access the foods available at farmers’ markets is needed (Measures Registry, 2014). Thus, the aim of this study was to develop an audit tool to measure the food environment at farmers’ markets and examine its psychometric properties, including face validity, discriminant validity, and inter-rater reliability.

Methods

Study Setting

This study was conducted in six counties (one rural and one urban county in each state) in North Carolina, Kentucky, and Montana. These states were selected based on coauthors’ membership in the CDC-funded Nutrition and Obesity Policy Research and Evaluation Network (NOPREN) (Nutrition and Obesity Policy Research and Evaluation Network, 2013). Farmers’ markets meeting the following criteria were included in the sampling frame: 1) open at least five months per year; and 2) listed on the state department of agriculture listing. Produce stands, community supported agriculture, roadside stands, or personal home gardens were excluded. A list of farmers’ markets for each state was collected from each state’s department of agriculture. Once farmers’ markets addresses were collected, each county was coded as rural (0) or urban (1) based on the 2013 United States Department of Agriculture rural-urban continuum codes (Rural-Urban Continuum Codes, 2013) (RUCC). RUCC range from 1–10, with 1 being metro and 10 being non-metro. Counties designated as 1, 2, or 3 were classified as urban, and counties designated as 4–10 were classified as rural. Researchers limited this study to a driving radius of two hours from the researcher’s place of work. One urban and one rural farmers’ market in the sampling frame were randomly selected from the master list using a random number generator. If two farmers’ markets existed in the selected county, the market registered by the county and included on the state Department of Agriculture list was selected. If the farmers’ market was open more than one day per week, the market manager was contacted and asked which day had the most vendors and customers. After market selection, each market was contacted to ensure that the market manager was willing to allow data audit collection, to verify the address; and to verify that there was a minimum of three vendors selling produce at the market. A total of six farmers’ markets were selected for this study. As we did not collect data on human subjects, this study was exempt from review by an Institutional Review Board.

Development of Farmers’ Market Audit Tool and Face Validity

The Farmers’ Market Audit Tool (F-MAT) is a short paper and pencil form that records the availability of food items within the market, as well as key characteristics about the operations of the market. The F-MAT was developed based on our collective experience measuring the consumer food environment in non-traditional food venues.

First, authors conducted a review of available audit tools that measure availability, placement, promotion, quality, and price within grocery stores, supercenters, supermarkets, and convenience stores. There were many consumer food environment assessment tools, including tools to examine the availability of ethnic foods, such as the Hmong Food Store Survey (Franzen & Smith, 2010) and the TxNEA-S (Gloria & Steinhardt, 2010), or for tools specific to urban areas, such as the Baltimore Healthy Stores Project (Song, 2009). However, we found no audit tool to assess the, availability and quality of food found at farmers’ markets. Thus, we based the newly developed F-MAT on the Nutrition Environment Measurement Survey – Stores (NEMS-S) (Glanz, Sallis, Saelens, & Frank, 2007).

To establish face validity, we circulated the first draft of the F-MAT to content experts. The experts were given an explanation of the purpose of the tool, the survey and accompanying instructions, and were asked to review the F-MAT for its apparent ability to measure the food environment at farmers’ markets.

Farmers Market Audit Tool Pilot-Testing

The research team in each state consisted of two auditors (undergraduate and/or graduate students) and one lead researcher (study authors). Each lead researcher trained the auditors on use of the F-MAT. In addition to in-person training, written user guides were given to each auditor to promote inter-rater reliability. During June and July 2013, pilot testing of the tool included one audit of each selected farmers’ market.

To examine inter-rater reliability, the two auditors on each team attended the farmers’ market on the same day and at the same time. The researchers conducted the farmers’ market audit as soon as possible after opening hours to ensure that all food items available for consumers to access were stocked. Auditors then independently (without consulting one another) completed all information required on the F-MAT: counting the total number of vendors and unique number of vendors that sold fruits and vegetables; finding each type of indicated produce, meat, egg, cheese, and grain on the tool and indicating availability (Yes/No), number of vendors selling each item (#), and quality (Acceptable/Unacceptable); and number of vendors offering samples and number of vendors offering fruits or vegetables as samples.

Data Analysis

Using the point system found on the audit sheet, which was adapted from the NEMS-S, a total score was derived based on each section’s individual scores. Data were coded for number of agreements and number of disagreements between paired F-MATs in the same state for each section on the F-MAT (vendor information, vegetables, fruits, meats, cheeses, eggs, breads, and samples). Inter-rater reliability was assessed by percent agreement within each market. It was hypothesized that there would be high inter-rater reliability in the same market. Discriminant validity on the F-MAT was assessed by kappa coefficients between markets within each state, between markets among states, and between rural and urban farmers markets. Kappa values were calculated for discriminant validity using the audit tool scores, whereas percent agreements were calculated for inter-reliability by examining number of agreements among raters It was hypothesized that there would be low agreement between scores for markets in different states (because of differences in local food availability) and low agreement between scores for urban versus rural markets, indicating discriminant validity. Kappa values were quantified using the following scale: 0.01–0.20 slight agreement; 0.21-.0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 substantial agreement; 0.81–0.99 high agreement (Landis & Koch, 1977). Negative agreement values were interpreted as the raters agreed less on an item than expected by chance (e.g., there was a systematic disagreement between observers due to diversity of food items available at each farmers’ markets). Stata (version 12.0, StataCorp) was utilized to calculate all statistics.

Results

Face Validity

Content experts provided expert input to ensure face validity (n=8; listed in acknowledgements). From reviewer input, final revisions were made (See Table 1 for audit tool and revisions and Table 2 for final tool). –insert Tables 1 and 2 here- The final F-MAT begins with information gathering about the market, including hours of operation, seasonal openings, address, and number of vendors (to assess size). To be consistent with previous audits and to be able to generalize the overall availability of healthy food items at a given farmers’ market, we focused on fruits, vegetables, meats, poultry, eggs, cheeses, and grains. These items represented what is commonly sold at various farmers’ markets across the United States. These items also represented some of the key foods and food groups within the 2010 Dietary Guidelines for Americans (Dietary Guidelines for Americans. 7th ed., 2010) and the Healthy People 2020 nutrition objectives (Healthy People 2020 Topics & Objectives – Objectives A-Z). The final tool comprised of 27 unique items. Some items are counted as one but include subcategories, such as ground beef that is lean or standard.

Table 1.

Establishing Face Validity for Farmers’ Market Audit Tool (F-MAT): Changes made to the F-MAT after expert review.

Item	Change	Reason for change
F-MAT Instructions	Further defined farmers’ market inclusion criteria	Provide raters with more information to define farmers’ market
F-MAT Instructions and Tool	Match order of instructions with tool	Ease of completing tool
F-MAT Geographic Information	Include information about location of farmers market in respect to other community locations	Provide context for understanding who the market serves and how it increases food access
F-MAT Tool Food Item Production Methods	Tool does not specify production methods (e.g., organic versus non-organic) among produce, cheeses, or grains	Lack of production method labeling on all farmers’ market products
F-MAT Tool Price	Tool does not specify price for food items	Lack of price labeling on all farmers’ market products
F-MAT Tool Bread/Grain Products	Tool does not specify grain products other than Plain Bread, Sweet Bread, or Plain – 100% Whole Wheat/Grain Bread	Exclusion of ‘Other whole grain products’ to decrease variability across rater’s knowledge of whole grain
F-MAT Tool Samples	Include number of vendors offering samples	Individuals frequently offer free samples of processed and fresh foods

Table 2.

Farmer’s Market Audit Tool (F-MAT)

Section	Variables
Audit information	Auditor and market identification; Date and start and end time of audit
Farmers’ market information	Address; Days, hours, and months of operation
Vendor assessment	Number of vendors; Number of fruit and vegetable vendors; Total number of unique fruits and vegetables; Number of food, snack, or meal vendors
Vegetable (tomato, squash, onion, cabbage, salad green, dark leafy green, broccoli, corn, cucumbers, bell peppers, hot peppers, cauliflower)	Availability; Number of vendors selling; Quality
Fruit (apples, strawberries, blueberries, watermelon, peaches, plums, cantaloupe)	Availability; Number of vendors selling; Quality
Meats (pork loin conventional, pork loin pastured, lean ground beef conventional, lean ground beef grass fed, chicken conventional, chicken pastured, fish, shell fish)	Availability; Number of vendors selling; Quality
Cheese (regular, organic, nutrition information, low calorie version)	Availability; Number of vendors selling; Quality
Eggs (regular, free range)	Availability; Number of vendors selling; Quality
Grains (plain white bread, sweet breads, plan 100% whole wheat or whole grain)	Availability; Number of vendors selling; Quality
Samples	Number of vendors offering; Number of fruit or vegetable offering

Food Availability and Quality at Farmers Markets

The mean score for food availability and quality at farmers’ markets was 38 (range = 28 – 50). Three of the farmers’ markets fell below the average while three were above the mean score.

Inter-rater reliability – Within Farmers Markets

Results from the inter-rater reliability (Table 3) testing indicate that there was a wide range of agreement between the raters within each farmers’ market and category of food across states. Across all sites but one there was high to substantial agreement among the sites. One site (North Carolina Urban Farmers’ Market 3) was reflected lower percent agreements. When food items were aggregated (e.g., fruit) the result suggests strong agreement between raters.

Table 3.

Inter-rater Reliability of Farmers’ Market Audit Tool (F-MAT)

		Kentucky		Montana		North Carolina
		Rural FM 1	Urban FM 1	Rural FM 2	Urban FM 2	Rural FM 1	Urban FM 3
Fruit and Vegetable Total		96.23	90.10	100.00	90.67	93.06	72.82
Pork Loin		100.00	100.00	100.00	100.00	100.00	100.00
Meat		100.00	100.00	100.00	87.50	87.50	87.50
Chicken		100.00	100.00	100.00	100.00	100.00	100.00
Cheese		100.00	100.00	100.00	100.00	100.00	50.00
Eggs		100.00	100.00	75.0	100.00	00.00	100.00
Bread		100.00	83.33	100.00	83.33	100.00	66.67
Samples		100.00	100.00	00.00	100.00	100.00	66.67

Discriminant Validity – Within States

Results (Table 4) assessing discriminant validity of scores within states, the kappa statistic, suggest overall low agreement between farmers’ markets within states. For each state, the fruit and vegetable, quality, and total score kappa was −0.50, indicating systematic disagreement among farmers’ market sites within states.

Table 4.

Discriminant Validity (Kappa Statistics) of the Farmers’ Market Audit Tool (F-MAT) between and within States

		Fruit and Vegetable Total	Quality	Pork Loin	Meat	Chicken		Bread	Samples	Total Score
Between States		-0.02	-0.02	1.00	-0.50	1.00	-0.50		-0.12	-0.05
Within Kentucky		-0.50	-0.50	0.00	0.00	0.00	1.00		0.00	-0.50
Within Montana		-0.50	-0.50	1.00	1.00	1.00	0.00		1.00	-0.50
Within North Carolina		-0.50	-0.50	1.00	1.00	1.00	0.00		0.00	-0.50
Rural Versus Urban		0.13	0.25	0.73	0.59	0.73	-0.20		-0.20	0.15

Discriminant Validity – Between States

Results (Table 4) assessing the discriminant validity of scores between states, the kappa statistic, suggest overall low-agreement. The fruit and vegetable and quality kappa was −0.02 indicating slight agreement. The agreement among sites for pork and chicken was 1.00, which indicates substantial agreement. The agreement for meat and bread was −0.50 indicating systematic disagreement.

Discriminant Validity – Rural Versus Urban

Results (Table 4) assessing agreement of scores between rural versus urban farmers market sites, the kappa statistic, suggest overall slight agreement between farmers market. Fruit and vegetable kappa was 0.13, indicating slight agreement among rural versus urban farmers markets. Pork and meat kappa were both 0.73, indicating substantial agreement among rural versus urban farmers’ markets.

Discussion

There has been increasing interest in promoting farmers’ markets to improve dietary intake (Brown, 2002; McCormack, Laska, Larson, & Story, 2010; Pitts, 2014). However, there has not been systematic assessment of farmers’ markets in terms of healthfulness or food availability and quality. Thus, the aim of this project was to develop and validate a farmers’ market audit tool that could be used across various farmers’ markets.

Face validity was confirmed by a diverse expert panel of reviewers. As our results also signified high inter-rater reliability, the F-MAT can credibly be used within farmers’ market sites by trained raters. The discriminant validity found between markets in different states, and in urban and rural areas within states indicates that the F-MAT is correctly capturing food availability and quality, as a higher score indicated more produce available at higher quality.

We found low agreement for fruits and vegetables among the various sites. One explanation for these differences is that the fruits and vegetables listed on the F-MAT were not available in very rural or in cold climates. Although the F-MAT was conducted in warmer months, there are vast regional differences in what types of fresh fruits and vegetables are available in various locations. The high agreement among the protein-based products suggests that these food items are more stable and not subject to temporal seasonal changes that are true of fruits and vegetables. Pork loin and chicken yielded high percent and kappa agreement because they were not available at several of the markets.

Previous studies have indicated that farmers markets are utilized by a broad range of consumers (Blanck, Nebeling, Yaroch, & Thompson, 2011; Byker, Shanks, Misyak, & Serrano, 2012; Jilcott Pitts, Wu, McGuirt, Keyserling, & Ammerman, 2013). Positive impacts for local economies have also been found (Brown, 2002). What is still less understood is the impact that farmers markets provide on food access and nutrient availability in communities (McCormack et al., 2010). Just as the NEMS-S laid foundation for understanding the food environment within retail stores (Glanz, 2007), this tool provides a means to advance our knowledge about the food environment at farmers’ markets.

There are several limitations to the development of this tool. Only three states were selected in the mid-Atlantic, central, and western region of the United States. Ideally more states would have been included in the development of this tool. The audits were done in the summer months and not year-round. However, we decided to conduct audits in the summer months because many states do not have farmers’ markets year-round.

Conclusion

The Farmers Market Coalition (a collaboration with the University of Wisconsin) is working to develop tools that allow more systematic evaluation of the benefits of farmers markets to communities (Farmers Market Coalition, 2014). The F-MAT is a reliable and valid method to assess food availability and quality at farmers’ markets. The measure was sensitive in that it captured differences between markets within states (urban versus rural) and between states. Future studies should focus on testing the F-MAT in a variety of rural and urban farmers’ markets, across states, and in different seasons. In addition, examining test-retest reliability would enable further refinement of the F-MAT. Associations between consumer eating behavior and farmers’ market food environments should be examined. Ultimately, such tools can inform policy change that support population health.