Measurement of the Local Food Environment: A Comparison of Existing Data Sources

Abstract

Studying the relation between the residential environment and health requires valid, reliable, and cost-effective methods to collect data on residential environments. This 2002 study compared the level of agreement between measures of the presence of neighborhood businesses drawn from 2 common sources of data used for research on the built environment and health: listings of businesses from commercial databases and direct observations of city blocks by raters. Kappa statistics were calculated for 6 types of businesses—drugstores, liquor stores, bars, convenience stores, restaurants, and grocers—located on 1,663 city blocks in Chicago, Illinois. Logistic regressions estimated whether disagreement between measurement methods was systematically correlated with the socioeconomic and demographic characteristics of neighborhoods. Levels of agreement between the 2 sources were relatively high, with significant (P < 0.001) kappa statistics for each business type ranging from 0.32 to 0.70. Most business types were more likely to be reported by direct observations than in the commercial database listings. Disagreement between the 2 sources was not significantly correlated with the socioeconomic and demographic characteristics of neighborhoods. Results suggest that researchers should have reasonable confidence using whichever method (or combination of methods) is most cost-effective and theoretically appropriate for their research design.

The association between neighborhood context and health is well documented. For example, residents living in socioeconomically disadvantaged neighborhoods have an increased risk of chronic health conditions such as obesity (1, 2), respiratory problems (3, 4), and coronary heart disease (5, 6), independent of individual risks. Additionally, residents in these neighborhoods have higher rates of smoking (7), poor nutrition (8, 9), and physical inactivity (7, 10–12). However, to provide better guidance on how policy interventions can address neighborhood health disparities, research must focus on more specific aspects of neighborhood environments that lead to increased health risks and unhealthy behaviors.

Until recently, researchers have relied mainly on census-based measures to assess associations between neighborhood characteristics and health (13–15), but they have increasingly turned to other methods to measure aspects of neighborhoods that are more theoretically linked to specific health outcomes. Two common sources of data on such neighborhood mechanisms are 1) spatially referenced secondary sources of data collected for other purposes (e.g., administrative data or market research data) and 2) direct observations of the physical and social environment made through systematic social observation (SSO) (16).

Secondary data sources, such as market research databases and municipal registries, can be very useful for measuring aspects of the environment related to the availability of food, exercise facilities, and health care (17–20). One advantage to using secondary data sources is that they usually provide complete geographic coverage of the area under study (e.g., all the stores that sell food in a city or metropolitan area), sometimes for a very limited cost. However, many health-relevant aspects of neighborhoods are not measured in secondary data sources—such as the level of disorder, the condition of buildings and recreational areas, and the type and price of food sold in local stores—so researchers often turn to other sources of contextual data.

Direct SSO of the neighborhood environment allows researchers to measure constructs not commonly found in secondary data sources by using a field instrument tailored to their needs (21–23). This method is usually more costly because it requires training raters and paying for their time, but it can be performed by survey interviewers who are already in the field, at a relatively low marginal cost. An important limitation to using SSO is that geographic coverage is typically more limited than it is with secondary data because researchers must pay for rater training and transportation to the sites being observed.

Both of these options represent promising alternatives to census data, but there are important unanswered questions concerning the “ecometric” properties of neighborhood-level measures constructed from such data sources, including issues of reliability, validity, and whether measures drawn from either source are more or less reliable or valid in certain types of neighborhoods (13, 24, 25). Although there is no “gold standard” for measuring neighborhood characteristics against which to compare measures from secondary data or SSO, we can begin to address these questions by assessing the level of agreement between equivalent measures derived from each method.

Another measurement concern that can be addressed empirically is whether the level of agreement between the 2 sources varies systematically across neighborhoods (26). For example, research has shown that people living in disadvantaged areas with few grocery stores are at higher risk of obesity (27, 28). If, however, a study relies on secondary data that fail to capture smaller businesses such as “mom-and-pop stores,” and disadvantaged neighborhoods are more likely to have these smaller businesses (28, 29), then the observed relation between lack of grocers and obesity may be an artifact of undercounting small stores in areas where obesity rates are highest. Systematic measurement error could also result from the potential failure of market research databases to count businesses in the “informal economy” or to include businesses that have recently closed, both of which could be more common in poorer neighborhoods. Likewise, SSO raters may make errors translating signs from other languages or not identifying multiple types of businesses at a single location.

In this study, we used 2 different data sources on the presence of stores that sell food or beverages in Chicago, Illinois, neighborhoods: a major market research database and an SSO covering the same time period. In this paper, we first assess the level of agreement between these 2 data sources at the block level, focusing on stores that sell food or beverages because proximity to these establishments has been suggested as one mechanism linking the residential environment to individual diet and health (18, 19, 27, 28, 30, 31). We also assess whether the level of disagreement regarding the presence of food stores between the 2 data sources is systematically related to the sociodemographic composition of neighborhoods. To the extent that the disagreement between the 2 sources is not correlated with key ecologic characteristics (e.g., those conventionally used in neighborhood-effects research), researchers can be more confident in using either method to explore the mechanisms that could help explain why neighborhoods matter for health.

MATERIALS AND METHODS

SSO data

Our observational data came from an SSO conducted as part of the Chicago Community Adult Health Study, a multistage area probability sample of 3,105 adults living in the city of Chicago (32). Each block on which one or more sampled residents lived was rated by trained observers. Raters were asked to walk around the perimeter of each block twice—once on the interior perimeter, including the block face on which the sampled resident lived, and once on the exterior perimeter, including the block face across the street from where the sampled resident lived—and to record their observations on a rating form. The rating form asked the raters to indicate only whether a given business type was present; raters were not asked to count the number of businesses, houses, or street features that they observed.

Observations were made between May 2001 and March 2003 on a total of 1,663 blocks in the city of Chicago sampled to be representative of all blocks in the city that contain residential housing (32). Each block is located in one of 343 neighborhood clusters, defined in previous studies and consisting of one or more geographically contiguous census tracts that follow major ecologic boundaries with relatively homogenous populations and comprise the entire area of the city of Chicago (33). Two observers rated 80 of the blocks to determine the interrater reliability for each item in the instrument; kappa statistics for interrater reliability of items measuring the presence of food and beverage stores ranged from κ = 0.37 (supermarkets) to κ = 1.00 (drugstores). We evaluated whether any of the following types of business establishments were present on the block: drugstores, convenience stores, liquor stores, bars, fast-food restaurants, other restaurants, greengrocers/delicatessens, and grocery stores/supermarkets. The definitions that raters used to code these establishments are presented in Table 1, along with interrater kappa statistics based on the 80 double-rated blocks for each business type. If a business could be categorized into multiple business categories (e.g., a bar and a restaurant), raters were instructed to code both business types as present on the block.

Table 1.

Definitions of Business Type Categories, SSO Variables, and Proprietary Business NAICS Codes Included in Business Type Categories, Chicago, Illinois, 2002

Business Type	SSO Variable Name: Descriptiona	NAICS Code: Descriptionb
Drugstores	Drugstores/pharmacy (interrater κ = 1.00): all drugstores, including large drugstore chains (e.g., Walgreens, Osco) that sell a wide variety of other merchandise. Count any store that provides a pharmacy for prescription medication.	446110: Pharmacies and drugstores engaged in retailing prescription or nonprescription drugs and medicines.
		Drugstores on sampled blocks: 89
Liquor stores	Liquor store (interrater κ = 0.36): includes any store that has alcohol as its primary merchandise. They may be of supermarket size or very small. Stores that sell alcohol with a range of foodstuffs would be considered supermarkets or convenience stores.	445310: Stores retailing packaged alcoholic beverages, such as ale, beer, wine, and liquor.
		Liquor stores on sampled blocks: 93
Bars	Bar/cocktail lounge (interrater κ = 0.77): includes places where the alcohol that is sold is consumed on the premises, and this is its main purpose, even if food is also provided. A “wine bar” would be included under this heading although they often offer a variety of meals. To qualify as a bar, it must be possible to obtain alcohol without also purchasing food.	722410: Bars, taverns, nightclubs, or drinking places primarily engaged in preparing and serving alcoholic beverages for immediate consumption. These establishments may also provide limited food services.
		Bars on sampled blocks: 181
Convenience stores	7-Eleven/convenience store (interrater κ = 0.61): includes small supermarkets open 18 to 24 hours selling a wide range of products including newspapers, food, small household items, toys, and stationery. Such stores may be part of a chain (e.g., White Hen Pantry, 7-Eleven) or privately owned. They may also be part of a gasoline service station if they sell a range of goods beyond just cigarettes, candy, and soft drinks.	445120: Convenience stores or food marts (except those with fuel pumps) primarily engaged in retailing a limited line of goods that generally includes milk, bread, soda, and snacks.
		447110: Gasoline stations in combination with convenience store or food marts.
		Convenience stores on sampled blocks: 137
Restaurants	Fast-food/take-out place (interrater κ = 0.64): includes chains (e.g., McDonalds, Burger King, Wendy's, Taco Bell) and restaurants offering only a limited range of “fast-food” items, such as pizza houses. This category includes sandwich bars, small coffee shops with limited seating, establishments that only have a take-out trade (e.g., some Chinese food, kebab houses), and any other eating place where no more than 2 or 3 people could linger to eat.	722211: Establishments primarily engaged in providing food services (except snack and nonalcoholic beverage bars) where patrons generally order or select items and pay before eating. Food and drink may be consumed on premises, taken out, or delivered to customers’ location. Some establishments in this industry may provide these food services in combination with selling alcoholic beverages. providing take-out services, or presenting live nontheatrical entertainment.
		Restaurants (fast-food) on sampled blocks: 941
	Other eating place/restaurant (interrater κ = 0.52): includes restaurants that have both take-out and eat-in services and restaurants where the space available for eating on the premises is greater than the space for customers to line up and wait for “take-out” items, except for fast-food chains (e.g., McDonalds), which are in the previous category.	722110: Establishments primarily engaged in providing food services to patrons who order and are served while seated (i.e., waiter/waitress service) and pay after eating. These establishments may provide this type of food service to patrons in combination with selling alcoholic beverages.
	To be considered an eating establishment, food should be the principal offering, or nonalcoholic beverages (e.g., coffee shops that provide seating and tables and also some food such as doughnuts, coffee cake, pastries, etc.)	722213: Snack and beverage: Establishments primarily engaged in 1) preparing and/or serving a specialty snack, such as ice cream, frozen yogurt, cookies, or popcorn or 2) serving nonalcoholic beverages, such as coffee, juices, or sodas for consumption on or near the premises. These establishments may carry and sell a combination of snack, nonalcoholic beverage, and other related products (e.g., coffee beans, mugs, coffee makers) but generally promote and sell a unique snack or nonalcoholic beverage.
		311811: Bakeries: Primarily engaged in retailing bread and other bakery products not for immediate consumption.
		Restaurants (“other eating place”) on sampled blocks: 200
		Restaurants (any) on sampled blocks: 1,052c
Supermarkets or grocers	Supermarket/grocery store (interrater κ = 0.36): includes stores that sell predominantly foodstuffs and small household items. The size of supermarkets may vary. Include large supermarkets (e.g., Jewel, Dominics) and also smaller food supermarkets. Some of these will be open 24 hours a day but they are distinguished from “convenience stores” by the fact that they provide a wider range of items.	445110: Establishments generally known as supermarkets and grocery stores primarily engaged in retailing a general line of food, such as canned and frozen foods; fresh fruits and vegetables; and fresh and prepared meats, fish, and poultry. Included in this industry are delicatessen-type establishments.
	Greengrocer/delicatessen (interrater κ = 0.52): includes “mom-and-pop” small neighborhood groceries and small specialized food stores, such as those selling food from one part of the world or one ethnic background, or health foods/vegetarian food shops. The store should be predominantly for the purchase of foods that need to be prepared off the premises, NOT those offering mainly ready prepared food to “carry out” (such as sandwich bars).	445210: Establishments primarily engaged in retailing fresh, frozen, or cured meats and poultry. Delicatessen-type establishments primarily engaged in retailing fresh meat are included in this industry.
		445220: Establishments primarily engaged in retailing fresh, frozen, or cured fish and seafood products.
		445230: Establishments primarily engaged in retailing fresh fruits and vegetables.
		Supermarkets or grocers on sampled blocks: 437

Abbreviations: NAICS, North American Industry Classification System; SSO, systematic social observation.

^aThe variable descriptions are the instructions given to the SSO observers.

^bThe definitions for the NAICS codes are published by the US Bureau of the Census (http://www.census.gov/epcd/ec97/industry/).

^cThe total from any restaurant is less than the sum of fast-food restaurants and “other eating places” because they are not mutually exclusive categories.

Proprietary commercial establishment data

The proprietary data were purchased from InfoUSA (Omaha, Nebraska), a commercial vendor that tracks businesses and provides business listings based on the North American Industry Classification System (NAICS) codes, a standard created and used by government agencies to categorize businesses. This proprietary data set lists 23,868 businesses that had 1) addresses in the city of Chicago during November 2002, a month chosen because 84% of the SSOs were collected within 3 months on either side of that month; and 2) NAICS codes that identified them as drugstores, convenience stores, liquor stores, bars, fast-food restaurants, other restaurants, greengrocers/delicatessens, or grocery stores/supermarkets. Definitions of these business categories and their corresponding NAICS codes are listed in Table 1, along with the number of such establishments in the proprietary data across all sampled blocks. Each establishment could identify itself with up to 4 NAICS codes. We classified a business as belonging to a specific food/beverage store category if at least one of the reported NAICS codes was in the food/beverage store category (establishments with multiple NAICS codes were classified into multiple business categories). Although the data proprietor provides a new list of businesses each month, it does not provide documentation regarding the frequency with which individual listings are verified; the lack of such information is a major disadvantage in using these data.

The data set included addresses and geographic coordinates for each business. After inspecting the data, we found small discrepancies between the coordinates provided by the data proprietor and the coordinates obtained by geocoding the addresses ourselves. Since the proprietor did not document the specific procedures used to obtain geographic coordinates for businesses, we geocoded the addresses of all of the 23,868 businesses using US Census Bureau TIGER/Line files (http://www.census.gov/geo/www/tiger/) and ArcGIS version 9.2 software (ESRI, Redlands, California). We were unable to obtain geocoded coordinates for 1,019 businesses. We conducted the analysis 1) without these observations (which we report in the tables) and 2) with the proprietor-provided coordinates for these observations, and we found no significant variation in our results.

Creating comparability between data sets

We used a multistep process to construct comparable measures across the 2 data sources. The first step created comparable business categories by matching the instructions used by the SSO raters to the NAICS codes used to classify businesses in the proprietary data. This step required collapsing some categories in both data sources. Table 1 provides the business categories used in the analysis and how they are defined in each data source.

Since the availability of fast-food restaurants has been considered particularly important to the health of residents (19, 30, 34), we also created 2 subcategories of restaurants: fast-food restaurants and “other eating places.” The “fast-food” category in the SSO data was compared with the “limited-service restaurants” category (NAICS code 722110) in the proprietary data (refer to Table 1 for more detail), whereas the “other eating places” category from the SSO (which captured any non-fast-food restaurant in the SSO) was compared with a roughly analogous category in the proprietary data that we created by counting the presence of any of the following: a “full-service restaurant” (NAICS code 722211), a “snack and beverage” establishment (NAICS code 722213), or a bakery (NAICS code 311811).

The next step involved creating geographically comparable units of analysis. The SSO was conducted on only 1,663 of the approximately 24,000 blocks in Chicago, and we had to ensure that we were using the same blocks with the proprietary data. We used geographic information systems software to create polygons that represented the blocks observed by SSO raters and matched a business from the proprietary data to a polygon if the geographic coordinates were inside the polygon. A business category was coded as being present on the block if at least one business in that category was inside the block boundary; otherwise, it was coded as not present. In the final step, the block-level measures from the proprietary database were merged with the SSO database, resulting in a combined database that contained measures from both data sources of whether each business category was present on each block.

Analytic strategy

To measure the correspondence between the 2 data sources, kappa statistics were calculated for the 8 business types. The kappa statistic is a measure of intersource reliability, defined as the ratio of the observed agreement to the expected agreement (i.e., the level of agreement that could be expected by chance, based on the marginal frequencies in both data sources) (35). We then assessed whether disagreement between the 2 data sources is systematically associated with the sociodemographic characteristics of neighborhoods by fitting logistic regression models that estimate the log-odds of finding disagreement (a dichotomous variable) about the presence of a given business type on a block.

Independent variables measuring the sociodemographic characteristics of neighborhoods were constructed from Summary File 3 of the 2000 US Census (http://factfinder.census.gov/). We constructed a disadvantage scale (α = 0.94) by taking the mean of the z-score values of the following variables: percentage of households with annual incomes of less than $15,000, percentage of households with annual incomes of $50,000 or less, percentage of families living in poverty, percentage of households receiving public assistance, percentage unemployed, percentage of female-headed households, percentage of never-married persons, and percentage of owner-occupied households (reverse coded). The variables used to construct this scale were selected based on a factor analysis of census tract data conducted for previous research (32).

Two variables were included to capture the influence of racial/ethnic composition on disagreement: the neighborhood percentage non-Hispanic white and a Hispanic/foreign born scale, which is the mean of the z-score values of the percentage of Hispanics and percentage foreign born (α = 0.86). We also included the percentage of residents who lived in the same house for 5 years and the logged population per square kilometer to measure residential stability and population density, respectively.

To control for the geographic location of blocks in the city of Chicago, which is likely to influence business patterns, we included a measure of distance from the Loop—Chicago's central business district—operationalized as a block's distance in kilometers from the Sears Tower. To account for differences in when blocks were rated in the SSO, we controlled for the difference (in months) between the date the block was observed and November 2002, the month for which the proprietary data were obtained. All analyses were conducted by using Stata version 9.2 software (Stata Corporation, College Station, Texas).

RESULTS

The results of the correspondence analysis are reported in Table 2. The rate of agreement between the SSO and proprietary data was quite high, ranging from 82% (for fast-food restaurants and supermarkets or grocers) to 96% (for drugstores). Restaurants had the highest intersource reliability (κ = 0.70), convenience and liquor stores and “other” (non-fast-food) eating places had the lowest (κ = 0.32–0.38), and the other categories (drugstores, bars, fast-food restaurants, and grocery stores) had levels of agreement (range of κ = 0.44–0.49) that were above convenience and liquor stores and “other” (non-fast-food) eating places but well below restaurants.

Table 2.

Kappa Statistics and Tabulations of Agreement on City Blocks Between Data Collection Methods, Chicago, Illinois, 2002^a

Business Type	Observed Agreement, %	Expected Agreement, %	Kappa	Agree: Not Present, %	Agree: Present, %	Disagree: Present in SSO, %	Disagree: Present in Prop., %
Drugstores	95.8	92.4	0.45*	93.9	1.9	1.8	2.4
Liquor stores	90.9	85.2	0.38*	87.5	3.4	7.3	1.8
Bars	88.6	79.5	0.45*	82.8	5.8	8.6	2.8
Convenience stores	88.3	82.8	0.32*	84.7	3.6	8.1	3.6
Restaurants	87.4	58.7	0.70*	64.7	22.7	8.4	4.2
Fast-food restaurants	81.8	64.2	0.49*	67.7	14.1	6.3	12.0
Other eating place	83.8	74.7	0.36*	77.4	6.4	13.3	2.9
Supermarkets or grocers	82.5	68.8	0.44*	71.9	10.6	8.3	9.3

Abbreviations: Prop., proprietary data source; SSO, systematic social observation.

* P < 0.001.

^aN = 1,663.

To better understand the nature of disagreement between the 2 data sources, we cross-tabulated measurements from each source and report, in Table 2, the frequency of blocks on which 1) both data sets agreed that the business type is present, 2) both agreed that it is not present, 3) only the SSO data source indicated that the business type is present, and 4) only the proprietary data source indicated that the business type is present. The results show that, where disagreement occurred, it was more often the case that a business type was recorded in the SSO but not the proprietary data, especially in the case of liquor and convenience stores. Exceptions included drugstores, grocery stores, and especially fast-food restaurants (where the NAICS definition is more inclusive), which were reported slightly to moderately more frequently in the proprietary data than in the SSO.

Odds ratios estimated from the logistic regressions of block disagreement are reported in Table 3. The major finding was the lack of a consistent relation between the sociodemographic characteristics of neighborhoods and the odds of the 2 data sources disagreeing. Of the 40 socioeconomic and demographic coefficients that we estimated, only 9 were statistically significant. Three of the outcomes (all restaurants, fast-food restaurants, and supermarkets) had no significant sociodemographic predictors, and another 3 outcomes (drugstores, liquor stores, and bars) had only one significant sociodemographic predictor. The sociodemographic variables associated with disagreement were 1) proportion white (associated with more disagreement on non-fast-food restaurants), 2) the Hispanic/foreign-born scale (associated with more disagreement on bars and non-fast-food restaurants), 3) the disadvantage scale (associated with more disagreement on liquor stores), 4) the residential stability scale (associated with less disagreement on drugstores, convenience stores, and non-fast-food restaurants), and 5) population density (associated with more disagreement on convenience stores and non-fast-food restaurants).

Table 3.

Odds Ratios (Standard Errors) of Block Disagreement Within Neighborhood Clusters on Demographic Characteristics^a by Business Type, Chicago, Illinois, 2002^b

	Business Type
	Drugstores	Liquor Stores	Bars	Convenience Stores	Restaurantsc			Supermarket or Greengrocer
					Any Restaurant	Fast-Food	Other Eating Place
Proportion white	1.001 (0.008)	1.007 (0.006)	1.007 (0.006)	0.993 (0.005)	1.001 (0.005)	1.004 (0.004)	1.014 (0.005)**	0.994 (0.004)
Hispanic/foreign-born scale	0.878 (0.151)	1.173 (0.136)	1.660 (0.183)***	0.909 (0.093)	1.150 (0.111)	1.132 (0.097)	1.401 (0.135)***	1.122 (0.097)
Disadvantage scale	0.946 (0.282)	1.737 (0.316)**	0.947 (0.203)	0.931 (0.167)	1.346 (0.212)	0.940 (0.145)	1.027 (0.188)	1.318 (0.190)
Residential stability	0.971 (0.014)*	0.990 (0.010)	0.996 (0.009)	0.975 (0.009)**	0.997 (0.009)	0.996 (0.008)	0.985 (0.008)*	0.990 (0.008)
Population density	1.244 (0.311)	1.084 (0.180)	0.879 (0.129)	1.655 (0.276)**	0.999 (0.140)	1.222 (0.159)	1.354 (0.190)*	1.217 (0.159)
Months from November 2002	1.013 (0.027)	0.993 (0.022)	1.005 (0.018)	1.011 (0.018)	1.016 (0.018)	1.022 (0.015)	0.980 (0.016)	1.032 (0.015)*
Distance to the Loop	0.925 (0.030)*	0.957 (0.021)*	0.910 (0.019)***	0.973 (0.019)	0.980 (0.018)	0.931 (0.016)***	0.969 (0.017)	0.939 (0.016)***
Constant	0.034 (0.006)***	0.091 (0.010)***	0.101 (0.011)***	0.114 (0.011)***	0.134 (0.012)***	0.195 (0.015)***	0.169 (0.015)***	0.175 (0.014)***

* P < 0.05; **P < 0.01; ***P < 0.001.

^aRefer to the “Analytic strategy” part of the Materials and Methods section of the text for an explanation of these characteristics.

^bN = 1,663.

^c“Fast-food” and “other eating place” are mutually exclusive subcategories of “any restaurant” and, combined, constitute the “any restaurant” category. Refer to the text for details.

The nature and direction of the observed sociodemographic effects for each type of establishment seem reasonable, if only marginally more than might occur by chance. There are no sociodemographic predictors for large and inclusive categories (all restaurants, all supermarkets/grocery stores) or quite identifiable ones (fast-food). Findings drawing on the results shown in Tables 2 and 3 showed the following: 1) drugstores are identified less often via SSOs, with more disagreement in areas of high residential stability (where NAICS-identified drugstores may be “hidden” in larger stores); 2) liquor stores are identified less often in the proprietary data, with more disagreement in disadvantaged areas (where the proprietary data response rates may be lower for all stores or especially this type, which may be subject to heightened law enforcement observation if reported); 3) bars are more reported via SSOs and with more disagreement in Hispanic areas, where there may again be underreporting via NAICS; 4) convenience stores are more often reported via SSO, with more disagreement in dense areas and less disagreement in residentially stable areas (where they may be underreported in the proprietary data or more easily visible via SSO); and 5) other eating places are more often reported via SSO, with more disagreement in white or Hispanic areas and denser areas and less disagreement in stable areas (where the somewhat less inclusive NAICS categories may generally miss some types of eating establishments unless the area is very stable).

Distance to the Loop was also associated with less disagreement for 5 outcomes: drugstores, liquor stores, bars, fast-food restaurants, and supermarkets, likely reflecting lower commercial density further away from downtown Chicago (and conversely more purely residential blocks) and thus fewer opportunities for disagreement to occur (because there are fewer businesses). We tested this hypothesis by repeating the models reported in Table 3 using subsamples that contained only those blocks with commercial activity. We did not have official data on land-use patterns to measure commercial activity, so we identified commercial blocks in 2 ways. First, we constructed a subsample of 1,037 blocks with commercial land use by eliminating blocks where the SSO rater did not observe any commercial land use (regardless of whether the proprietary data recorded a food-related business there). We also constructed a second subsample of 656 blocks where any of the 6 business types were reported present on the block in the proprietary data, even if the SSO raters failed to observe a food-related business there. In both subsamples, we found nonsignificant associations between distance to the Loop and disagreement for all businesses except bars, but the associations between the sociodemographic characteristics and the outcomes were similar to those shown in Table 3.

DISCUSSION

The results of this analysis are promising for researchers investigating the role of the residential environment on individual outcomes. Although neither of these 2 data sources can be taken as a “gold standard,” the moderate to high levels of agreement between the 2 sources suggest that the presence of commercial establishments in residential neighborhoods is comparably measured across the 2 methods of data collection. The intersource reliability between the SSO and proprietary measures is comparable to interrater reliabilities obtained for SSO observations (21, 36) and is higher than the rates of agreement between respondent reports and proprietary databases reported in a prior study (17).

Across the 6 business types we investigated, we did not find a consistent pattern of association between sociodemographic characteristics and disagreement that would lead us to conclude that one data source consistently differs from the other in certain types of neighborhoods. The only consistent predictor of disagreement was distance to the Loop: disagreement was less likely in blocks further away from the center of the city. Supplementary analysis revealed that this association disappeared when the sample of blocks was limited to blocks with commercial establishments. This finding bolsters our interpretation that distance to the Loop is a proxy for commercial density, which creates more opportunity for disagreement simply because there are more businesses to rate or list.

Most types of businesses were more likely to be reported on a block by SSO observers than by proprietary business listings. We suspect that this is partly the result of the definitions used to classify the different businesses since the SSO rater instructions generally had more lenient definitions than the NAICS codes. It is also possible that SSO reports were more accurate than the proprietary business data listings because raters were able to obtain additional information about businesses, such as the merchandise or services that they sell or produce, by visiting them. Businesses classify themselves in the proprietary data by using NAICS categories, but there are no assurances that businesses will list all of the different categories they could be part of (and they can list as many as 4 only), and we know nothing about response rates.

Limitations

Although our results suggest that these data sources provide reliable estimates of the residential environment, several limitations should be noted. First, the SSO instrument was not designed to count the number of business establishments. Although we were able to compare how well these data sources capture the presence of businesses, not having data on the number of businesses meant that we could not compare measures of business density in neighborhoods. Future studies should compare both presence and density. Second, as we mentioned above, the definitions of the 6 business types differed slightly across the 2 data sources. This difference likely inflated the amount of disagreement on each block, resulting in conservative assessments of intersource reliability. It also suggests that future studies using SSO methodology to measure business establishments should use standard definitions, such as NAICS codes, to allow for maximum comparability between studies.

Finally, this study examined agreement across only the 2 data sources for 6 business types and examined these types in only a single city at a single point in time. Although Chicago has a wide diversity of neighborhoods, these analyses should be extended to other cities and types of residential environments including suburban and rural communities. Furthermore, we considered only those blocks with some residential housing units and hence cannot generalize our results to blocks with no residential land use. Thus, although this study provides evidence that the 2 data sources yield consistent measures of the residential environment, it represents only a first step in assessing the reliability between these data sources, and further investigations that include more items and more geographic locations are warranted. This analysis presents a straightforward strategy that can be used to assess the reliability of other measurements across different geographic contexts.

Conclusions

The levels of reliability found between these 2 methods of data collection suggest that researchers can use measures of the residential environment derived from either data source with roughly equal confidence. For studies aiming to characterize the neighborhood food environment, the data sources provide fairly comparable measures of the types of establishments present in the neighborhood; therefore, selection of one data source over the other should depend on the needs of the researcher and the costs and benefits unique to each method of data collection. For instance, the proprietary data provide complete geographic coverage, whereas the SSO relies on a sample of city blocks; thus, researchers interested in the total availability or density of food or retail establishments would probably find the proprietary data more cost-effective. On the other hand, the SSO provides researchers with a tool to collect more nuanced measures of the residential environment such as physical and social disorder and can probably make more nuanced distinctions among establishments because of the opportunity for direct observation. Where nuanced distinctions are necessary, these results suggest that SSO might be a more appropriate method. Therefore, both of these methods of data collection should be included in the toolkit researchers use to investigate the role of the residential environment on health, and they can be utilized as alternatives or complements, depending on the research objectives and financial resources of investigators.

Glossary

Abbreviations

NAICS

North American Industry Classification System

SSO

systematic social observation