Impact of Policy and Built Environment Changes on Obesity-related Outcomes: A Systematic Review of Naturally-Occurring Experiments

Abstract

Policies and changes to the built environment are promising targets for obesity prevention efforts and can be evaluated as “natural”- or “quasi”-experiments. This systematic review examined the use of natural- or quasi-experiments to evaluate the efficacy of policy and built environment changes on obesity-related outcomes (body mass index, diet, or physical activity). PubMed (Medline) was searched for studies published 2005–2013; 1,175 abstracts and 115 articles were reviewed. Of the 37 studies included, 18 studies evaluated impacts on nutrition/diet, 17 on physical activity, and 3 on body mass index. Nutrition-related studies found greater effects due to bans/restrictions on unhealthy foods, mandates offering healthier foods, and altering purchase/payment rules on foods purchased using low-income food vouchers compared to other interventions (menu labeling, new supermarkets). Physical activity-related studies generally found stronger impacts when the intervention involved improvements to active transportation infrastructure, longer follow-up time, or measured process outcomes (e.g., cycling rather than total physical activity) compared to other studies. Only three studies directly assessed body mass index or weight, and only one (installing light-rail system) observed a significant effect. Studies varied widely in the strength of their design and studies with weaker designs were more likely to report associations in the positive direction.

Introduction

Over the past 50 years, changes to stimuli and environments have impacted diet and physical activity, resulting in the current obesity epidemic^1–3. Due to the magnitude of the epidemic, researchers and policy makers are increasingly interested in environmental and policy interventions as strategies for population-wide improvements in physical activity, diet and subsequent reductions in obesity ⁴. Unlike individually targeted strategies such as physical education and behavioral skills training that are commonly local and tend to be unsustainable, environmental changes have strong potential to promote and sustain behavior changes over a long time period ⁵. Studies suggest a positive association between living near parks, recreational facilities, and playgrounds and higher levels of physical activity in adults and children ^{4, 6, 7}, and suggest a positive association between quality of the food environment, better quality diet ⁸ and a lower prevalence of obesity ⁹.

However, a number of study design challenges are present when examining how populations respond to environments and policies that relate to obesity. Most studies have been traditional observational studies in which exposures are pre-existing and defined via an investigator-defined rubric of lower and higher levels of favorable or unfavorable environmental conditions and outcomes are assessed cross-sectionally or over time ¹⁰. Traditional observational studies have rich data and allow assessment of correlations between environmental features and outcomes; however, there are known challenges with these designs. Cross-sectional designs are prone to positive bias due to residential self-selection into environmental conditions ¹⁰; prospective studies are often biased toward the null because environments remain relatively static over a long period of time which allows for assessment of the cumulative exposures but may necessitate life-course approaches in order to observe large changes in outcomes. Ideally researchers would approximate a randomized controlled trial (RCT), the gold standard for assessing causal effects, by randomly allocating environmental change in the target population ^{11, 12}. However, environmental and policy changes are implemented in the real world, on a large scale, and funded by public dollars, all conditions that make it very difficult to randomly allocate populations to treatment ¹³. Instead, environmental and policy changes are evaluated as natural- or quasi-experiments where investigators do not control or withhold allocation to the exposure of interest. For example, outcomes of interest can be compared between populations newly exposed to policies or environmental changes and unexposed (treatment and comparison groups), and/or to compare changes within the same populations before and after a policy goes into effect (pre-post observations) ^{14, 15}.

Policy makers are searching for viable population-based solutions to reverse the obesity epidemic and have been increasingly funding broad environmental and policy interventions. Rigorous science is needed to evaluate these natural- or quasi-experiments^{11, 16, 17}. This article reviews the state of the science on this topic from a substantive and methodological perspective; in particular: 1. Which policies and built environment changes have been evaluated via natural- or quasi-experiments and the results from these studies; 2. Study design issues which includes methods of assessment; and 3. Limitations and areas where additional science is needed.

Methods

Study scope

A systematic review was conducted to identify all published studies in the medical literature relating to natural- or quasi-experiments in obesity research.

Search and selection processes

We searched PubMed (Medline) to identify English-language natural- or quasi-experiments published between January 1, 2005 and January 1, 2014 that had an abstract. We conducted two separate searches on key words in a study’s title or abstract and included all articles that were identified by both. The first search used the following words: “eating”, “diet”, “nutrition”, “BMI”, “Body Mass Index”, “obesity”, “body weight”, “overweight”, “weight gain”, “weight loss”, “adiposity”, “physical activity”, “physically active”, “exercise”, “energy expenditure”, “bike”, “bicyl*”, “walk*”, “built environment”, “food environment”, “physical activity environment”. The second search used: “evaluation”, “policy”, “implementation”, “pre-post”, “pre-policy”, “differences in differences”, “difference in differences”, “difference-indifferences”, “time series”, “time series”, “quasi experiment”, “quasi-experiment”, “social experiment”, “natural experiment”. The asterisks enabled the search to include all words beginning with “bicyl” and “walk”. Because studies of natural experiments are not always described as such, it proved necessary to include broad search criteria and to supplement the literature search with other papers based on expert knowledge of the topic.

We included studies where 1) the intervention was a natural event due to a new policy (defined as municipal or federal government regulations and laws including school district policies) or change to the built environment that could affect physical activity, diet, or obesity; and 2) where the study collected data on obesity-related outcomes, which we defined as body mass index (BMI), weight, diet, and physical activity.

We included only studies that met our definition of a natural- or quasi-experiment, specifically: 1) studies where investigators did not control allocation of the intervention and intervention was not a randomized trial ¹⁸; 2) the exposure was well-defined (a sharp difference in conditions) and not a rubric defined by the investigators ¹⁹; and 3) participants were not able to knowingly self-select into the treatment group ²⁰. The later criteria meant that we excluded relocation studies like the RESIDential Environment Project ²¹ and Moving to Opportunity ¹⁸ due to the probability that participants self-selected into the new housing development or neighborhoods. Natural experiments presume that assignment to intervention and comparison groups is random or “as if” random ^22–24. Only 13 of the studies reviewed met that criterion. The rest would be considered quasi-experiments rather than natural experiments. However, we retain the word natural experiments for the remainder of this review because most quasi-experiments are not naturally occurring and because in the field of public health, many studies refer to themselves as natural experiments even in the absence of random assignment²².

We excluded the following types of studies: 1) Studies that evaluated programs but were not policies or changes to the built environment; 2) Because our primary interest was in larger-scale population-level interventions, we excluded interventions that were operationalized at the individual level ²⁵ and excluded studies of interventions that only affected one building or one office space ²⁶; 3) Health promotion media campaign or communications ²⁷; 4) Where the first measurements of intervention effects were done a long time after the intervention was first implemented (>10 years ²⁸); 5) Case-only one-time measurements; 6) Qualitative studies, non-empirical studies such as review articles, published study protocols, and published work that was not peer-reviewed; and 7) Studies where the outcome was not measured in human subjects (e.g. menu offerings, food prices without measuring consumer purchases, store inventories ²⁹ or animal studies).

Data abstraction and data synthesis

One investigator (SLM) screened 1,175 abstracts and two investigators (SLM, AHA) reviewed 115 full-text articles for inclusion, which included 15 articles that were not identified in the PubMed search but were identified based on expert knowledge ^{28, 30–43}. Figure 1 outlines the review process.

Figure 1.

Study Flow Diagram

One reviewer (SLM) abstracted relevant data from each included study: study design, timing of data collection relative to the natural experiment, sample location, study population, sample size and sampling methodology, outcomes, statistical analyses, and results. A second reviewer (AHA) checked the abstraction. We classified outcomes into three categories: 1) obesity/weight/BMI, 2) physical activity, and 3) nutrition or diet.

We evaluated all the included studies with regards to key methodological criteria related to study design. We created the following rubric: Strongest design (+++): Within-person longitudinal studies with a comparison group; Intermediate design (++): Within-person longitudinal case-only studies, or repeat cross-sectional studies with a comparison group, including time series; Weakest design (+): One-time cross-sectional studies with a comparison group, or repeat cross-sectional case-only studies, including time series. The rubric prioritized pre-post measurements and a comparison group because they reduce the influence of confounders and account for external factors that may affect the outcome over time ¹⁴. Probability sampling procedures can reduce self-selection and confounding and thus are discussed in the review although not incorporated into the rubric.

We qualitatively synthesized the included articles for relevant substantive and methodological findings and summarized the results by study outcome.

Results

Thirty-seven articles met the pre-specified inclusion criteria and were included in the review. Tables 1–3 present details on the designs and study populations of included studies, outcomes of interest, type of natural experiment evaluated, and whether results were in an expected direction (an improvement in the outcome of interest associated with the intervention), unexpected direction (intervention associated with worse outcomes), were null, or had mixed results (expected direction for some subgroups/analyses, unexpected or null for others). A majority of studies (29, 78%) were conducted within the United States; the rest were in Australia, the United Kingdom, Canada, Chile, and New Zealand. Almost half of studies focused on adults (16 studies, 43%), eight focused on children/adolescents (22%), and 10 included a combination of age groups (27%). The success of the intervention did not differ by country or by age group.

Table 1.

Substantive Findings and Study Design for Natural and Quasi-Experiments Evaluating Changes to Obesity, Weight, or Body Mass Index

Reference	Type of Intervention and Setting	Strength of Study Design	Study Design	Study Population and Size	Outcome, Timing and Method of Outcome Assessment	Direction (expected, unexpected, null)
Corvalan, 200845	Policy: School nutrition. Day care meal standards. (National, Chile)	+++	Within person longitudinal with comparison group1	67,841 children aged 2–5 at 538 nursery schools	BMI z-scores, Obesity; 24 months post-intervention; Direct measurement	Mixed: Expected for phase 1 (reduction in BMI z-score and prevalence of obesity), null for phases 2 and 3 (no change)
Kaushal, 200744	Policy: Food voucher eligible participants. (National, U.S.)	++	Repeat cross-sectional with comparison group1	72,173 low income immigrant adults	BMI; 24 months post-intervention; Self-report (survey)	Null (no association between food stamps and BMI)
MacDonald, 201037	Built Environment: Active Transport. Light rail (City in U.S.)	+++	Within person longitudinal with comparison group	498 adults	BMI, Obesity; 6 months post-intervention; Self-report (survey)	Expected (reduction in BMI and odds of obesity)

¹Indicates assignment to intervention and control groups was random or as-if random.

Table 3.

Substantive Findings and Study Design for Natural and Quasi-Experiments Evaluating Changes to Diet/Nutrition

Reference	Type of Intervention and Setting	Strength of Study Design	Study Design	Study Population and Size	Outcome, Timing and Method of Outcome Assessment	Direction (expected, unexpected, null)
Angell, 201259	Policy: Trans fat ban in restaurants (City in U.S.)	+	Repeat cross-sectional, case only1	14,855 purchases made by adult restaurant customers	Change in mean grams of trans fat per purchase; 20 months post-intervention: Direct measurement (receipts)	Expected (reduction in mean trans fat per purchase following ban)
Auchincloss, 201330	Policy: Nutrition labeling in restaurants (City in U.S.)	+	One-time cross-sectional with comparison group	648 adult restaurant customers	Calories, saturated fat, carbohydrates, sodium per restaurant purchase; 19 months post-intervention; Direct measurement (receipts)	Expected (fewer calories, less saturated fat, and sodium purchased at restaurants with labeling)
Buttenheim, 201265	Food Environment: Food voucher payment systems at farmers markets. (City in U.S.)	+	Repeat cross-sectional, case only	48 months (time series) of customer sales from multiple vendors at a single farmer’s market	Farmers Market sales; 9 months post-intervention; Direct measurement (transactions)	Expected (increase in farmers market purchases)
Cradock, 201163	Policy: School Nutrition. Sugar-sweetened beverages not allowed in vending or à la carte (City in U.S.)	++	Repeat cross-sectional with comparison group1	2,033 public high school students	Daily consumption of sugar-sweetened beverages; 20 months post-intervention; Self-report (survey)	Expected (declines in consumption of sugar-sweetened beverages after policy compared to national trends)
Cullen, 200664	Policy: School Nutrition. Changes to snack bars and vending. (City in U.S.)	+	Repeat cross-sectional, case only	2,790 middle school students at 3 middle schools.	Daily mean calories and other nutrients, servings of healthy and unhealthy foods and beverages; Outcome measurement overlapped with intervention; Direct measurement (transactions)	Mixed (Consumption of sugar-sweetened beverages declined but consumption of chips/candy from vending machines increased)
Cummins, 200566	Food Environment: Supermarket in food desert. (City in Scotland)	+++	Within person longitudinal with comparison group.	412 households-men and women aged ≥16 living near and far from supermarket	Daily fruit and vegetable portions consumed; 10 months post-intervention; Self-report (survey)	Null (no change in fruit/vegetable consumption associated with supermarket)
Dumanovsky, 201133	Policy: Nutrition labeling in restaurants (City in U.S.)	+	Repeat cross-sectional, case only	15,798 adult restaurant customers	Calories in restaurant purchases; 9 months post-intervention; Direct measurement (receipts)	Mixed-Null for full sample (no change), expected for some chains (reduction in calories purchased post policy)
Elbel, 200934	Policy: Nutrition labeling in restaurants (City in U.S.)	++	Repeat cross-sectional with comparison group1	1,156 adult fast food restaurant customers in low-income, minority communities	Calories, saturated fat, sodium, sugar content of restaurant purchases; 1 month post-intervention; Direct measurement (receipts)	Null (no change in calories purchased after policy)
Elbel, 201155	Policy: Nutrition labeling in restaurants (City in U.S.)	++	Repeat cross-sectional with comparison group1	349 children/adolescent fast food customers	Calories, saturated fat, sodium, sugar content of restaurant purchases; 1 month post-intervention; Direct measurement (receipts)	Null (no change in calories purchased after policy)
Finkelstein, 201156	Policy: Nutrition labeling in restaurants (City in U.S.)	++	Repeat cross-sectional with comparison group1	Transactions from 14 chain restaurants	Average calories per restaurant transaction; 1 month post-intervention; Direct measurement (receipts)	Null (no difference in trends in calories per transaction between intervention and control)
Mendoza, 201060	Policy: School Nutrition. Changes to all school food environments, including snack bars and vending. (State in U.S.)	+	Repeat cross-sectional, case only	12,788 self-reported food records from middle school students at 3 schools	Energy density of student lunch (kcal/g); 12 months post-intervention; Self-report (survey)	Expected (reduction in energy density following policy)
Mullally, 201061	Policy: School Nutrition. Sugar-sweetened beverage vending ban, broad set of recommendations to offer healthier meals. (Province in Canada)	+	Repeat cross-sectional, case only	1,533 5th-6th grade children at 11 elementary schools	Proportion of students meeting recommendations for fruit, vegetable, and milk consumption, consuming <3 servings of low nutrient dense foods; 12 months post-intervention; Self-report (survey)	Expected (reduction in low nutrient dense foods, increase in meeting recommendations for fruit, vegetables, and milk after policy)
Odoms-Young, 201338	Policy: Food voucher eligible foods. (National, U.S.)	++	Within-person longitudinal, case only 1	273 Hispanic and African American children aged 2–3 years old enrolled in WIC and their mothers	Mean daily servings of healthy foods, home availability of healthy foods; 6 months post-intervention; Self-report (survey)	Expected (increase in fruit and low-fat dairy consumption, increase in home healthy food availability)
Sadler, 201341	Food Environment: Supermarket. (City in U.S.)	++	Repeat cross-sectional with comparison group	352 adults who were the primary shopper for their household	Mean daily servings of fruits and vegetables; 10 months post-intervention; Self-report (survey)	Null (no impact of intervention on fruit/vegetable consumption)
Snelling, 201262	Policy: School Nutrition. Standards for school lunches, reduce unhealthy food in vending machines. (National, U.S.)	+	Repeat cross-sectional, case only	Purchases from 17,819 students at 3 high schools	Change in calories and fat purchased by students; 12 months post-intervention; Direct measurement (transactions)	Expected (reduction in calories and fat purchased after policy)
Sutherland, 201057	Food Environment: Nutrition labeling in supermarkets (Region in U.S.)	+	Repeat cross-sectional, case only	Purchases from 168 outlets of a Northeast supermarket chain	Percentage of purchases that were classified as healthy; 12 months post-intervention; Direct measurement (transactions)	Expected (increase in purchasing of foods and beverages of high nutritional quality after intervention)
Tandon, 201142	Policy: Nutrition labeling in restaurants (City in U.S.)	+++	Within person longitudinal with comparison group1	133 6–11 year old children and their parents	Average calories purchased; 2 months post-intervention; Direct measurement (receipts)	Null (no change in average calories purchased from before to after policy)
Vadiveloo, 201158	Policy: Nutrition labeling in restaurants (City in U.S.)	++	Repeat cross-sectional with comparison group1	1,170 adult restaurant customers at 4 chain restaurants	Types of foods purchased and frequency of fast food consumption; 1 month post-intervention; Direct measurement (receipts)	Mixed-Overall null (no change in types of food purchased) but those who reported noticing/using labels consumed fast food less frequently

Obesity, weight or BMI measurements

Only three studies assessed impacts on BMI or weight (Table 1): one focused on economic policies, specifically whether receipt of food stamp benefits ⁴⁴ was associated with higher BMI or weight, and two focused on food/built environment changes ^{37, 45}. Overall these studies found little effect on BMI, with the exception of the study from MacDonald et al (a longitudinal design with comparison group), that found use of a new light rail system in Charlotte, NC was associated with an average reduction of 1.18 (95% confidence interval:−2.22, −0.13) in self-reported BMI and reduced odds of becoming obese over time ³⁷. Only one study ⁴⁵ directly measured height and weight to calculate BMI; the other two studies relied on self-report.

Physical activity

Seventeen studies assessed impacts on physical activity (Table 2).

Table 2.

Substantive Findings and Study Design for Natural and Quasi-Experiments Evaluating Changes to Physical Activity

Reference	Type of Intervention and Setting	Strength of Study Design	Study Design	Study Population and Size	Outcome, Timing and Method of Outcome Assessment	Direction (expected, unexpected, null)
Berniell, 201331	Policy: School Physical Activity. Health education (State in U.S.)	++	Repeat cross-sectional with comparison group1	11,026 parents of children aged ≤12	Probability of engaging in light physical activity ≥ once per week; Outcome measurement overlapped with intervention; Self-report (survey)	Mixed: Expected for fathers (increase in probability of engaging in physical activity), null for mothers
Bohn Goldbaum, 201346	Built Environment: Park playground and other park improvements (City in Australia)	++	Repeat cross-sectional with comparison group	149 children aged 2–12 and their parents at 2 parks	Mean # of children engaging in moderate-to-vigorous physical activity; 9 months post-intervention; Direct observation	Null (no difference in physical activity between intervention and control park)
Brink, 201052	Built Environment: Playgrounds in schoolyards (City in U.S.)	+	One-time cross-sectional with comparison group	1,185 children at 9 elementary schools	Mean energy expenditure rate; <1 year post-intervention at recently built schoolyard; ≥2 years post-intervention at established school yards; Direct observation	Expected (higher mean energy expenditure rate in children at intervention than control playgrounds)
Brown, 200732	Built Environment: Active Transport. Light rail stop added (City in U.S.)	++	Within-person longitudinal, case only	51 adults (47 with accelerometry data)	Bout of 8+ minutes of moderate activity; ~9 months post-intervention; Direct measurement (accelerometer)	Expected (self-reported rides on light rail associated with increased moderate activity)
Cohen, 201247	Built Environment: Park outdoor exercise equipment (City in U.S.)	++	Repeat cross-sectional with comparison group	Adults and children at 22 study parks. Surveys: 2,636 adults; Observation: 9,476 users	Estimated energy expenditure by park;<12 months; Direct observation and self-report (survey)	Expected (increase in estimated energy expenditure in intervention parks)
Evenson, 200549	Built Environment: Path/trails (City in U.S.)	++	Within-person longitudinal, case only	366 adults living within 2 miles of trail	Time spent walking, bicycling, moderate, vigorous, and transportation activity; 2 months post-intervention; Self-report (survey)	Null (no change in time spent engaging in physical activity)
Fitzhugh, 201050	Built Environment: Path/trails (City in U.S.)	++	Repeat cross-sectional with comparison group	Children, adolescents and adults in 3 neighborhoods. Exact counts not provided	2-hour counts of physical activity in neighborhood;13–14 months post-intervention; Direct observation	Expected (2-hour counts of physical activity increased in intervention relative to control neighborhood)
Fuller, 201354	Built Environment: Active Transport. Bicycle share (City in Canada).	++	Repeat cross-sectional with comparison group	7,012 adults	Cycling for ≥10 minutes in past week; 5 months post-intervention; Self-report (survey)	Expected (greater odds of cycling among those exposed to the bike share program)
Fuller, 201253	Built Environment: Active Transport. Effect of transit strikes on use of Bicycle share (City in England).	+	Repeat cross-sectional, case only1	95 days-time series. Adults	Mean bicycle trips per day and mean trip duration; 55 days (Strike 1) and 25 days (Strike 2) post-intervention; Direct observation	Expected (increase in daily bicycle trips following transit strikes)
Gustat, 201235	Built Environment: Path/trail and playground (City in U.S.)	++	Repeat cross-sectional with comparison group	1,191 English-speaking adults aged 18–70 who had lived in the neighborhood for ≥3 months	Number of people engaging in moderate and vigorous physical activity; ~12 months post-intervention; Direct observation and survey	Expected (increase in observed physical activity in intervention compared to control neighborhoods)
Krizek, 200936	Built Environment: Active Transport. Bike lanes and off-street bike paths (City in U.S.)	++	Repeat cross-sectional with comparison group	Adults in the Minneapolis/St. Paul area. Exact count not provided	Proportion of bicycle commuters among all commuters; Varying, 1–<10 years post-intervention; Self-report (survey-US census)	Expected (increase in levels of bicycle commuting in areas closer to new bike facilities)
MacDonald, 201037	Built Environment: Active Transport. Light rail (City in U.S.).	+++	Within person longitudinal with comparison group	498 adults	Meeting weekly recommended physical activity levels; 6 months post-intervention; Self-report (survey)	Null (light rail use not associated with increased odds of meeting recommended physical activity levels)
Parker, 201139	Built Environment: Active Transport. Bike lanes (City in U.S.)	+	Repeat cross-sectional, case only	Adult and child cyclists. Exact count not provided.	Average number of cyclists observed daily; ~6 months post-intervention; Direct observation	Expected (increase in average number of riders per day after bike lanes painted)
Parker, 201340	Built Environment: Active Transport. Bike lanes (City in U.S.)	++	Repeat cross-sectional with comparison group	Adult and child cyclists. Exact count not provided.	Average number of cyclists observed daily; 3 months post-intervention; Direct observation	Expected (larger increase in number of cyclists in the street with a new bike lane)
Quigg, 201248	Built Environment: Park playground (City in New Zealand)	+++	Within person longitudinal with comparison group	154 children aged 5–10 years old	Mean total daily physical activity; 3 months post-intervention; Direct measurement (accelerometer)	Mixed-Expected for children with lower BMI (increase in PA after playground built), reverse for children with high BMI (decrease in PA)
Veitch, 201251	Built Environment: Park playground and other park improvements (City in Australia)	++	Repeat cross-sectional with comparison group	2,050 adults and children at 2 parks.	Number of people walking or being vigorously active; 3 months post-intervention; Direct observation	Expected (increase in number of people walking and being vigorously active in intervention park)
West, 201143	Built Environment: Path/trails (City in U.S.)	+++	Within person longitudinal with comparison group. 1	166 adult residents of households near the greenway	Days in past week spent walking, engaging in moderate or vigorous physical activity; 11 months post-intervention; Self-report (survey)	Expected (increase in walking and moderate PA among those living closest to greenway)

¹Indicates assignment to intervention and control groups was random or as-if random. All of these had a separate comparison group except for Fuller et al 2012⁵³ which was an interrupted time-series (case-crossover) for all trips made using bicycle share bikes.

Most (n=9) assessed physical activity impacts due to greenspace and outdoor play/exercise equipment. They assessed activity effects due to changes in park playgrounds and other outdoor exercise equipment ^46–48, paths/trails ^{43, 49, 50}, a combination of the two ^{35, 51}, or elementary school yard improvements ⁵². The studies typically used repeat cross-sectional with comparison group design. Impacts were usually assessed within one year after implementation (range 2–14 months) and over half (n=6) found increases in physical activity ^{35, 43, 47, 50–52}. For example, average energy expenditure rate among students was significantly higher (0.36 vs. 0.27, p<0.001) at schools with renovated schoolyards compared to controls ⁵². In general, studies with positive results had longer follow-up times (greater than 6 months). Studies with null or mixed results were mostly smaller samples ^{46, 48} or had very short follow-up periods ⁴⁹.

New amenities may promote residents to substitute one type of activity for another but not impact overall total physical activity levels. Most of the studies that assessed change in population response to park amenities and new paths/trails collected data via a combination of surveys and systematic observations (only one out of nine studies assessed physical activity using accelerometry ⁴⁸⁾. About one-half assessed impacts on total physical activity with mixed results (2 out of 5 reported expected results ^{35, 43}) while the remainder assessed process outcomes and all of them found expected results (4 out of 4 assessed volume of activity in a particular location, use of a park, or change in type of activity while at a park ^{47, 50–52}).

Seven studies assessed physical activity impacts due to active transportation interventions (interventions that promote the use of active means of travel, such as walking and biking) and largely found positive results. However, only two of these studies assessed change in total physical activity and only one found expected results ³². For example, one light rail transit study found no effect on total physical activity (despite finding an association with self-reported BMI ³⁷), while another found increases in accelerometer-measured moderate physical activity bouts ³². The remaining five active transportation studies did not assess impacts on overall physical activity but assessed process outcomes and all found expected results (use of active transportation ^{36, 39, 40, 53, 54}). For example, two large studies found increased cycling after implementation of the London and Montreal bicycle share programs ^{53, 54}, and several studies of improvements to urban bike infrastructure found significant increases in cycling ^{36, 39, 40}.

Finally, one study assessed whether state health education policies changed parents’ activity levels and results were mixed, with self-reported physical activity increases observed for fathers but not mothers ³¹.

Nutrition/diet

Eighteen studies assessed impacts on diet (Table 3).

Most of the studies (n=8) focused on responses to nutrition labeling and used food purchase receipts to assess the potential impact on calories and dietary quality ^{30, 33, 34, 42, 55–58}. Six of these studies assessed impacts soon after implementation (range 1–9 months, average 3 months) and overall had no impact on food purchasing or on improving nutritional outcomes. Two studies assessed impacts at least one year post-implementation and found expected results ^{30, 57}. For example, a study of a sit-down chain restaurant (using a one-time cross-sectional with comparison group design) found that customers at restaurants with menu labeling purchased food with 151 fewer kilocalories (95% CI: 33, 270) compared to customers at restaurants without labeling, as well as decreased saturated fat and sodium ³⁰.

Six large sample studies used sales and survey data to evaluate the impact of regulatory improvements to restaurant food environments (trans-fat ban) ⁵⁹ or school food environments (restrictions on sugary foods and beverages or higher fat foods, and/or increases in availability of milk and fruits/vegetables) ^60–64. These studies assessed impacts 12–20 months post-implementation (most were repeat cross-sectional, case only) and reported favorable impacts on purchases or self-reported food consumption. For example, after a school nutrition policy change, elementary students had increased odds of meeting recommendations for vegetables and fruit (OR: 1.44, 95% CI: 1.00–2.07) ⁶¹.

Two studies assessed impact on nutrition after federal policy changes in the quality of foods that could be purchased with low-income food vouchers (WIC changes to include more fruits, vegetables, grains, and lower fat milk ³⁸) or local changes to vendor payment systems so that food vouchers would be accepted at farmers markets ⁶⁵. They assessed impacts 0–9 months post-implementation and found healthier foods within the home and modest within-person improvements in diet ³⁸ and improvements in purchases of fruits and vegetables and use of farmers markets ⁶⁵.

Two studies evaluated impacts on self-reported fruit and vegetable consumption after opening a large supermarket in a food desert ^{41, 66}. The studies were fairly small (100–200 households at the intervention site), assessed impacts 10 months post-implementation and found no significant impact.

Study design

The studies identified in the literature search employed several types of study designs, including longitudinal, cross-sectional, and time-series, with and without comparison groups. Six studies employed the strongest design (+++), 19 studies employed intermediate designs (++), and 12 studies used weaker designs (+). Among those with the strongest design rating, two studies (33%) had results in the expected direction, compared to 10 (53%) of the intermediate strength studies, and 10 (83%) of the studies with weaker designs.

Probability of selection

A majority of studies did not employ probability-based sampling e.g., random selection of sites in the intervention area or individual respondents within these sites. Fifteen out of 37 studies (41%) included probability sampling,^{31, 33, 35–37, 41, 43, 44, 48, 49, 53, 56, 59, 63, 66} typically by randomly sampling stores or participants from an existing sample frame. For example, Angell et al randomly sampled 300 locations from all 1,625 licensed restaurant locations of 13 restaurant chains in New York City ⁵⁹.

Modeling and adjustment for unmatched treatment and comparison groups

An important assumption of natural or quasi-experiment evaluations is that there are no systematic differences between the treatment and control groups. Adjustment for potential confounders or selection effects is an approach to minimize such differences. In this review, two studies used propensity score weighting to improve comparability between the treatment and control group and found improvements in BMI ³⁷ and physical activity ⁴⁷. Fifteen studies implemented regression models that adjusted for a variety of covariates (5, or 30%, found significant effects) ^{30–34, 42, 44, 45, 48, 49, 53, 58, 59, 63, 66}. Six studies adjusted models for only one or two covariates, of which 4 (67%) found results in the expected direction ^{35, 46, 56, 60, 61, 65}. Fourteen studies did not used adjusted regression models, of which 11 (73%) found results in the expected direction ^{36, 38–41, 43, 50–52, 54, 55, 57, 62, 64}; many of these studies involved comparing counts of trail or playground activity.

Discussion

Summary of substantive findings

In this systematic review of natural and quasi-experiments in obesity research, we found a growing body of literature evaluating the effect of policies or changes to the built environment on obesity and, in particular, obesity-related behaviors, nutrition or physical activity. Natural experiments may play an important role in identifying effective interventions ²². Our results suggest that certain types of interventions have more success than others in improving outcomes of interest. For example, a majority of studies evaluating regulations that required improvements to the food environment, either through local ⁵⁹ or school policies ^60–64 found improvements in purchasing or self-reported diet, while studies that simply required the posting of nutritional information^{33, 34, 42, 55, 56, 58} found little effect, with a few exceptions ^{30, 57}. In addition, studies that evaluated the effect of a new supermarket in a previously underserved area found no effect ^{41, 66} while studies that improved the ability of low income people to use benefits to purchase fruits and vegetables found improvements to purchasing or home availability of healthy food ^{38, 65}. Physical activity-related studies generally found stronger impacts when the intervention improved infrastructure for active transportation ^{32, 36, 39, 40, 53, 54} or had a longer follow-up period. Nevertheless, many of the studies reporting favorable impacts only assessed process outcomes (e.g., food purchases, use of bike/transit infrastructure) rather than improvements in overall diet or physical activity. In this review, only a few studies assessed impacts on BMI/weight; thus, evidence is lacking on whether environmental and policy modifications are successful in maintaining healthy weight and/or reducing overweight. In order to assess impacts on BMI, evaluators will likely need to study multiple co-occurring natural experiments that encompass a range of nutrition and physical activity environments and will need to observe changes over at least a 12 month period ^67–69.

Comparison of results from traditional observational designs

Relative to the large volume of work that has been published using traditional observational studies, few natural experiments have been published that assess impacts on obesity-related behaviors due environmental and policy modifications. Studies evaluating natural experiments are limited to studying sharp difference of conditions; thus substantive overlap with traditional observational studies is limited. Nevertheless, evidence from traditional observational studies suggests that living near playgrounds, parks, and other recreational facilities may be associated with higher levels of physical activity in adults and children, although there were exceptions^{4, 6, 7}. Natural experiments reviewed here were similarly equivocal: studies of trails/paths mostly reported expected impacts ^{35, 43, 50} with one exception ⁴⁹. Impacts from new or renovated playgrounds were mixed: about one-half reported expected associations. Traditional observational studies have found that public transportation can be a significant contributor to physical activity ⁷⁰. Natural experiments confirmed that people use non-automobile transportation when it is made accessible ^{36, 39, 40, 53, 54} but only one study explicitly linked new transit options to increases in physical activity levels ³²; thus evidence is largely lacking on whether new active transportation amenities results in substitution of one specific type of activity for another (e.g., cycling for walking) or whether it has significant impact on total physical activity levels.

Traditional observational studies suggest a positive association between quality of the food environment (in particular neighborhood availability of supermarkets), better quality diet⁸ and lower prevalence of obesity⁹. At the time of this review, only two studies assessed whether diet improves after introducing a new supermarket into a food desert and found no impact on fruit and vegetable intake ^{41, 66}.

Ability to study changes to policy

Traditional observational studies are largely unable to assess obesity-related impacts from new policy implementation. Thus, a key advantage to natural experiments is the ability to focus on policy-relevant changes and real-world efficacy. Evidence generated by natural experiments is of particular use to policymakers who need unbiased evaluations of the effect and implementation of policies ¹¹. This review found that policy bans on certain types of food at schools and restaurants mostly reported expected impacts while nutrition labeling policies, which simply warned customers of unhealthy purchases, mostly reported weak impacts. Such information may provide insight to policymakers on types of policies to focus on for future obesity prevention efforts. More natural experiments are needed to strengthen the evidence base and also explore whether timing of exposure and/or longer/repeat exposures enhances or reduces impacts on obesity-related outcomes.

Impact of intervention in light of study design strengths/weaknesses

A majority of studies identified in this review were repeat cross-sectional with a comparison group (rated intermediate design strength, ++) although several used weaker designs: one-time cross-sectional studies with a comparison group or repeat cross-sectional case-only studies. Longitudinal studies offer an advantage in that they can assess time-varying change in both the exposure and outcome; however, even well-done longitudinal observational designs may bias true effects of the environment to the null because environmental conditions may be static or slow to change and may not result in between-group differences in the rate of change in the outcome during the observed follow-up time. Studies with weaker cross-sectional designs were more likely to have results in the expected direction; however, drawbacks in study design limit their contribution to the evidence base. For example, using an intervention and comparison group posttest-only design, it is not possible to eliminate the possibility that the groups differed with respect to the dependent variable prior to the intervention. Similarly in a one-group pre-post design it is not possible to determine whether secular changes (something other than the intervention) occurred between the pretest and posttest to cause the outcome. Due to the exogenous nature of the treatment assignment of study units, natural experiments can improve the plausibility of counterfactual contrasts, reduce self-selection bias, and improve causal inference ¹³. However, as seen from this review, some natural experiments have weak designs that offer little improvement over traditional observational studies. While natural experiments by definition do not involve randomizing study units to a particular exposure, random selection of units from a sample frame helps ensure that the included sites and/or individuals are representative of the target population of interest; failure to randomly select the sample may result in a systematic bias of the results if likelihood of selection is associated with the exposure and outcome of interest²⁴. However, few studies employed probability sampling and a number of studies did not even adjust for confounders in regression models. Adjustment for confounders should be implemented whenever possible to improve inferences.

Lack of control leads to challenges in study design and data collection

Because natural experiments are not under the control of investigators, the timing of policies or built environment changes can impact the rigor of study design and data collection. There may not be sufficient time to collect pre-intervention data (necessitating a one-time cross-sectional design). Less than optimal conditions can translate into smaller samples and over-reliance on questionnaires and observational methods rather than direct measurement. More studies are needed to evaluate the effect of policies and built environment changes on individual outcomes such as weight change or total physical activity, beyond the process measures often reported such as use of a trail or active transportation infrastructure. Long delays in implementing the intervention or in how the intervention is implemented can severely impact post-test loss-to-follow up and study rigor. For example, in a study evaluating the effect of a new supermarket on access to healthy food ⁶⁶, local stakeholders living in the planned intervention area blocked construction of the market, and it was instead built in the area investigators had planned to use as the control site, effectively switching the intervention and control areas ²². Funding opportunities for natural experiments are very limited due to their time-sensitivity and grant applicants often receive poor rankings relative to other applicants who have collected pilot data and have multiple opportunities for review. Similarly, inadequate funding and time-sensitivity can limit the length of time between implementation of the intervention and collection of the outcomes data and number of measurements. Studies with less than 12 months of follow-up may not provide enough time for residents/customers to be affected by the intervention or for full implementation of the intervention to occur. In addition, studies with only two time points to assess change may not provide a valid measure of change. Innovative modeling approaches may be needed to overcome shortcomings in the design that result from these challenges.

Conclusion

In conclusion, current research suggests some policy and built environmental interventions, especially active transportation infrastructure improvements, bans or restriction on unhealthy foods, and altering purchase/payment rules for low-income food vouchers, can increase certain types of physical activity and improve diet. It is not clear, however, whether these changes result in reduced obesity, and more research is needed on the effect of built environment changes like park improvements, trails, and active transportation infrastructure on total physical activity, beyond the process outcomes commonly measured. Natural experiments provide certain advantages over traditional observational research, including the ability to focus on policy-relevant changes and real-world efficacy. However, challenges related to lack of control, timing, and funding often necessitate the use of weaker study designs, which limits the strength of evidence from such studies.