Evaluating Environmental Change Strategies: Challenges and Solutions

Abstract

In this introductory article we define environmental change strategies (ECS), summarize the primary challenges associated with evaluating ECS, and provide an overview of the methods researchers have employed to begin to address these challenges. This special issue provides a range of examples, from researchers and practitioners in the field, of different approaches for addressing these challenges. These articles present new methods to understand and test how ECS are implemented and propose methods to evaluate their implementation. The content of the articles covers multiple public health issues, including substance abuse prevention, tobacco control, HIV prevention, and obesity prevention. This special issue is intended to build the evidence base for effective ECS, generate compelling discussion, critical analyses, and spur future research that will help improve the implementation and evaluation of ECS.

Introduction

With their emphasis on community, state, or federal level interventions, environmental change strategies (ECS) are a natural topic for the American Journal of Community Psychology. Most of the articles in this special issue focus on population-based interventions implemented at the community or state level. Through the lens of community psychology, we explore and understand the environmental influences on decision-making. Much of the work in implementing and evaluating ECS is done by community groups and coalitions. And the effects of ECS can be seen in improved health throughout the community.

Although much research has been published documenting the impacts of ECS, we produced this special issue of the American Journal of Community Psychology as a way to synthesize a definition of ECS, document the challenges associated with evaluating ECS, and provide a compendium of articles that demonstrate strategies for addressing these evaluation challenges.

Defining Environmental Change Strategies

ECS are population-based interventions that change the environment or context in which individuals make decisions (Frieden 2010; Klitzner 1998; Mcleroy et al. 1988; Schmid et al. 1995; Stokols 1992; Treno and Lee 2002). These strategies typically address public health issues by employing policy, enforcement, and communication campaigns. Shortly after taking over as Director for the Centers for Disease Control and Prevention (CDC), Thomas Frieden published a framework for public health action that used a pyramid to describe the impacts of different types of public health interventions (Frieden 2010). Frieden (2010) identified ECS, defined as “interventions that change the environmental context to make healthy options the default choice” (p. 591), as the second most potent influence on population behavior change, exceeded only by fundamental changes in socio-economic conditions.

A compelling body of evidence is accumulating that demonstrates the effectiveness of ECS to improve the overall health of neighborhoods, communities, states and nations (Centers for Disease Control and Prevention 1999; Frieden 2010; Maddock 2012). Furthermore, ECS have been used to address a diverse range of public health issues, including alcohol abuse prevention (Grube 1997; Kenkel and Manning 1996; Stockwell and Gruenewald 2001; Wagenaar 1993), tobacco control (Chaloupka et al. 2012), and physical activity promotion (Schmid et al. 1995). The recent announcement by the mayor of New York City of a city policy limiting the size of sugar sweetened beverages is an example of the application of ECS as a strategy to address the growing obesity problem (Grynbaum 2012). The success of ECS to influence health at a population level has led to private and federal funders increasingly requiring grantees to implement ECS (e.g., SAMHSA’s Drug Free Communities and Strategic Prevention Framework State Incentive Grants, CDC’s Communities Putting Prevention to Work, Robert Wood Johnson Foundation’s Active Living Research Grants, Kaiser Permanente’s Community Health Initiative).

Challenges to Evaluating Environmental Change Strategies

As ECS become increasingly central to public health intervention implementation, it is imperative that we further develop methodologies to evaluate these interventions. There are several challenges inherent to evaluating ECS. Basic issues include defining the geographic locality of an intervention and developing a rigorous evaluation design (Coulton 2005; Holder et al. 1997b; Treno and Lee 2002). Conducting process and outcome evaluations introduce another level of challenges. Process evaluation challenges include assessing implementation fidelity (Treno and Lee 2002), measuring dosage (Holder et al. 1997a), and operationalizing, quantifying, and attributing effects of a strategy to changes in intermediary outcomes (DeGroff et al. 2010; Fagan et al. 2008; Ohmer 2008). Outcome evaluation challenges include the lack of available data to assess intervention effects (DeGroff et al. 2010; Ohmer 2008), accounting for contextual factors (Holder et al. 1997b), and determining the appropriate start times for the strategy and conducting time series analyses (Holder et al. 1997a, b; Ohmer 2008). Although many of these questions were raised more than 15 year ago, we are still seeking ways to address them.

In This Issue

This first section of this special issue describes some of the key challenges associated with evaluating ECS. The second and third sections provide examples of process and outcome evaluations of ECS.

The process evaluation articles (Nargiso et al. 2012; Miller et al. 2012; Gabriel et al. 2012) focus on measuring and documenting the progress that organizations make in their efforts to implement and evaluate ECS. Because implementing ECS can be a lengthy process, these process measures can help demonstrate progress before actual outcomes associated with the ECS might be realized. Assessing these process measures also provides organizations with an opportunity to make mid-course corrections to the ECS if they find they are not on target. Gabriel and colleagues (Gabriel et al. 2012) also provide an example of how organizations can collect and use baseline data to assess changes that occur after an ECS has been implemented.

The outcome evaluation articles highlight a variety of methods that can be used to assess the impact or outcomes of ECS at the local (Johns et al. 2012; Flewelling et al. 2012; Freisthler et al. 2012) and state (Phillips et al. 2012) levels. The final article discusses the results of a simulation model that can be applied at any level—local, state or national—to assess the impact of price changes on soda consumption (Levy and Friend 2012).

Process Evaluations of Environmental Change Strategies

Process evaluations provide a way to assess whether a strategy is being implemented as planned (i.e., with fidelity) and to determine what factors are influencing the relative success of the strategy. McLeroy and colleagues state that, “an essential component of ecological strategies is active involvement of the target population in problem definition, the selection of targets of change and appropriate interventions, implementation and evaluation”. “The process of ecological strategies is one of consensus building” (Mcleroy et al. 1988). This active involvement of members of the target population is usually provided through a coalition, and so process evaluations of ECS can benefit from understanding and evaluating how coalitions themselves function.

The first two articles in the special issue provide methods for conducting process evaluations of coalition involvement in implementing ECS (Nargiso et al. 2012; Miller et al. 2012). Nargiso et al. (2012) consider the types of capacities needed to implement ECS and introduce a rubric for quantifying broad coalition capacity as well as innovation-specific or environmental-specific capacities. The paper also examines the link between the types of capacity and intermediate outcomes of ECS.

Understanding what makes a coalition strong is important, but equally important is understanding the progress a coalition must make to achieve its goals. Measuring intermediate achievements thought to lead to long-term successes is critical to understanding how ECS are implemented, however, little research has been done to document these process-oriented milestones (Kreger et al. 2007). Miller et al. (2012) explore the strengths and weaknesses of various indicators of intermediate success in creating structural changes among coalitions organized to prevent HIV exposure among high-risk adolescents in their local community. This research provides examples of key indicators that can be tracked to demonstrate progress in implementing ECS.

The final process evaluation article by Gabriel et al. (2012) provides a description of how one project collected baseline data at the local level with limited resources and how that data can be used to assess the success of their ECS. As part of their implementation of ECS targeting alcohol abuse, they surveyed community members, including parents of middle and high school students, to assess attitudes and beliefs related to underage drinking. In addition to providing data for evaluation purposes, data from the survey also informed the selection of appropriate social marketing campaign messages and fostered support for increased enforcement of current laws.

Outcome Evaluations of Environmental Change Strategies

Outcome evaluations assess the extent to which a strategy achieves its objectives. Traditional methods of analysis do not adequately capture the dynamic sequelae of the implementation of ECS in the real world or the outcomes achieved through ECS. Evaluation methods for ECS should be able to address the challenges associated with documenting the relationship between changes in knowledge, attitudes, and behaviors and the implementation of the ECS as well as understanding the time-lag inherent in assessing outcomes associated with ECS. Mixed methods that balance qualitative and quantitative approaches can demonstrate effectiveness better than evaluations that rely on one method to measure the success of an ECS. McKinlay (1993) recommends the concept of appropriate methodology, which encourages us to understand what we are trying to achieve and what types of data can be used to assess whether we have accomplished our objectives. Similarly, Brownson et al. (2009) highlight the importance of considering that “evidence comes in numerous forms” (p. 1580).

The second set of articles in this special issue provides examples of how to use mixed methods to assess the outcomes of an ECS. Three articles (Johns et al. 2012; Flewelling et al. 2012; Freisthler et al. 2012) provide examples of outcome evaluations conducted on local level ECS. Johns et al. (2012) demonstrate how to use multiple measures, timepoints, and methods to conduct a baseline assessment of the impact of New York City’s Smoke Free Parks and Beaches Law. The law went into effect in May 2011 and the New York City Department of Health and Mental Hygiene is currently conducting a long-term evaluation of the environmental change strategy. The authors discuss the practical and methodological challenges they faced and describe how their initial findings will help inform future evaluations of the law.

Flewelling et al. (2012) examine the results of a randomized trial to test the individual and combined effectiveness of five related local ECS designed to reduce underage access to alcohol. In the third local level ECS evaluation article, Freisthler et al. (2012) analyze the impact of local policies and regulations related to medical marijuana dispensaries in California. Their research demonstrates how to identify elements of a policy or regulation and related outcome measures in order to determine which policy components effectively change an environmental condition, in this case crime rates. The authors use spatial analyses techniques to analyze the relationship between the various components of the law and crime rates around the dispensaries.

Phillips et al. (2012) evaluate a state level ECS, with their assessment of the Arkansas Act 1220 of 2003 to Reduce Childhood Obesity. The evaluation was designed to understand the variation in implementation of the ECS across the state. They use a mix of qualitative and quantitative approaches to identify the barriers and facilitators to implementation and to demonstrate the impact the law has had on school environments and on student and family behaviors. The findings from this research also inform the discussion of when we can expect to see changes in behavior once an environmental change strategy is implemented.

Another type of analytic tool that allows us to test the effects of different ESC on population health is simulation modeling. Obesity among youth has become a critical public health issue in this country, calling for novel approaches. In the second simulation article, Levy and Friend (2012) developed a simulation model to examine how changes in policies, including school-based access, education, and price, targeting sugar-sweetened beverage (SSB) consumption among youth impact SSB consumption and overall caloric intake. The research provides a concrete example of how simulation modeling can be used to synthesize a body of research and make predictions about how individual policies, interactions of policies, and unfolding effects of policies over time, may affect behavior.

Discussion

In order to demonstrate the effectiveness of ECS, we need to use a variety of tested evaluation methods that result in robust data which can in turn be used to tell stories about the success of these strategies. The empiricism regarding the evaluation of ECS is growing quickly as the field moves to address the many challenges associated with evaluating ECS. This special issue was developed as a resource for those evaluating ECS in an effort to expand the methodological examples and resources available to evaluate ECS and build the evidence of their effectiveness. We intend for this issue to improve our understanding of the challenges associated with evaluating ECS, and to provide concrete approaches to addressing these challenges across a range of evaluation designs and analyses. We look forward to continued interest in this area through critical analyses and future research that will help improve the implementation and evaluation of ECS.