Lessons from a Mixed-Methods Approach to Evaluating Active Living by Design

Abstract

Background

Beginning in 2003, Active Living by Design (ALbD) established innovative approaches across 25 communities to increase physical activity through community design, public policies, programming, and communication strategies.

Purpose

The complexity of the ALbD projects called for a mixed-methods evaluation to understand implementation as well as perceived and actual impacts of these efforts.

Methods

Six primary evaluation methods addressed three primary aims: (1) to assess impacts of physical projects and policy changes on community environments; (2) to document intervention strategies implemented, as well as intended and unintended consequences; and (3) to identify strengths and challenges in planning, developing, and implementing interventions. The ALbD evaluation included cross-site comparisons and more in-depth case studies. This article describes the methods used to address the three aims.

Results

Analysis of the strengths and challenges associated with the different methods, including partnership capacity surveys, Concept Mapping, an online Progress Reporting System (PRS), key informant interviews, focus groups, and photos and videos. Additional methods, including environmental audits and direct observation, were explored to specifically assess environmental changes. Several important challenges included the lack of baseline data, difficulty in evaluating natural experiments, the need for ongoing policy surveillance, and the need to capture longer-term endpoints.

Conclusions

The mixed-methods evaluation of ALbD advances implementation and evaluation science related to community-based efforts for promoting active living through identification of methods and measures to capture multicomponent and complex interventions as well as translation of a range of approaches to create community change across a variety of populations and settings.

Introduction

Research over the past decade indicates that attributes of neighborhood environments (e.g., access to recreational facilities, mixed-use development) are associated with recreational and transportation physical activity.^1–5 Despite these associations, it is unclear which community intervention approaches, individually or collectively, are most effective in increasing physical activity behaviors. Likewise, the planning and implementation strategies to create short-term changes in the community, such as improving equitable access to facilities and services, increasing available support, and sharing resources, tend to be overlooked or under-reported.^6–8

Background on Active Living by Design

To fill this void, the Active Living by Design (ALbD) National Program Office (NPO) provided guidance to establish innovative approaches to increase physical activity through community design, public policies, programming, and communication strategies. Beginning in November 2003, ALbD supported 25 community partnerships across the U.S. to demonstrate how changing community design could affect physical activity.⁹ ALbD used a “high touch, low dollar” approach, in which each community partnership received an average of $40,000 per year and customized technical assistance from a Project Officer over a 5-year period. These awards were much lower than the NIH-supported community-based prevention projects of the 1980’s, which had annual budgets of $1 million to $1.5 million for 10 years or more.¹⁰

These NIH projects demonstrated that community conditions could be changed to promote physical activity; however, the cost was prohibitive for widespread adoption in many organizations and settings. More recently, the CDC’s “STEPS to a Healthier US” grants have been funded at much higher levels than that for ALbD.¹¹ Therefore, ALbD often used an approach that assessed whether core staff support would be sufficient to advocate for community changes and leverage other resources. An important feature of ALbD was that it sought to combine evidence-based practice (implementing interventions shown to be effective and consistent with community preferences)^{12, 13} with practice-based evidence (developed in the real world rather than in highly controlled research conditions).¹⁴

Background on the Evaluation Approach

The complexity of the ALbD projects called for a mixed-methods evaluation, which is also called a “triangulated” set of methods. Such mixed-methods approaches often result in greater validity of inferences, more-comprehensive findings, and more-insightful understanding.¹⁵ Triangulation generally involves the use of multiple methods of data collection and analysis as well as theory and practical knowledge to determine points of commonality or disagreement.^16,17 Triangulation is often beneficial because of the complementary nature of the data.

Although quantitative data provide an excellent opportunity to determine how variables are related to other variables for large numbers of people, it typically provides little in the way of understanding how interventions are adapted and why these relationships exist (so-called contextual evidence^12,18). Qualitative data can help provide information to explain quantitative findings, or what has been called “illuminating meaning.”¹⁷ The triangulation of qualitative and quantitative data can provide powerful evidence of effectiveness and can also offer insight into the processes of change in organizations and populations.¹⁹

The ALbD evaluation had three primary aims: (1) to assess impacts of physical projects and policy changes on community environments; (2) to document intervention strategies implemented, as well as intended and unintended consequences; and (3) to identify strengths and challenges in planning, developing, and implementing interventions. This article describes the methods used to address the three aims.

Previous Evaluation Efforts

For reasons beyond the control of the Robert Wood Johnson Foundation (RWJF) or the ALbD NPO, a plan to initiate evaluation from the beginning of the program was discontinued in October 2005, and a new plan for evaluation was instituted in November 2006 (i.e., the start of the fourth year of the program). This evaluation plan consisted of a three-part program of evaluative inquiry: (1) a cross-site evaluation tracking each community’s short- and intermediate-term achievements; (2) a substudy of environmental changes as new physical projects in six communities were implemented from the fourth year onward; and (3) case studies of behavior change in two communities with successful policy and environment changes, Somerville MA²⁰ and Columbia MO.^21,22 Thus, consistent with the principle of triangulation, these components provided a body of evidence from which to infer answers to the three aims.

The evaluation began in the fourth year of the community partnership intervention activities and continued for 3 more years, ending approximately 1 year after the intervention activities of the ALbD 5-year program. Most communities (23 of 25) received 12–18 month sustainability grants that allowed them to continue their efforts throughout most of the evaluation time period. This extended funding period for sustainability proved beneficial to the evaluation with respect to maintaining engagement and interest of the grantees.

Active Living by Design Community Action Model

As noted in previous literature,²³ the cross-site evaluation activities were guided by the ALbD Community Action 5P Model components: preparation, promotions, programs, policy influences, and physical projects. The 5P Model specified short-term changes (e.g., increased partnership capacity and policy changes) that were presumed to lead to intermediate changes (e.g., alterations to the physical environment and increased use of facilities, streets, and sidewalks for walking and bicycling). These intermediate changes, in turn, were presumed to lead to healthy lifestyle changes such as increased physical activity.²³ However, the evaluation focused only on the short- and intermediate-term endpoints, because the aim was to examine whether supporting community partnerships would be sufficient to change the built environment to be more conducive to physical activity.

A Community-Based Participatory Evaluation

The cross-site evaluation and assessment of new physical projects were guided by the well-established principles of community-based participatory research, which include highlighting strengths at the community level, involving local partners in the evaluation, and recognizing multilevel, ecologic approaches to health promotion.²⁴ In doing so, the evaluation team worked with grantees, the RWJF, ALbD staff, and the community partnerships to implement evaluation activities. For example, community-generated reports were used to develop interview and focus group questions, asking communities to identify interview and focus group participants, collecting information about the community context, and obtaining community validation of qualitative data analysis findings.

The evaluation team also collaborated with community partnerships to develop dissemination materials. Even though translation and dissemination were not central aims of this evaluation, the evaluation team worked to ensure that the findings had face validity from the communities’ perspective and were communicated effectively for a range of audiences and venues, including policymakers, planners, conferences, and publications. For example, community stakeholders provided input on how best to package materials for local policymakers. In addition, the evaluation team supported the community partnerships in developing peer-reviewed publications for a best-practices supplement,²⁵ and the community partners edited and approved case reports developed by the evaluation team.

The evaluation team worked with RWJF and the ALbD National Program Office to minimize the potential burden experienced by grantees participating in multiple evaluation-related activities (e.g., the three-part evaluation, the ALbD NPO PRS²⁶) occurring over the same time period. For instance, the scheduling of interviews and site visits was coordinated with the ALbD NPO so that these activities did not occur in the same month, or preferably quarter, as other evaluation-related activities.

Evaluation Substudy

Because the evaluation began in the fourth year of the program, it lacked baseline data. Thus, assessment of change was not feasible across all 25 communities, but where physical projects were still in the planning phase, it was potentially feasible to assess changes within a subset of the sites. Accordingly, a substudy of six communities was designed to assess changes in the community environment in Years 4 and 5 when new physical projects were implemented in this time frame. The following selection criteria were used:

1. Stage of intervention: physical projects planned but not implemented at the start of the evaluation time period;

2. Evidence of a policy or physical project intervention: community partnerships with a contract for work in place or a time frame for project completion;

3. Focus of intervention: portfolio of physical projects capturing a range of interventions related to both increasing transportation-related and recreational physical activity;

4. Population: portfolio of physical projects representing a range of interventions for vulnerable populations or children;

5. Focus on environment and policy change: physical projects representing larger-scale changes as opposed to smaller-scale promotional or programmatic changes (e.g., stair use prompts);

6. Generalizability to other communities: physical projects that could be adapted to many other communities based on various climates or geography (e.g., urban versus rural); and

7. Capacity of partnership: physical projects likely to be implemented based on resources and expertise of the partners in the community partnership.

Methods

To address the primary evaluation aims, the evaluation team developed a mixed-methods, triangulated approach that included data from grantees, the ALbD NPO, and the external evaluation partners (Transtria LLC and the Prevention Research Center in St. Louis). Table 1 describes the linkages between the aims and the eight evaluation methods. Each is briefly described in the following sections and contrasted in Table 2, based on the experience of implementing the evaluation methods. All evaluation tools are available at www.transtria.com. Due to knowledge gained in the course of the evaluation, two methods (Tables 1 and 2) and in the following sections (i.e., environmental audits, direct observation) were explored for feasibility across six communities included in the substudy.

Table 1.

Active Living by Design evaluation aims and corresponding methods

Aims	Cross-Site Methods (n = 25 communities)						Substudy Methods (n = 6 communities)
	Partnership capacity surveys	Concept mapping	Progress reporting system	Key informant interviews	Focus groups	Photos and videos	Environ- mental audits	Direct observation
Aim 1: To assess the environmental impacts of physical projects and related policy changes, and, where appropriate, the influence of these interventions on physical activity behavior			X	X	X	X	X	X
Aim 2: To document the range of interventions implemented across the communities as well as associated intended and unintended accomplishments		X	X	X	X	X
Aim 3: To identify strengths and challenges in the process of planning, developing, and implementing the interventions	X	X	X	X	X

Table 2.

Active Living by Design evaluation methods, properties, strengths, and weaknesses

Method	Purpose	Indicators	Participants/ Observations	Strengths	Weaknesses
Partnership capacity surveys (Administered February 2007 to November 2008)	To identify the characteristics of the partnership, its leadership, and its relationship to the broader community.	Partnership’s purpose and goals Partnership functioning Leadership Partnership resources Partnership’s relationship with the broader community	Community partnership members and staff (n = 28 respondents and 25 communities)	Requires few resources for data collection or analysis Enables site and cross- site analysis of partnership characteristics	Does not address the capacity of individual partners Requires additional information to understand structures and functions
Concept mapping (Administered February 2007 to November 2008)	To use a participatory approach to identify, categorize and prioritize successful active living strategies for creating community change and increasing physical activity behavior.	Actions or changes that occurred in the community to support active living through: creating community changes (e.g., new policies or environments); and increasing physical activity behavior of community members.	Community partnership members, staff, and community members (n =43 respondents; n = 23 communities)	Uses a participatory approach Analyzes qualitative data using a quantitative structure Allows for overall and subgroup comparisons Produces visual images of results	Time-intensive Conceptually challenging (sorting and rating many ideas) Requires expertise for analysis and interpretation
Progress Reporting System (PRS) (Administered July 2004 to May 2010)	To track planning and implementation activities as well as intended and unintended consequences of these activities in real time.	Partnership (activities, products) Preparation (assessment, resource generation) Promotions (media coverage) Programs Policy (advocacy, planning products, advisory councils) Physical Projects Sustainability (long-term planning)	Project director and/or coordinator, ALbD National Program Office staff (n = 25 communities)	Focuses on goals, tactics, and benchmarks created by the community partnerships Keeps a log of all activities conducted	Time-intensive Depends on quality/ complete entries Requires expertise for categorizing entries
Key informant interviews (Administered February 2007 to October 2009 [includes follow-up])	To gain insight into the overall ALbD initiative and the community partnership’s efforts from the perspective of key staff and partners and to set the stage for the site visits by the evaluation team.	Lead agency and community partnership characteristics (historical, current, strengths, challenges) Planning and implementation activities Intended and unintended consequences	Staff (n = 31 pre-site visit, 57 site visit, and nine follow-up respondents in 25 communities) Partners (n = 1 pre- site visit, 69 site visit, and five follow-up respondents and 23 communities)	Gathers what, who, where, when, how, and why responses Captures emotional responses Offers flexibility to clarify or probe in areas of interest	Time-intensive to analyze Reflects only one perspective Requires expertise or experience in areas of interest
Focus groups (Administered February 2007 to November 2008)	To validate what has been reported in the ALbD PRS and to reflect on the overall ALbD initiative and community partnership efforts through subgroup discussions with various stakeholders -- community partners and staff (planners and implementers) as well as community members (those benefiting from the interventions).	Community assets and needs Lead agency Community partnership Planning and implementation activities Intended and unintended consequences Strengths and challenges of the initiative Technical assistance provided by the ALbD National Program Office	77 total focus groups Staff (n = 67 in 23 communities) Partners (n = 215 in 25 communities) Community members (n = 201 in 24 communities)	Gathers what, who, where, when, how, and why responses Captures social and emotional responses Offers flexibility to clarify or probe in areas of interest Obtains multiple perspectives Generates new ideas or questions	Time-intensive to analyze Often requires travel (in- person) Restricted to only a few topics rather than a broad spectrum of topics
Photos and videos (Administered February 2007 to November 2008)	To capture physical activity behavior, environmental conditions, or intervention activities.	Images of people and their behaviors Images of environmental conditions (before and after intervention) Images of the impact of various intervention activities (participation in a design workshop, promotional materials)	Streets, trails, recreation facilities, and community members (n = 25 communities)	Provides visual representation of project impacts Conveys project impacts to diverse audiences	Expensive depending on equipment and production Requires consent for photo release
Environmental audits (Administered February 2007 to August 2007)	To serve as a guide for taking photographs of the project area and to document the implementation of physical projects.	Types of residential and nonresidential land uses Pedestrian and bicyclist infrastructure Street design characteristics Traffic-calming and safety measures Parks, playgrounds, and recreational facilities (presence and condition)	Street audits (n = 45 segments in five communities) Trail audits (n = 3 in three communities) School facility audit (n = 1 in one community)	Uses a validated tool for data collection Allows for pre/ post comparison Assesses the impact of policies or physical projects on environmental conditions	Not comparable across different communities or physical projects (see text) In certain cases did not have facilities or environments to audit at baseline Time- and resource- intensive The need to audit multiple settings (schools, communities, worksites) makes a single audit tool ineffective Data reduction and analysis can be complicated
Direct observation (Administered February 2007 to August 2007)	To document the impact of physical projects on the physical activity behavior of community members.	Counts of individuals (e.g., children, adults) as well as their physical activity level (sedentary, walking, biking, running) in selected environments	Streets (n = 11 locations in five communities for 30 hours of observation) Trails (n = 3 trails in three communities for 8 hours of observation School facility (n = 1 facility in one community for 1 hour of observation)	Allows for pre/post comparison Evaluates the impact of physical changes or improvements on behavior	Depends on external factors (e.g., weather, special events) Requires many observations (times of day, days of week)

Partnership Capacity Survey

Partnership capacity involves the ability of communities to identify, mobilize, and address social and public health problems.^27–29 Modeled after earlier work from the Prevention Research Centers,³⁰ a 38-item partnership capacity survey solicited perspectives of the members of 25 community partnerships on the structure and function of the partnership. Participants completed the survey online and rated each item using a 4-point Likert scale (strongly agree to strongly disagree). Responses were used to reflect partnership structure (e.g., new partners, committees) and function (e.g., processes for decision-making, leadership in the community). The partnership survey topics included: the partnership’s goals are clearly defined, partners have input into decisions made by the partnership, the leadership thinks it is important to involve the community, the partnership has access to enough space to conduct daily tasks, and the partnership faces opposition in the community it serves.

Concept Mapping

Concept mapping was used to engage community partnerships in the definition and operationalization of successful active living strategies. This technique provides a visual representation of the complex relationships among ideas and integrates qualitative and quantitative methods.³¹ In addition, it provides broadly dispersed participants with the opportunity to identify ideas and participate in the interpretation of their group perceptions.³² The process includes six overall steps: (1) preparation (select a group of participants and determine focus); (2) group brainstorming to generate statements; (3) structuring statements through a sorting process to create clusters; (4) representation of the statements/clusters using a map; (5) interpretation of the maps; and (6) utilization of the maps.

Responses can be analyzed across all communities as well as by subgroup (e.g., racial and ethnic populations) to determine priorities and how they may differ across populations and settings. This method was used as part of the ALbD evaluation to determine the most important actions that occurred across the 25 communities for creating change and increasing physical activity. Detailed application of this method to the ALbD initiative is summarized in a companion paper³³ in this supplement to the American Journal of Preventive Medicine (AJPM).

Progress Reporting System

Described in detail elsewhere,²⁶ the PRS gathered historical and detailed information regarding community partnerships’ activities and what had been learned. Using a web-based system, key project personnel from all partnerships documented and categorized (using the 5P Model) specific project activities. ALbD NPO staff monitored and reviewed the entries and generated reports to examine progress made with respect to the goals, tactics, and benchmarks from the work plans that had been previously developed by grantees. This system contributed to the ALbD evaluation by documenting community partnership activities and accomplishments. For example, local ALbD staff and partners reported playing a lead, contributing, or indirect role in securing ~$256 million for active living programs and environmental supports.²⁶

Key Informant Interviews

Key informant interviews provide an opportunity for in-depth dialogue with individuals who have expertise, experience, or perspectives that can be helpful to understanding more-detailed information about the community partnerships’ activities. Key informants often have important and unique information about a policy or program.³⁴ In some cases, they may be superior to focus groups because one does not need to assemble a group, and more-sensitive information might be disclosed. Interviews, containing a standard set of questions and probes, were conducted by phone or in-person with project staff and partners before, during, or after each site visit.

The sample began with the lead project staff and followed snowball sampling methods to identify additional respondents. Sample topics included: how long the community partnership was in operation, why the partnership was established, what organizations/agencies/coalitions served on the partnership, whether community members were involved in the partnerships, the major strengths/challenges of the partnership in meeting ALbD goals, other sources of funding besides ALbD, factors that contributed to successfully bringing in other resources, and ways to sustain the partnership after ALbD funding.

Interviews were recorded, transcribed, and subsequently coded by theme (e.g., engaging partners, policy development, working with children). Follow-up telephone interviews were conducted with all of the community partnerships who had site visits between February 2007 and August 2007 (n = 8), in order to ensure that their policy and physical project strategies were adequately captured by the evaluation.

Focus Groups

Focus groups allow evaluators to draw on the collective expertise, experience, or perspectives of several individuals in order to generate input, solicit feedback, or build consensus regarding the community, the partnership, or the activities.³⁵ Focus groups gather information that may not be obtained from key informant interviews because the group process allows for interaction among participants, often leading to information-rich discussions. Multiple focus groups, using standard questions and prompts, were conducted during site visits with all 25 partnerships, including individuals representing various subgroups (i.e., staff, partners, and community members). As is common in focus group methods, the groups were kept as homogenous as possible. The community partnerships determined the composition of focus groups.

Facilitators directed the conversations using the questions/prompts and allowed participants to guide the conversations by their comments. Sample prompts asked respondents to describe: populations/settings chosen for interventions; promotion, program, physical project, and policy successes; strategies that did not work; steps taken to implement the interventions; interventions that had the greatest impact; and technical assistance from the ALbD NPO that was either helpful, not helpful, or missing. Focus groups were recorded, transcribed, and subsequently coded by theme (e.g., community assets, funding or resource challenges, perceived benefits of physical projects).

Photos and Videos

Digital photographs or videos visually portrayed active community members, the condition of facilities or environments, and the impact of interventions (e.g., changes to the environment). Photos or videos were taken of the project areas following established methods in order to assess³⁶: planned and completed physical projects such as the addition of pedestrian or bicyclist infrastructure, activities or events such as walking groups and fitness classes, and community members’ behaviors to highlight physical projects and activities. Community partnership staff provided a tour of all of the current or future sites for intervention activities for the evaluation team. On the tour, the evaluation team used photos and videos to supplement and validate findings from the qualitative data collected (e.g., images of new physical projects, images of promotional signage). Photos and videos involving human subjects proved difficult given that it was often not feasible to obtain photo release forms from community members.

Environmental (Community) Audits

An environmental audit is an unobtrusive, systematic assessment of factors in the physical and social environment that can hinder or facilitate physical activity (e.g., street pattern, number and quality of public spaces, children playing in public spaces).³⁷ Audits document specific features of the environment or changes to the environment coinciding with intervention implementation. Environmental audits can be conducted before and after the implementation of a physical change in the environment. For example, an environmental audit might assess factors affecting walkability, before versus after the addition of sidewalks or completion of a community trail.

For the evaluation substudy, the audit tool was derived from the Active Neighborhood Checklist³⁸ and served primarily as a guide for taking photographs of the project area in order to determine if environmental conditions for physical activity changed as a result of the changes implemented. Evaluators learned that the usual audit method had to be adapted for this substudy for the following reasons: (1) the evaluation team lacked sufficient time at each community visit to conduct audits of the entire project area, (2) the focus on specific physical projects represented a range of different settings (e.g., school recreational facilities, trail development) that would require use of multiple audit tools, and (3) the community partnerships did not have the staff or resources to participate in data collection. Therefore, the evaluation team modified the intended use and application of the Active Neighborhood Checklist to save time, focus on specific physical projects, and maximize resources. Photographs, guided by the audit tool, were taken at sites for specific physical project during baseline (February–August 2007).

The physical projects were not implemented in time for follow-up data collection (February 2009 to August 2009), with the exception of one site that had partially completed the installation of new parks and streetscape changes. Even in these sites at the time of evaluation, parks were not yet publicly accessible and the streetscape changes were not systemwide. Despite these timing challenges, the evaluation team provided tools and on-site training for two partnerships (Somerville and Columbia MO), and findings from direct observation for one of these communities are highlighted elsewhere in this supplement to the American Journal of Preventive Medicine.²¹

Direct Observation

Direct observation for physical activity research records the number of community members, their selected characteristics (e.g., age group, gender), and their activity level (e.g., sedentary, walking, biking, running) at a particular location over a specified time period. Observations can be made before and after the implementation of a physical change in the environment (e.g., addition of sidewalks to school, completion of a community trail) to determine if an increase in physical activity has occurred, as determined by the number of individuals or the intensity of activity in which individuals are engaged. For example, reliable observational tools have been developed for school settings (System for Observing Play and Leisure Activity in Youth; SOPLAY)³⁹ and parks (System for Observing Play and Recreation in Communities; SOPARC).⁴⁰

For baseline data collection in the evaluation substudy (February–August 2007), the evaluation team conducted direct observation of community members using facilities (e.g., trails, playgrounds) or environments (e.g., parks, streets) scheduled to be developed or redeveloped as part of the physical projects. The evaluation team had limited time in each community to perform direct observation in the desired manner (e.g., good weather conditions, observations at multiple times per day on multiple days per week). While the evaluation team planned to train available partners, staff, or community members to conduct direct observation, this was not feasible given competing priorities for their time. In addition, the baseline data collection was not feasible for some physical projects occurring on undeveloped, and therefore, unused land (i.e., no community members to observe). As a result, the evaluation team used photos and videos to track users (or the absence of users) with respect to facilities or environments.

As noted, the physical projects were not implemented in time for follow-up data collection (February–August 2009). Even in the one site that partially completed installation of parks and streetscapes, the parks were not yet publicly accessible and the streetscape changes were not systemwide at this point. Despite these timing issues related to use of the audit and direct observation methods in the substudy, the evaluation team provided tools and onsite training to the two communities funded by Active Living Research. The analytic approach and findings from direct observation for one of these communities are highlighted elsewhere ²¹ in this AJPM supplement.

Findings: Challenges to the Evaluation

Based on the collective experiences of the project team in evaluating ALbD, several important challenges emerged. Many of these are summarized in Table 2 as they pertain to a particular evaluation method. Here, several more global challenges are noted and how they were addressed in the current evaluation. While they do impose limitations on the findings reported elsewhere in this AJPM supplement, the mixed-methods approach produced conclusions that were better supported than they would have been if only single methods had been employed.⁴¹

Given that the large-scale evaluation of ALbD began at Year 4 of the 5-year cycle, the lack of baseline data for ALbD presented a substantial challenge. A related limitation is that many of the ALbD interventions represented “natural experiments.” These are naturally occurring circumstances in which different populations are exposed or not exposed to a potentially causal factor (e.g., a new policy) such that it resembles a true experiment in which study participants are assigned to exposed and unexposed groups. Natural experiments are unpredictable in their timing and scope, which brings the accompanying evaluation challenges. Mercer et al.⁴² provides useful advice to those designing evaluations that often involve tradeoffs between the search for gold standards and messy, real-world conditions that add complexity and context to translational research.

In some communities, the larger-scale physical projects were not fully implemented during the evaluation time period or the community partnerships encountered challenges that led them to focus on alternative physical projects. To some extent, this limitation “comes with the territory” when studying environmental and policy changes in the real world. Nevertheless, particular findings show promise within the cross-site findings,⁴¹ in Somerville,²⁰ and in Columbia.^21,22 In these evaluations, physical projects were plausibly related to changes in the physical and social environment for walkability and bikability. Rigorous attribution of cause was not possible, but uncertainty about the approach was greatly reduced. When one considers the relatively low funding levels and takes the range of data collection methods into account, the overall record of the ALbD program is promising.

Regarding measurement, any one method has limitations, but across the program of evaluative inquiry, the use of mixed methods tended to point toward a common theme of progress (triangulation). The measurement of behavior, in this case physical activity, provides an example. Some community partners conducted direct observations of bicylcing and walking,^20,41 one used accelerometers with children,²² and others asked people to self-report their physical activity.^21,43 Taken together, however, they strengthen the claims overall about behavior change. The same applies to changes in the neighborhood ecology. Some communities engaged in environmental audits, whereas others surveyed residents about their perceptions of environmental supports or barriers to physical activity.

Another challenge involved the difficulty in documenting ongoing changes in policy. Although local policy change shows high potential for addressing active living,⁴⁴ there are few established approaches for conducting local policy surveillance.⁴⁵ The information obtained using the qualitative and quantitative methods took a substantial amount of time and effort to analyze and summarize. Yet, these data could serve as a basis for ongoing policy surveillance. For ALbD, the PRS was an important tool for tracking local policy decisions such as new ordinances or pedestrian master plans.²⁶ Tools such as the ALbD PRS can be useful in allowing local partners and public health workers to track policy change.

Often, large-scale programs pay attention to behavioral endpoints, yet fail to capture longer-term issues such as institutionalization and maintenance.⁴⁶ This was true for the current evaluation, as the evaluation was not intended to assess long-term changes in physical activity and active living, but rather focused on more-proximal short- and intermediate-term outcomes. To some degree the companion evaluations of Somerville,²⁰ Columbia,^21,22 and, in particular, Wilkes-Barre PA,⁴⁷ provide complementary information about institutionalization and maintenance. Use of evaluation frameworks such as the RE-AIM model (an acronym for: Reach, Effectiveness, Adoption, Implementation, and Maintenance)⁴⁸ should help evaluators to more fully capture longer-term changes.

Conclusion

The types of environmental and policy change initiatives addressed by the ALbD national program and its grantees proved to be crucial in creating supports for routine physical activity. The mixed-methods evaluation of the ALbD experience should benefit other community-based efforts to address population health, including obesity and other chronic diseases. However, to address population health, additional time should be allowed for post-intervention follow-up to obtain a clearer understanding of the impact on physical activity and other health outcomes, such as chronic disease risks factors, obesity and other chronic diseases.