Measuring Community Programs and Policies in the Healthy Communities Study

Abstract

Childhood obesity is a challenging public health issue facing communities throughout the U.S. Local efforts are believed to be essential to assuring environments that support physical activity and healthy food/beverage consumption among children and their families. However, little is known about how broadly and intensively communities are implementing combinations of programs and policies that address childhood nutrition, physical activity, and weight control. The Healthy Communities Study is a nationwide scientific study in diverse communities to identify characteristics of communities and programs that may be associated with childhood obesity. Data collection occurred in 2013–2015; data analysis will be completed in 2016. As part of the Healthy Communities Study, researchers designed a measurement system to assess the number and scope of community programs and policies and to examine possible associations between calculated “intensity” scores for these programs and policies and behavioral and outcome measures related to healthy weight among children. This report describes the protocol used to capture and code instances of community programs and policies, to characterize attributes of community programs and policies related to study hypotheses, and to calculate the intensity of combinations of community programs and policies (i.e., using the attributes of change strategy, duration, and reach).

Introduction

Childhood obesity is a critical public health challenge facing communities.^1–3 To accelerate progress in obesity prevention, communities throughout the U.S. have engaged—to varying degrees—in creating environments that support healthy nutrition, physical activity, and healthy weight.⁴ These activities include efforts to:

1. increase healthy food/beverage choices (e.g., through programs to improve access to healthy foods in stores and school lunches); and

2. increase physical activity (e.g., through changes in bike lanes, walking trails, and school policies to assure more time for youth to be physically active).^4,5

Yet, we know little about the scope and intensity of efforts by communities to implement multiple programs and policies relevant to childhood obesity. To accelerate health promotion efforts, we need to understand how community efforts occur under real-world conditions and to learn whether combinations of community programs and policies (CPPs) are associated with children's diet and physical activity behaviors and healthy weight outcomes.⁶

Acquiring this information requires a measurement system for documenting the implementation of multiple CPPs, in different sectors, over time.⁶ Because not all CPPs are equally influential, even under optimal implementation conditions, it also requires accounting for differential levels of influence, while also estimating the intervention strength of CPPs, singly and in combination.⁷

This report outlines the community measurement system used in the Healthy Communities Study (HCS). The HCS was initiated by the NIH, and this protocol is part of a research design developed by Battelle Memorial Institute, NIH, and university partners. The goal of the HCS is to examine associations between characteristics of CPPs and diet, physical activity, and BMI among children. The overall research design and other components are described more fully in other papers in this supplement. This protocol was used to capture and characterize CPPs occurring in up to 125 diverse communities throughout the U.S. This report describes the community measurement methods used in the HCS.

Methods

Context for Protocol Development

A core research question in the HCS is whether an association exists between CPPs and outcomes related to childhood obesity. The study required a systematic method⁷ to capture, code, characterize, and calculate the intensity of community efforts to prevent childhood obesity. The Work Group for Community Health and Development at the University of Kansas (KU Work Group) led the development of the community measurement protocol based on its prior experience in community measurement.^7–10

A Community Measurement Subcommittee of the HCS was formed to oversee development of the protocol. Members included representatives from NIH, CDC, Battelle Memorial Institute, and the university partners on the study. The aim of this community measurement approach was to identify CPPs and characterize them sufficiently to explore the main hypotheses of the study; those focused on determining if specific combinations of CPPs are associated with BMI, diet, and physical activity among children.

Overview and Development of Protocol

Drawing on community measurement methods developed by the KU Work Group,^7,8 the protocol was designed to capture and characterize the multiple and varied community programs and policies emerging over time in diverse communities.

For this study, CPPs were defined as the presence of a program, policy, or other activity in the community related to improved nutrition, increased physical activity, or weight control (preventing childhood obesity). Data collection occurred in 2013–2015; data analysis will be completed in 2016. Retrospective collection of CPP data allowed for CPPs to be documented for a longer study period (2003–2015). Types of CPPs included:

1. programs (e.g., nutrition program in a youth organization);

2. policies (e.g., new physical activity requirement in a school district); and

3. environmental changes (e.g., expanded bike path).

To be coded as an instance of a CPP, the activity had to be related to the goals of the study, occur during the study period, directly or indirectly address children aged 4–15 years, and be implemented in or affect the high school catchment area (that defined the community).

Two primary methods were used to capture instances of CPPs. Initially, community liaisons (CLs) conducted interviews with key informants (KIs; mean, 12 per community) drawn from multiple sectors. KIs were individuals identified as having knowledge of community efforts in their organizations and other settings and sectors (e.g., schools, government). Priority roles of KIs from targeted sectors included school principals, leads of health coalitions, health department and hospital representatives, parks and recreation staff, and staff of other governmental and community-based organizations. Then, document abstraction was used to capture instances of CPPs from archived reports, web searches, and other written sources identified through searches and information from KIs.

Not all community programs and policies are likely to have an equivalent effect on health behaviors and related obesity outcomes, even in an ideal implementation scenario. Moreover, the implementation of each program and policy is likely to vary across communities, as a result of differences in local infrastructure, funding, staff availability, and other factors.

“Intensity” scoring provides a way to take the potential influence of these activities into account and to estimate the “dose” of intervention actually delivered in each community. This scoring approach also allows for an examination of the association between different amounts and kinds of CPPs with key outcome variables (i.e., dietary intake, physical activity, and BMI) at different times in the study period. In a complex system, an intensity score for a given CPP should take into account multiple attributes, such as reach or change strategy, as well as real-world circumstances that influence associations between programs and policies and changes in childhood obesity.

Accordingly, the team defined the construct of intensity⁷ (of CPPs) to reflect three characteristics:

1. behavioral intervention strategy (e.g., modifying access was weighted higher than providing information);

2. duration (e.g., an ongoing policy was weighted higher than a 1-day event); and

3. reach (e.g., a program reaching 20% of the population was rated higher than one reaching 2%)^7,11–13.

To capture the dynamic unfolding of comprehensive community efforts over time, a composite intensity score was computed for each community for each year of the study.

The community measurement protocol was approved by the HCS Executive Committee, IRBs of Battelle Memorial Institute and the University of Kansas, the Observational Study Monitoring Board, and the Office of Management and Budget.

Steps in Implementing the Community Measurement Protocol

A four-step protocol was implemented to:

1. Capture community programs and policies using KI interviews and document abstraction. CLs were trained by the KU team to use the community measurement protocol to conduct structured interviews with KIs; the training included workshops, feedback on performance in scoring, and practice to mastery. CLs reviewed community information sources to locate potential KIs within the specific priority roles (e.g., school principal, staff in parks and recreation). Using phone calls and written communications, they communicated the types of information sought and confirmed whether the person was appropriate before requesting an interview time via phone or in person. They also requested any key documents (e.g., final reports, reports to funders) that could help identify relevant CPPs.

   Using a structured KI interview, CLs asked about instances and attributes of CPPs that occurred in or affected the defined community over a 10-year period. The interview protocol sought to obtain:

   1. a list and description of specific CPPs that the KIs organization implemented during the study period;
   2. the timing of the activity (e.g., year of onset, offset); and
   3. a detailed characterization of each CPP (e.g., strategy used, behavioral objective addressed).

   The focus was on obtaining information on the complete set of CPPs implemented in each community and their full characterization, including information to examine the primary study hypotheses. Initial open-ended questions probed for CPPs (e.g., What specific programs (policies) were implemented by your organization to…) and information about who did what. This information was used by researchers to help characterize CPPs after the KI interview.

   Follow-on questions probed for more specific information, and these were followed by response options. After capturing CPPs, the KI interview also sought information to help characterize factors that facilitated and restrained community efforts. Questions and responses were prompted through a Federal Information Security Management Act–compliant Information Management System maintained by Battelle.

   In addition, document abstraction was used to capture other instances of CPPs and to help clarify or confirm information reported in KI interviews. Research staff obtained and reviewed documents (e.g., annual program reports, reports to funders) and searched for web-based information about community activities. The information was used to capture candidate CPPs that were then scored using a codebook and scoring instructions.

2. Code instances of community programs and policies.

   Coding of activities captured through KI interviews and document abstraction was conducted by CLs and Battelle staff, with training and technical support provided by the KU team based on prior research.^7–9 Supports for systematic scoring included a codebook with definitions, examples and non-examples, and scoring instructions that defined an instance of a CPP.

   For a program or policy to be scored as a CPP, it had to meet all of the following criteria:

   1. It occurred (e.g., was not just planned).
   2. It was a program, policy, or other change to the environment (e.g., walking trail) that existed in the community during the study period.
   3. It was related to nutrition, physical activity, or weight control/prevention of childhood obesity.
   4. It targeted or benefited children aged 4–15 years.
   5. It occurred in or benefited children in the defined community.

   The Battelle team scored each documented activity as an instance (or not) of a CPP. The KU team ensured data quality by independently scoring a randomly selected set of identified CPPs. The reliability standard used was an interobserver agreement of ≥80%; lower levels of agreement triggered re-training and certification of coders.

3. Characterize community programs and policies for key attributes.

   Once captured and coded, instances of CPPs were further characterized for attributes used in intensity scoring (see below) as well as other HCS hypotheses (e.g., behavioral objective addressed) and aspects of interest (e.g., sector in which implemented).

   The CL posed specific questions during the KI interview to help characterize CPPs. For instance, to characterize the behavioral objective addressed by the CPP, CLs asked: What were the key behavioral objectives of the community program or policy? What behaviors of children were expected to change? These were followed by response options; for example, those for behavioral objectives related to physical activity included increase walking or biking to/from school, increase exposure to physical education, among others.

   To characterize CPPs for intensity scoring, each CPP was coded for three specific attributes:

   1. behavioral intervention strategy used (i.e., providing information and enhancing skills; enhancing services and support; modifying access, barriers, and opportunities; changing consequences; or modifying policies and broader conditions);
   2. duration (i.e., description of the event as a one-time occurrence, occurring more than once, or being ongoing); and
   3. reach (i.e., what proportion—high, medium, or low—of the total priority population was involved in or experienced the program or policy?).

   Each category was assigned a numerical value based on its relative strength for the attribute.⁷

4. Calculate intensity scores for community programs and policies.

   Utilizing the scores collected through the characterization of CPPs described above, the study team rated each dimension on a scale of 0 (minimum) to 1 (maximum), and summed the results to obtain a single intensity score⁷ for each program and policy. Scores were then summed for all CPPs in place for each year of the study period.

   Table 1 describes the categories and assigned values for weighting the potential impact of a particular CPP. The values across all three domains were summed to create a single overall intensity score. Using this approach, the CPPs documented in the HCS could range in score from 0.3 (weakest and potentially of less influence on longer-term outcomes) to 3.0 (strongest and potentially of greater influence). The formula used to calculate intensity scores was as follows:

   $$\sum \text{Strategy value}+\text{Duration value}+\text{Reachvalue}.$$

   Initial data analyses for intensity scoring will be computed using a formula that weights all elements equally; subsequent analyses will test variations of weighting that aim to capture the complex interactions among CPPs.

   Table 2 provides two examples of CPPs observed in participating communities, organized by primary goal addressed, type of characterization, and intensity score (using methods described above).

Table 1. Calculating the Intensity Score for Documented Community Programs and Policies Using 3 Factors⁷.

Characterization and weight for intensity scoring
Dimension	Scoring rubric for characterizing intensity of documented community programs/policies (1=highest intensity; 0=lowest intensity)	Examples
Type of Behavioral Intervention Strategy	High (1.0) – Modifying policies and systems	Competitive pricing for food choices
	Changing consequences;
	Modifying access, opportunities, and barriers	Establishing a community garden or walking path
	Medium (0.55) – Enhancing services and support;	Providing peer support for physical activity
	Low (0.1) - Providing information and enhancing skills	Providing an educational program for physical activity
Duration	High (1.0) – Ongoing (i.e., throughout the designated year of the project/study period;;	A new bike bath continuously available
	Medium (0.55) – Occurring more than once during that year;	A program that has 16 sessions over the year
	Low (0.1) - One-time event	A local health fair or 5 k
Reach	High (1.0) – 21% or more of the population to benefit/exposed to the CPP;	A city-wide pedestrian policy having an impact on all residents
	Medium (0.55) – 6-20% of the population exposed to the CPP;	A policy implemented at a few schools in the district
	Low (0.1) - 0-5% of the population exposed to the CPP	A program engaging fewer than 50 children

CPP, community programs/policies

Table 2. Examples of Scoring for Community Programs and Policies.

Community/Program policy (Goal addressed)	Duration	Reach	Behavioral intervention strategy used	Intensity score
Created walking path/greenway to connect neighborhoods and schools (Physical activity)	Ongoing	High	Modifying access, barriers, and opportunities	1.0
Provided an educational seminar to parents attending elementary school Parent Teacher Association meeting about how to promote healthy eating among children. (Healthy eating)	One-time event	Low	Providing information and enhancing skills	0.10

Assuring Quality of Community Measurement

Several mechanisms were used to ensure the quality of data collected using the community measurement protocol. First, the KU team provided training for Battelle staff and CLs in capturing, coding, and characterizing instances of CPPs. Second, the KU team conducted direct observations of KI interviews (via phone) and scored those sessions for compliance with the protocol. The minimum score required for all observations was 80% compliance with the protocol; CLs who received lower than the minimum score received refresher training, feedback, and additional monitoring. Third, weekly meetings of the HCS Quality Assurance and Quality Control Committee reviewed reports about the distribution of KIs across sectors and the degree of completeness of the data. Fourth, the Battelle and KU staff shared responsibility for scoring documented activities as CPPs. Independently, each party used descriptions of the activity and responses to key questions to indicate whether each CPP met all criteria for inclusion. Fifth, the KU team also scored all elements of the variables used in the intensity score to assure reliable reporting and coding by the KIs and CLs.

Discussion

This community measurement protocol was designed for the HCS, a large, multi-community study with the primary aim of examining whether there is an association between CPPs and behaviors and healthy weight outcomes of children in participating communities. Thus, the overall goal of the community measurement system was to identify CPPs related to promoting healthy nutrition and physical activity of children and to characterize these sufficiently to estimate their intensity. Intensity scores were computed for each discrete CPP, and for all CPPs in a community, over a 10-year study period. Employing a mixed-methods approach, the protocol used KI interviews and document abstraction—as well as systematic coding and characterization—to permit computations of intensity scores for overall community efforts.

A large-scale, retrospective study of this kind poses challenges for community measurement. First, implementing the protocol required extensive logistical support given the time, people, and other resources required to complete more than 1,500 interviews with KIs in up to 125 communities. Training and assuring quality performance of CLs required extensive initial training in the protocol and monitoring and feedback for quality assurance.

Second, time and resource constraints for KIs and CLs made it difficult to guarantee a complete capture of CPPs that occurred during the study period. Based on pilot testing, the protocol was simplified and refined to gather basic information about CPPs earlier in the KI interviews. Although there is likely a bias towards remembering longer-lasting programs/policies (compared with something of a short duration), data from initial communities showed that KIs could recall community programs/policies as far back as 10 years. If there is bias, it is assumed that it is systematic and can also be managed analytically. Although KIs reported that implementation of CPPs had occurred, the absence of a direct measure of the actual level of implementation is a limitation of this study.

Third, data cleaning was challenging because it required review and refinement of captured information for clarity and completeness, sometimes splitting reported clusters of multiple programs and policies into discrete CPPs. It also required harmonizing data from different KIs and document abstractions into descriptions of activities suitable for coding and characterization.

The HCS community measurement protocol has a number of strengths. First, it helps fill a key gap noted in prior IOM reports on childhood obesity,^1,4,6 that is, the need to systematically unravel and operationalize the “complex web of influences” on physical activity and nutrition that may affect population-level outcomes related to childhood obesity. This community measurement protocol⁷ offers a replicable and standardized approach for capturing and characterizing the multiple and varied childhood obesity prevention programs/policies adopted by local communities over time.

Second, intensity score data from the community measurement system, when integrated with behavioral and outcome measures, will permit analyses of possible associations between community efforts and related child behavior change and healthy weight outcomes. Systematic and consistent measurement across communities will allow for first-ever examinations of such relationships. When further validated, this intensity scoring approach could be useful for predicting the potential collective impact of comprehensive efforts to prevent childhood obesity, as well as for associating specific components of community efforts with beneficial outcomes.

Finally, this community measurement protocol permits the first-ever profile of comprehensive, community efforts to prevent childhood obesity in a diverse sample of U.S. communities. Such data will be vital if we are to better understand how a community's adoption and implementation of combinations of programs and policies may lead to improved health for all its children.