Evaluating School Obesity-related Policies Using Surveillance Tools: Lessons from The ScOPE Study

Abstract

The evidence evaluating the association between school obestiy prevention policies and student weight is mixed. The lack of consistent findings may result, in part, from limited evaluation approaches. The goal of this paper is to demonstrate the use of surveillance data to address methodological gaps and opportunities in the school policy evaluation literature using lessons from the School Obesity-related Policy Evaluation (ScOPE) study. The ScOPE study uses a repeated, cross-sectional study design to evaluate the association between school food and activity policies in Minnesota and behavioral and weight status of youth attending those schools. Three surveillance tools are used to accomplish study goals: Minnesota School Health Profiles (2002–2012), Minnesota Student Survey (2001–2013), and National Center for Educational Statistics. The ScOPE study takes two broad steps. First, we assemble policy data across multiple years and monitor changes over time in school characteristics and the survey instrument(s), establish external validity, and describe trends and patterns in the distribution of policies. Second, we link policy data to student data on health behaviors and weight status, assess nonresponse bias and identify cohorts of schools. To illustrate the potential for program evaluators, the process, challenges encountered, and solutions used in the ScOPE study are presented.

Background

There have been no significant changes in obesity prevalence in youth or adults between 2003–2004 and 2011–2012 (Ogden, et. al., 2014). Despite a decade of school-based efforts, reviews of the literature evaluating the effectiveness of school policies upon student weight outcomes report mixed and inconclusive findings (Chriqui, Pickel, & Story, 2014, Jaime & Lock, 2009). A few cross-sectional reports exist describing direct associations between healthy weight among youth with states-level mandated policies (Taber, Chriqui, & Chaloupka, 2012), strong school district-level wellness policies (Coffield, Metos, Utz, & Waitzman, 2011) and local school-level policies and practices (Fox, Dodd, Wilson, & Gleason, 2009; Kubik, Lytle, & Story, 2005; O’Malley, Johnston, Delva, & Terry-McElrath, 2009). The lack of consistent findings between healthy school policies and weight among youth may be a result of limited methodological approaches (Kropski, Kekley, & Jensen, 2008; Zenzen & Kridli, 2009). Specifically, there is a call for better study designs that address threats to internal and external validity, longitudinal studies, and improved measurement of overall consumption and activity patterns, both in-school and outside school settings (Chriqui, Pickel, & Story, 2014). Longer evaluation periods may be required to see the impact of school policies on student weight (Kropski, Kekley, & Jensen, 2008). Another limitation of the school policy evaluation literature has been a focus on the singling out of one or two policies, especially nutrition related policies, without considering the overall policy environment.

The School Obesity-related Policy Evaluation (ScOPE) study seeks to address these research gaps through the use of existing surveillance instruments. Unique strengths of the ScOPE study include the ability to: comprehensively evaluate the school food and physical activity policy environments; follow schools over time, including cohorts of schools; link school policy data to the behavioral and weight data of students in those schools; and identify patterns in these relationships by geographic location, grade level, school type, and student characteristics. The goal of this paper is to demonstrate the use of surveillance data to address multiple identified gaps in the school policy evaluation literature. To illustrate the evaluation potential for program evaluators, the process, challenges encountered, and solutions used in the ScOPE study are presented.

Methods

The ecological model emphasizes the multiple influences upon energy balance. To capture the complexity of inputs into the school policy environment and student outcomes the ScOPE Study engages an advisory board. Creating a diverse and active advisory board consisting of childhood obesity policy experts from state agencies, public health advocacy and legal fields is helpful to identifying the policy context. As result, a timeline of relevant national, federal, state and local initiatives identified by the advisory board as having potential to influence or place the ScOPE Study evaluation findings in their context was developed and is available from the project website http://z.umn.edu/scope.

The ScOPE Study Design & Measures

ScOPE uses a repeated cross-sectional study design and includes school-level policy and practice data for Minnesota middle and junior-senior high schools and individual-level behavioral and weight status data for 6^th, 9^th and 12^th grade youth. Three primary existing surveillance data sets are used to accomplish ScOPE study goals: Minnesota School Health Profiles (2002–2012), Minnesota Student Survey (2001–2013), and National Center for Educational Statistics Common Core Data. These data sources are described below. The University of Minnesota Institutional Review Board approved this study.

Data Source for School-Level Policies and Practices

Minnesota School Health Profiles (Profiles) is a survey of school health policies and practices sponsored by the Centers for Disease Control and Prevention Division of Adolescent and School Health (CDC-DASH). Profiles surveys school principals (i.e., Principal Survey) and lead health education teachers (i.e., Teacher Survey) in schools with one or more of grades six through 12 every two years. Profiles data are used to monitor implementation of school health policies and educational practices, including, physical activity, food service, and nutrition. CDC-DASH oversees methodology, questionnaire development, and analysis of Profiles data. Data collected from each state Profiles can be used to analyze school nutrition and physical activity policies and practices. Results are weighted to adjust for differing patterns of non-response for years with response rates of 70% or higher. Yearly response rates for all participating Profiles states and cities are available online (http://www.cdc.gov/healthyyouth/profiles/history.htm).

Assembling Profiles schools across years

In Minnesota, a stratified random sample of schools was chosen for each year that Profiles was administered. Approximately 300 schools were surveyed every two years. A total of 866 unique schools were surveyed during the 2002–2010 study period. Due the proportionately large sample of schools selected for Profiles at each time point, individual schools were selected by chance in multiple years. The majority of schools participated three or fewer times (once (44%), twice (32%) or three times (18%)). Forty-five schools participated four times and two schools participated at all five data collection points from 2002–2010. A subset of these schools was included in consecutive collection years. This subset also allows evaluators to identify areas of stability and changes in school-level characteristics within a given school over time. For example, increasing or decreasing enrollment in the free and reduced priced school lunch program, minority enrollment and school geographical location.

The inclusion of the same schools at multiple time points also offers the unique opportunity to track changes in policy and student outcomes in a cohort of schools over time. This cohort design nested within our larger ScOPE study design helps control for unmeasured factors that do not vary over time and allows a more rigorous test of the effects of changing policy on student outcomes within those schools. In addition, by being nested within the larger census of Minnesota schools, with known characteristics, this design provides an opportunity to assess potential selection bias and external validity of the findings.

The Data Source for Student-Level Weight, Diet and Activity Behaviors

Minnesota Student Survey (MSS) is a voluntary and anonymous self-report survey, which is administered to 6^th, 9^th, and 12^th grade students every three years. Between 2001 and 2010, the percentage of operating school districts that participated in the survey ranged from 88 to 91 percent, and the percentage of all Minnesota 6^th, 9^th and 12^th grade students in regular public schools who submitted usable surveys ranged from 68 to 72 percent. Performed statewide, the survey addresses a wide range of topics, including diet, physical activity, and weight related behaviors, among middle school and high school age students. All public and alternative schools with students in eligible grades throughout Minnesota are invited to participate. While Minnesota uses the MSS, most states monitor youth weight and diet and activity behaviors using the Youth Risk Behavioral Surveillance system. Similar to Profiles, assembling MSS across multiple years provides the opportunity to identify secular trends and school-level patterns in student behaviors and weight.

Linking MSS to Profiles

For the ScOPE study, we linked schools with Profiles (e.g., policies) data to their corresponding MSS (e.g., student behaviors, weight) data. To facilitate linkage between the Profiles and MSS data, the Minnesota Department of Health provides a numbering system that uniquely identifies each school district and school building. Unlike the Profiles survey which is completed by a random sample of schools every two years, the MSS survey is offered to all schools every three years, which helps ensure a match to student outcome data for nearly all schools with Profiles data.

Data Sources for School-Level Demographic Characteristics

National Center for Education Statistics Common Core of Data (NCES) is a publicly available source for data on school characteristics and is updated annually (http://nces.ed.gov/ccd/). Key school-level variables drawn from the NCES include lowest/highest grade taught, student race/ethnicity, enrollment, free/reduced price lunch enrollment, locale, county, school type (e.g., magnet, charter, alternative learning center), gender by grade and race and school latitude/longitude.

Assessment of Available Data Quality

To assess nonresponse bias, analyses were conducted to establish whether Profiles schools differed from non-Profiles schools on school characteristics from the NCES data (e.g., grade level, free and reduced priced meal enrollment, minority enrollment and location) at each year. In addition, analyses were conducted using MSS data to establish whether students in schools with Profiles data were differed from students in schools without Profiles data for important behavioral and weight outcomes. Tables are available on the study website. Next, analysis focused on schools with data from both the Profiles and the MSS (i.e. ScOPE study schools). Descriptive statistics were calculated for ScOPE schools and for students within ScOPE schools. Preliminary analysis results identified significant differences in school policy prevalence and student behavioral outcomes between middle schools and high schools resulting in stratified analysis by grade level.

Initial Results and Discussion

Table 1 demonstrates the utility of existing state surveillance data as a school policy evaluation tool by providing a snapshot of the data time points available when combining Profiles (e.g., school policy) and MSS (e.g., student behaviors and weight) over multiple years in Minnesota. Table 1 also highlights challenges encountered in instrument changes and is discussed later in the paper.

Table 1.

A snapshot of the data time points and key instrument changes when combining school policy and youth behavior surveillance survey’s over multiple years in Minnesota.

	2002	2003	2004	2005	2006	2007	2008	2009	2010	2011	2012	2013
School Policies1	X		X		X		X		X		X
Student Behaviors2		X				X			X			X4
Student Weight3						X			X			X4

¹The School Health Profiles (Profiles) is administered to a random selection of secondary schools every two years. School participation in the Minnesota Profiles ranged from 75% in 2002 to 66% in 2012.

²The Minnesota Student Survey (MSS) is administered to all 6^th, 9^th and 12^th grade students, voluntarily, in Minnesota schools every three years. Student participation rates across 6^th, 9^th and 12^th grades for the MSS ranged from 58% to 81% in 2007 and 59% to 79% in 2010. Twelfth graders had the lowest participation rates in both years.

³Student self-report height and weight was added to the MSS in 2007

⁴The following changes were made to the MSS beginning 2007: administration from hard copy to online; administration to 5^th, 9^th and 11^th grades; one item assessing physical activity (previously two items assessing type and duration) and; questions and response options assessing fruit and vegetable intake (e.g., 1 item versus 3, servings versus times). Overall response rates to the 2013 MSS ranged from 62% to 71% across grades. Eleventh graders had the lowest participation rate.

Table 2 presents a preliminary examination of select school policies and practices and student behavioral and weight outcomes in the ScOPE study schools. These are crude results not corrected for sampling design or demographic changes; they are intended to give a sense of the data available for future studies. Overall, these unadjusted bivariate analyses reveal some movement in policies and practices in the desired direction. From 2002 to 2010 the percent of schools with fruits and vegetables available at school events increased about 13% (7.8% to 20.6%) in middle schools and about 23% (4.7% to 27.4%) in high schools. Middle schools report decreased prevalence of vending machines and school stores’ soda pop availability, but these trends are not evident in high schools. A required physical education course for high school students increased from 6% of schools in 2004 to 21% in 2010. There is little change in the percentage of schools offering intramural opportunities in either middle or high schools.

Table 2.

Highlights of the school policies and student behaviors and weight variables available in the ScOPE study

	2002	2010
School practice and policy predictors	Middle School High School % or Mean (SD)	Middle School High School % or Mean (SD)
Yes, fruits/vegetables available at school events	7.8% 4.7%	20.6% 27.4%
Yes, vending machine or school store is available for students to purchase snacks/beverages	89.1% 94.7%	68.5% 92.5%
Yes, students can purchase soda or sports drinks at school that has vending	100% 99.3%	75.5% 97.1%
Yes, PE is required in 6th grade/11th grade	100% (2004) 6.4% (2004)	100% 20.8%
Yes, intramural opportunities or clubs are available to all students	84.1% 53.0%	79.2% 57.8%
	2003	2010
Student behavioral outcomes	Middle School High School	Middle School High School
Servings of fruit and vegetable yesterday	3.1 (1.9) 2.7 (1.7)	3.2 (1.9) 3.0 (1.8)
Glasses of soda yesterday	1.7 (1.9) 1.8 (1.8)	1.1 (1.6) 1.2 (1.6)
Days of 30 minutes of moderate activity	4.0 (2.2) 3.9 (2.3)	4.3 (2.2) 4.1 (2.3)
	2007	2010
Student weight outcomes	9th graders 12th graders	9th graders 12th graders
Overweight (≥85% – < 95%BMI)	12.0% 12.6%	14.2% 12.8%
Obese (≥95% BMI)	6.5% 8.8%	8.0% 9.4%

The raw student behavior data indicate a slight increase in mean daily fruit and vegetable intake, a decrease in mean soda consumption by about a half a serving a day and little to no movement in physical activity levels. The prevalence of self-reported overweight and obesity increased for 9^th grade students in Minnesota ScOPE schools from 2007 to 2010.

Evaluating the effectiveness of national, state and local obesity prevention policies, including those directed at schools, is a nationwide priority in the U.S. The ScOPE study aims to address these priorities in school policy evaluation using existing surveillance data. The present paper describes this process as an example of using existing resources for program evaluators. Key steps to assess the structure, linking and quality of the data have been identified. Next steps include assessment of an expanded list of policies and student outcomes and multivariable analyses to account for the hierarchical structure of the data (e.g., students within schools). The development of the ScOPE data set also provides opportunities to follow a cohort of schools and explore differences in exposures (e.g. sum of policies), across school characteristics (e.g. locales, school types, grade levels) and varying effects upon students (e.g. sex, age, free/reduced lunch eligibility, overweight/obese). One successful application involving a cohort of 37 schools from 2002 to 2006 is discussed elsewhere (Nanney et al., 2014).

Challenges to Using State Surveillance Tools

Instrument changes over time

Like most omnibus surveys, the MSS and Profiles Surveys are evolving instruments that try to respond to emerging needs. Between 2001 and 2010, the survey instruments were stable in some subject areas and underwent substantial change in others. Establishing sets of core questions across survey implementation years is an important activity. Tracking when policy and behavioral questions were added, deleted, slightly modified and substantially changed provides the context for comparisons across years. For ScOPE, groups of questions consistent across 10 years and shorter time periods exist (e.g. 2002–2006 and 2008–2012). See the ScOPE study website for examples (http://z.umn.edu/scope). Finally, it is important to monitor for changes in survey administration and methodology. In Minnesota, the MSS underwent major revisions in 2013 including administration to primarily computer rather than paper-and-pencil and the student participant grade configuration changed. While these inevitable changes bring an end to many long-term trend lines, it is still possible to examine links between policies and behaviors and weight.

Analytic challenges

Analyses are complicated by the various data sources used in this project. Profiles data are collected every other year from a stratified random sample of schools in the state. The CDC provides sampling weights to account for this design and allow for estimation of unbiased sampling statistics. However, the CDC does not provide sampling weights for those years in which the overall response was below 70%, which occurred two years in Minnesota. In order to retain as full a dataset as possible all years are retained in the ScOPE analyses. Sensitivity analyses were conducted to compare findings in models that incorporated sampling weights and those that excluded weights but included the variables used for stratification in regression models (Kubik, et al., 2013). No differences in substantive findings were noted for several key outcome variables in these analyses. Based on these findings sampling weights are omitted from all the years rather than use design-based weights for some years and not others. Instead we adopt a regression model based approach to inference that included the variables the CDC used to define stratification schemes as terms in all analyses (Little, 2004).

ScOPE Generalizability

An important question to ask is whether policy evaluation findings can be reasonably generalized to other states. The ScOPE study used publicly available findings from a national study of state mandated school food and activity policy environments (Nanney et al., 2010) available at http://z.umn.edu/schoolnutrition. Minnesota was characterized as average with a “moderately comprehensive” school policy environment (Nanney et al., 2010). Further work indicated the policies and practices where Minnesota schools lead, are average, and lag by comparing school-level policies and practices of about 300 Minnesota schools to over 6,000 schools in 27 other states (2008) (Nanney, Davey, & Kubik, 2013).

Conclusion

The ScOPE study offers a framework for program evaluators to use existing surveillance data to address the methodological shortcomings cited in the school policy evaluation literature. Use of surveillance data provides the ability to comprehensively describe the school food and activity environments, measure overall diet and activity patterns, link the environments to behaviors of students in those schools and examine temporal relationships. The opportunity to address threats to external and internal validity and conduct longitudinal cohort evaluations adds methodological rigor. Longitudinal cohort studies over extended time periods afforded by the use of existing surveillance surveys may be especially important to capture time likely needed for policy traction. The ScOPE study identifies two broad steps to follow. First, assemble policy data across multiple years and monitor changes over time in school characteristics and the survey instrument(s), establish external validity, and describe trends and patterns (e.g. disparities) in the distribution of policies. Second, link policy data to student behavioral data and assess nonresponse bias and identify cohorts of schools. Finally, apply multivariable analyses to examine policy effects upon students. This approach has been successful applied and has yielded meaningful findings (Nanney et al., 2014). The approach presented addresses key gaps in the school policy evaluation literature and is cost efficient and sustainable for program evaluators.