Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Jun 1.
Published in final edited form as: Health Educ Behav. 2017 Sep 1;45(3):423–434. doi: 10.1177/1090198117726570

Development of a Systems Science Curriculum to Engage Rural African American Teens in Understanding and Addressing Childhood Obesity Prevention

Leah Frerichs 1, Kristen Hassmiller Lich 1, Tiffany L Young 1, Gaurav Dave 1, Doris Stith 2, Giselle Corbie-Smith 1
PMCID: PMC5994767  NIHMSID: NIHMS941036  PMID: 28863266

Abstract

Engaging youth from racial and ethnic minority communities as leaders for change is a potential strategy to mobilize support for addressing childhood obesity, but there are limited curricula designed to help youth understand the complex influences on obesity. Our aim was to develop and pilot test a systems science curriculum to elicit rural African American youth perspectives on childhood obesity and enhance their understanding of and support for obesity prevention solutions. The curriculum was designed so it could be integrated with existing positive youth development curricula that help youth advocate for and implement identified solutions. We conducted four workshop sessions with youth that engaged them in systems learning activities such as guided systems diagramming activities. The participants (n = 21) completed validated surveys presession and postsession that assessed their causal attributions of obesity and support for obesity prevention policies. The youths’ perception that environmental factors cause obesity increased (p < .05), and perceptions that individual behavior and biology cause obesity did not change. Their support for policies that addressed food access and food pricing significantly increased (p < .05). The youths’ system diagrams elucidated links between multilevel factors such as personal attitudes, social influence, and the built environment, which provides important information for designing synergistic solutions. The changes we observed in youths’ perceptions of obesity and support for policy changes have important implications for youths’ interest and willingness to advocate for social and environmental changes in their community. The strategies have a promising role in supporting community mobilization to address childhood obesity.

Keywords: childhood obesity, participatory research, systems science


Childhood obesity is a complex public health issue (Institute of Medicine, 2012). Obesity disparities by race and ethnicity continue to widen (Skinner & Skelton, 2014), and are prevalent in the rural south (Wang, Gortmaker, & Taveras, 2011). In North Carolina, the odds of being obese is 50% higher for rural compared with urban children (North Carolina Department of Health and Human Services, 2009). New strategies that help communities design multilevel, synergistic solutions are needed to address childhood obesity.

The risk for obesity is influenced by interacting causal mechanisms across socioecological levels (Huang, Brownson, Esposito, Green, & Homer, 2013). Interventions targeting one or a few risk factors have resulted in limited population-level impact (Devi, 2008; Huang et al., 2015; Kraak & Story, 2010; Roberto et al., 2015). A collective, societal response is required to disentangle and address the complex factors causing childhood obesity and the disproportionate risk in children of color. Efforts that mobilize a community to collaboratively address multiple levels of change (e.g., individual behaviors, social norms, built environment) are promising. Evidence indicates these efforts lead to reductions in population weight gain and are equally if not more effective among underserved individuals (Boelsen-Robinson et al., 2015). Yet these efforts are not widely adopted (Huang et al., 2015) and readiness for implementing cross-sectoral, collaborative programs and policies may be limited among underserved racial and ethnic minority communities (Frerichs et al., 2012; Pitts et al., 2013; Sliwa et al., 2011).

One potential strategy is to engage youth as leaders to mobilize support for addressing childhood obesity (Frerichs et al., 2015; Millstein & Sallis, 2011). Positive youth development refers to efforts to develop youths’ interests, skills, and abilities, often focusing on leadership and communication skills (Lapalme, Bisset, & Potvin, 2014; Wallerstein & Sanchez-Merki, 1994). Some positive youth development programs guide youth to engage in socially productive activities, for example, advocating for environmental and policy changes (Arredondo et al., 2013; Linton, Edwards, Woodruff, Millstein, & Moder, 2014). Positive youth development curricula exist that can be applied to help youth take action for obesity prevention (e.g., ACT for Youth, 2016; YES! Youth Empowered Solutions, 2016); yet youth (and their adult mentors) also need guidance to understand the factors shaping obesity in order to choose the best actions.

By helping youth understand the multilevel and interconnected influences on obesity, they will be better equipped to contribute to and support a wide range of impactful solutions that address individual, social, and environmental change. Evidence suggests that one’s perceptions of the causes of an issue influences one’s support for societal action (Weiner, 2006). Individuals who more strongly agree that environmental factors cause obesity indicate stronger support for obesity prevention policies than individuals who emphasize personal responsibility (Niederdeppe, Porticella, & Shapiro, 2012), and the extent to which one can make connections between individual and environmental factors may increase policy support (Niederdeppe, Shapiro, Kim, Bartolo, & Porticella, 2014). Furthermore, the interventions employed in community efforts must recognize connections to ensure synergy of investments. For example, interventions that improve individual’s healthy food attitudes without improving healthy food access fail to provide the supportive environmental context for healthy choices (Story, Kaphingst, Robinson-O’Brien, & Glanz, 2008); yet interventions that improve access without considering food preferences will similarly flounder (Cummins, Flint, & Matthews, 2014).

Studies that involve youth in planning and implementing childhood obesity prevention interventions have emerged (Bogart et al., 2014; Dzewaltowski et al., 2009; Eskicioglu et al., 2014; Forneris et al., 2010; Jones, Spence, Hardin, Clemente, & Schoch, 2011; Santos et al., 2014; Smith & Holloman, 2013; Utter et al., 2011). These studies indicate that youth may be effective at promoting changes in individual healthy behavior among their peers, and are promising advocates for environmental and policy change. Although a considerable body of research exists on positive youth development (Catalano, Berglund, Ryan, Lonczak, & Hawkins, 2004), there is limited research on specific strategies to improve youth’s systems learning about the complexity of childhood obesity and guide identification of synergistic multilevel solutions. Systems science engagement activities have potential to improve understanding of and development of more synergistic solutions for complex issues (Frerichs, Lich, Dave, & Corbie-Smith, 2016). In this study, we describe a process for designing and evaluating a systems science curriculum used to elicit rural African American youths’ perspectives on childhood obesity and improve their understanding of and support for addressing childhood obesity.

Method

This study involved two phases: (a) development and (b) pilot testing of systems science engagement strategies, both embedded within a community-based participatory research partnership (CBPR; Corbie-Smith et al., 2011; Corbie-Smith et al., 2015). The study was reviewed and approved by the institutional review board at the University of North Carolina at Chapel Hill.

Conceptual Framework

Systems science is an approach to problem solving that views an issue as being determined by an overall system of interconnected factors rather than targeting discrete factors (Lich, Ginexi, Osgood, & Mabry, 2013). Some systems science approaches include participatory engagement with stakeholders who influence or are influenced by the system of interest (Frerichs et al., 2016; Hovmand, 2014; Vennix, 1999; Voinov & Bousquet, 2010). The goal of engagement is bidirectional: (a) to improve stakeholder understanding of and intention to address factors influencing complex problems and (b) to improve researcher understanding of stakeholder’s knowledge about important processes, decision rules, and mechanisms that influence the issue.

Systems science participatory strategies posit that pathways exist from communication to cognition and cognition to behavior (Rouwette, Vennix, & Felling, 2009; Figure 1). The communication to cognition pathway is explained by dual process theories of persuasion, which hypothesize that high motivation and deep processing of messages will lead to stronger and more stable changes in cognition (e.g., attitudes) than low motivation and shallow processing. The cognition to behavior pathway is explained by the theory of planned behavior, which hypothesizes that constructs of cognition, that is, behavioral, normative, and control beliefs, shape one’s intentions and subsequent behaviors. Systems science participatory strategies act through two mechanisms: (a) visual diagramming and interaction with simulation models and (b) structured group facilitation exercises, which aim to improve processing of complex information and group consensus and commitment to system action. For example, in the case of childhood obesity, we anticipate improved understanding of causes of obesity, including environmental factors, and increased support for a range of recommended individual, social, policy, and environmental strategies.

Figure 1.

Figure 1

Conceptual framework for systems science participatory strategies.

Note. The figure highlights the theoretical basis for use of participatory systems science. The two main frameworks—(a) Dual Process Theories of Persuasion and (b) Theory of Planned Behavior—are indicated at the top of the figure with their pathways (i.e., communication to cognition, cognition to behavior) in italics below. Systems science mechanisms of influence—diagramming and modeling and structured group facilitation—are placed in ovals with arrows directed toward the theoretical pathway and the subsequent outcomes each is hypothesized to target.

Setting

The project was implemented in one North Carolina County, in which African Americans comprise 57% of the population. The county is rural, with a population density of 112 persons per square mile (U.S. Census Bureau, 2010). This county also has the highest unemployment rate in North Carolina and 24% of African American residents live in households with incomes below federal poverty levels (U.S. Census Bureau, 2010).

Phase 1: Engagement Strategy Adaption and Development

Community partners from the larger CBPR partnership met monthly for 5 months with academic research team members. The community partners provided feedback and guidance on recruitment processes, evaluation measures, and the systems science engagement activities. Project updates were also provided to members of the larger CBPR partnership during regularly scheduled coalition meetings.

We developed materials that integrated leadership development and systems science engagement strategies. An outline and description of the activities can be found in Table 1. In summary, the four sessions included the following: (a) introduction to leadership and complex systems problems, (b) using diagramming to understand problems affected by complex systems, (c) identifying intervention strategies within complex systems, and (d) communicating about complex systems.

Table 1.

Descriptions of the Systems Science and Leadership Activities Employed During the Four Youth-Engaged Workshop Sessions.

Session schedule and topic Description of activities Approximate duration of activity Existing sources used or adapted
Session 1: Introduction to Leadership and Complex Systems Science
Community builder and leadership styles activity Community builder: Participants were divided into groups of four to five and completed a team building activity. The groups were provided with miscellaneous materials (duct tape, newspaper) and given 5 minutes to choose and create an animal statue in their small groups. After completion, the participants discussed group dynamics, leadership styles, group strengths, and weaknesses. 30 Minutes Youth Engaged in Leadership and Learning Curricula (John W. Gardner Center for Youth and their Communities, 2007)
Leadership Styles: Participants completed a leadership compass activity. Each participant was provided a handout of characteristics of four major styles of leadership, and asked to circle characteristics that most apply to them. Youth were then divided into four corners of the room based on their identified strongest leadership style, and discussed the group’s distribution, strengths and challenges of each style, and the implications of different styles for addressing obesity prevention in their community. 30 Minutes
Introduction to complex systems and food systems role-playing activity Introduction to complex systems: Brief introduction with didactic presentation to introduce concepts of complex systems and causal loop diagramming. 15 Minutes Materials created for project
Food systems role-playing activity: Youth participated in a hands-on activity to simulate how food choices are influenced by interactions of access and food preferences over time. More information can be found in the supplemental materials available online at http://heb.sagepub.com/content/by/supplemental-data. In brief, two teams of youth were formed and provided with plastic tubs filled with dried beans of two colors (representing healthy and unhealthy items) and a ¼ teaspoon utensil for selecting beans out of the tub. The tubs had different proportions of the two bean colors; representing differential access. The goal of the simulation game was to obtain as many points as possible by getting beans out of the tub. Several rounds were played, and dependent on outcomes of each round, the points for and proportions of the different beans in each tub were changed. 45 Minutes Adapted from “The Bean Game” (Waters Foundation, 2016)

Session 2: Diagramming Complex Systems
Identifying variables to include in the diagrams Variable elicitation: Participants watched two short videos, each highlighting different aspects and important factors to consider related to obesity prevention. The participants were provided with a worksheet to complete while watching the video. The worksheet asked them to list the factors they heard during the video that influenced obesity and the outcomes of obesity. After watching the video, participants discussed their thoughts and a list of the factors they identified was compiled on flip chart paper. 1 Hour Adapted from “Variable Elicitation” Group Model Building Script (Hovmand, Rouwette, Andersen, Richardson, & Kraus, 2013)
Charting outcomes over time Graphs over time: Participants were shown how to draw an example graph that illustrates how a factor or variable changes over time. They were each provided with several graph templates that had an x-axis of time and y-axis (with fill-in blank for the “variable” of interest). They were guided to complete the worksheets using the variables elicited in the previous exercise. 30 Minutes Adapted from “Graph Over Time” Group Model Building Script (Hovmand et al., 2013)
Connecting the variables in diagrams Connection circles: The facilitator drew a circle on flip chart paper and guided them through an example of developing their own causal loop diagram. They were given blank paper and asked to also draw a circle on the paper. Then they were guided through a series of steps to add variables, arrows between variables, specify the direction of the relationship between variables, and finally, identify feedback loops in their diagram. The youth were allowed to work in small groups, but asked to each create their own diagram. The youth were all asked to include two variables (i.e., diabetes and soda consumption) in their diagrams, but to include any other variables they thought were valuable to include. This was done to help focus the conversation and allow for potential linkages. 1.5 Hours Connection Circles (Quaden, Ticotsky, Lyneis, & Walker, 2006)

Session 3: Identifying Leadership Opportunities and Intervention Strategies Within Complex Systems
Recap of key insights of previous session activities Recap graphs over time and connection circles: A selection of graphs over time and participant connection circles were highlighted and discussed at the beginning of the session. 10 Minutes Materials created for project
Presentation of surveillance and research data on obesity Research-based graphs over time and diagrams: Participants were also shown graphs over time that were constructed with U.S. surveillance data and research. For example, highlighting trends in obesity over time among youth in different racial/ethnic minority groups, consumption of different types of food, and activity levels. A causal loop diagram of these factors was also presented. 10 Minutes Materials created for project
Concept simulation model Interaction with a concept simulation model: A simulation model was constructed to specifically highlight a system with both a balancing and reinforcing feedback loop. The specific model highlighted a reinforcing loop that involved changes in weight, motivation to lose weight, and soda consumption; and a balancing loop that involved changes in motivation to lose weight, physical activity, and subsequent energy drink consumption. Different scenarios were described wherein variables of soda consumption, energy drink consumption, physical activity, motivation levels were changed, and participants were asked to predict the output. Output was then shown and they discussed their predictions and how the dynamics influenced the actual outcomes. 15 Minutes Adapted from “Concept Model” Group Model Building Script (Hovmand et al., 2013)
Identifying interventions Identifying interventions and strategies using seed diagrams: Four “seed” causal loop diagrams were created and drawn on flip chart paper. The diagrams were informed by the connection circles the youth created in the prior session. The participants were divided into small groups of four to five, to work on one of the four diagrams, and provided with a worksheet to complete. In the small groups, a facilitator guided them through completing the worksheet which involved activities such as identifying other variables that were needed in the diagrams, the factors they might want to target for change, the types of strategies that might be best to change those factors, and the potential resources and challenges for implementing their strategies. Each small group shared and discussed their worksheet with the whole group. Strategies from all small groups were compiled into one list. 2 Hours Adapted from “Elaborating on a Causal Loop Diagram” Group Model Building Script (Hovmand et al., 2013)

Session 4: Communicating About Complex Systems
Communication activity Communications exercise: Participants divided into pairs and sat back-to-back. In each pair, one participant was provided with a template of a shape and the second with blank paper and a drawing utensil. The participant with the template was instructed to guide his or her partner to draw an exact duplicate of the shape on his or her sheet, using only verbal directions. After they were done, the participants compared the provided shape with what was drawn. 20 Minutes Communications exercise (Teampedia, 2015)
Developing a presentation Developing a presentation: A list of strategies from the prior session (based on the four diagrams) were compiled on flip chart paper. Participants divided back into their small groups, and were asked to choose one of the strategies to formulate a presentation, and given a worksheet to guide their development of the presentation. A facilitator worked with each group, and guided them through completing the worksheet to develop their presentation. The worksheet guided them through issues such as identifying their target audience for the presentation and highlighting important system structures like feedback loops that guided the choice of strategies. 2 Hours Materials created for project
Practicing a presentation Practicing presentations: A few additional community members were invited to the final part of the session, and the participants practiced their developed presentations to the whole group. They were allowed to ask questions of each other and discuss their ideas. Community members also provided feedback on youth’s presentations and ideas. 45 Minutes Materials created for project

Research team members and community partners co-facilitated the sessions. The session activities included adaptions of scripted1 group facilitation techniques that aimed to improve youth’s understanding of obesity. Youth were guided to produce visual diagrams that highlighted system structures related to obesity. Youth produced two types of systems diagrams: (a) graphs over time (Andersen & Richardson, 1997) and (b) causal loop diagrams (Sterman, 2000). Graphs over time are typically produced in the early stages of a project, with the goal of eliciting deep processing of past and potential future trends. After key system variables are identified, they are plotted on a two-dimensional graph (y-axis = the variable levels and x-axis = time). The graphs help identify patterns in how variables are changing in relation to each other and inform hypotheses about more complex system structures contributing to the issue. Causal loop diagrams are a visual representation of variables in a system and explicitly document how variables might produce observed trends. For example, arrows between variables are labeled with “same” or “opposite” to indicate the direction of relationships, and closed cycles of variables are labeled as reinforcing or balancing feedback loops, that is, where an initial variable propagates changes in one or more variables and ultimately reinforces or balances the initial variable’s change. To create their diagrams, youth watched short segments of two documentaries on obesity in the United States. Research team members facilitated a discussion about the data and evidence provided in the documentaries. Youth were guided to create their diagrams by critically considering the data and evidence as well as their own perceptions. With an expanded curriculum, the youth would inform their diagrams with more extensive use of existing scientific literature, surveillance data, and primary data collection.

The session facilitators reviewed the youth’s diagrams and corresponding verbal discussions to identify themes in dynamics and complex structures and guide development of four “seed diagrams.” The “seed diagrams” were used to help the youth consider potential solutions for addressing obesity in their community.

Phase 2: Engagement Strategy Pilot Testing and Evaluation

Participants

Community partners recruited participants for the sessions, inviting individuals who had been involved with their organizations’ programs. Eligible participants were African Americans between the ages of 13 and 17 years. We did not use inclusion/exclusion criteria based on weight or body mass index to enhance diversity in responses. Our target sample size was n = 20, which provided 80% power to detect a standardized mean difference of 0.68 standard deviations in survey outcomes at p < .05 (assuming 25% attrition).

Instrumentation

Participants completed a survey at baseline and immediately after each of the four sessions. We adapted2 the following subscales of a validated measure of youth psychological empowerment (Ozer & Schotland, 2011), each with 4-point Likert-type scale response options (1 = very untrue of me; 4 = very true of me). Participatory Behaviors: Five-item measure of youth’s experience with advocacy-related behaviors (α = .83); Sociopolitical Skills: Five-item measure of youth’s confidence in their ability to understand and address social and health issues in their community (α = .81); Motivation to Influence: Four-item measure of youth’s belief in the importance of engaging with others in the community to address health issues (α = .80).

The survey also included measures of causal attributions of obesity (Niederdeppe, Shapiro, & Porticella, 2011) and support for obesity policies and programs (Barry, Brescoll, Brownell, & Schlesinger, 2009). All items used 4-point Likert-type scale response options (1 = strongly disagree; 4 = strongly agree). The measures included three subscales that measure beliefs about the extent to which individual behavior (four items), biological (three items), and societal and environmental factors (seven items) cause obesity. Participants also rated their support for a list of recommended obesity prevention programs and policies including programmatic (three items), food access (five items), physical activity access (three items), advertising (two items), and food pricing (two items) strategies (Barry et al., 2009).

We interviewed seven youth postintervention with both high and low levels of participation. The interviews included semistructured questions to assess satisfaction and acceptability of the system diagramming activities (e.g., “Tell me about what activities were your favorite and what you liked about them”). Interviews were audio-recorded.

Analysis

Survey data were analyzed using SAS 9.3 (SAS Institute, Cary, NC). We used repeated measures linear regression mixed models to assess changes in all measures from baseline to the final postsession time-point. Subscale measures were summed and each was regressed on session, adjusting for gender and age. Qualitative data, including the youth diagrams and interviews, were analyzed following a qualitative descriptive approach (Sandelowski, 2000, 2010). Two members of the research team reviewed materials and generated categorical themes that most closely reflected the participant’s words and diagrams. The themes were discussed with additional research team members (including community partners) to establish dependability (Shenton, 2004).

Results

Twenty-one adolescent African American youths participated in at least one session. Nearly two thirds (61.9%, n = 13) completed all four sessions. Approximately two thirds of participants were girls (66.7%, n = 14). The mean (SD) age of participants was 16.3 (1.4) years. A greater proportion of girls completed all four sessions than boys (71.4%, n = 10 girls vs. 42.9%, n = 3 boys). The mean age did not differ between completers and noncompleters.

Diagramming Outcomes

Each youth produced at least one “graph over time.” Youth highlighted trends in variables from either an individual or community perspective, but there was a wide variation in the specific variables and patterns depicted. From a community perspective, one participant drew increasing “diabetes in teens” alongside decreasing “farmers’ markets with healthy foods” (Figure 2a). Another participant outlined trends in the healthfulness of one’s diet alongside one’s amount of money (Figure 2b).

Figure 2.

Figure 2

Example obesity behavior over time and causal loop diagrams that youth created.

Note. The figure provides two examples of behavior over time (Panels a and b) and causal loop diagrams (Panels c and d) that the youth created during the group model building sessions. Text in gray boxes is added where handwriting is difficult to read.

The youth participants also completed causal loop diagrams that indicated connections were being made between variables. Some participants made connections of stress and self-esteem with behavior or health outcomes (Figure 2c) and others identified connections between access and poor dietary choices (Figure 2d).

We identified four themes based on the diagrams: (a) connections among self-esteem, behaviors, and weight; (b) relative value of fast versus healthy foods; (c) school and parental influences; and (d) sugar-sweetened beverage marketing. Seed diagrams with the participants’ markings from solution identification sessions are available as supplementary materials (example shown in Figure 3). The youth identified and presented on proposed solutions including a health promotion summer camp led by adolescent youth and fielding a neighborhood petition to collect signatures of community members who would support healthier fast-food options in the community.

Figure 3.

Figure 3

Example seed causal loop diagram about obesity prevention cocreated with youth.

Note. The figure provides an example of the seed causal loop diagrams that was created based on youth’s behavior over time and connection circle exercises and presented to them for further expansion in a following session. The example includes the seed diagram and the additional markings provided by the youth. The diagram focused on feedback loops involved in shaping the relative value of “fast” versus healthy food.

Youth Participant Survey Outcomes

The youths’ mean (SD) of the research and action participatory behaviors scale significantly increased from 9.9 (3.8) at baseline to 14.5 (5.3) after the final session (p < .05; Table 2). Sociopolitical skills and motivation to influence, however, did not significantly change. Youths’ attributions of individual and biological factors as causes of obesity did not significantly change, but their attributions of environmental factors increased from 19.3 (3.3) to 22.5 (2.6; p < .05). Participants increased support for food access from 12.7 (1.9) to 14.7 (2.2; p < .05) and food pricing policies from 4.9 (0.9) to 6.1 (0.9; p < .001), but did not exhibit significant changes in support for advertising, programmatic, or physical activity access policies.

Table 2.

Mean (SD) of Survey Measures and Significance of Change From Baseline to Postworkshop Sessions.

Measure Scale range Baseline, M (SD) Post 1, M (SD) Post 2, M (SD) Post 3, M (SD) Post 4, M (SD) pa
Research and action self-efficacy
 Participatory behaviors 5–20 9.9 (3.8) 12.5 (3.9) 13.0 (4.1) 15.3 (4.1) 14.5 (5.3) <.0001*
 Sociopolitical skills 5–20 15.7 (2.1) 16.2 (2.2) 17.1 (2.1) 16.7 (2.0) 16.6 (3.1) .312
 Motivation to influence 4–16 13.5 (2.0) 13.0 (1.9) 13.6 (2.1) 13.9 (2.2) 14.1 (1.9) .329
Causal attributions
 Individual 5–20 11.9 (1.8) 11.5 (1.7) 10.9 (2.1) 11.1 (1.8) 11.3 (2.2) .694
 Biological 3–12 6.0 (1.89) 6.2 (1.5) 6.3 (1.4) 6.1 (1.6) 6.5 (1.4) .162
 Environmental 7–28 19.3 (3.3) 20.0 (3.7) 21.6 (3.4) 22.4 (2.7) 22.5 (2.6) .026*
Support policy and program strategies
 Programmatic 3–12 9.4 (1.3) 9.4 (1.9) 9.9 (1.4) 9.7 (1.5) 9.8 (1.8) .813
 Food access 5–20 12.7 (1.9) 13.4 (2.4) 14.2 (1.8) 13.8 (2.1) 14.7 (2.2) .031*
 Physical activity access 3–12 8.7 (2.0) 8.8 (2.1) 9.4 (1.5) 9.1 (1.7) 9.5 (1.8) .284
 Advertising policies 2–8 4.5 (1.0) 5.3 (1.3) 4.7 (1.5) 5.5 (1.4) 5.7 (1.3) .051
 Food pricing policies 2–8 4.9 (0.9) 4.9 (1.4) 5.9 (0.7) 5.8 (0.9) 6.1 (0.9) .001*
a

p Value is from repeated measures mixed regression models that controlled for age and gender.

*

p < .05.

Interview Satisfaction and Acceptability Outcomes

The youth participants had varying perspectives about the systems diagramming activities. One viewed them unfavorably, indicating it “felt like school.” In contrast, one noted the diagramming activities were his favorite because,

Once I got it, it was a whole lot that was going on through my head and I wanted to make my own chart, it overwhelmed me because and I finally knew how to put things in the correct spots.

The remaining found the diagramming activities acceptable, but indicated they needed more time because they were only beginning to understand the concepts when the session ended.

Discussion

In this study, we adapted and developed systems science engagement strategies that aimed to elicit rural African American youths’ perspectives on childhood obesity and improve their understanding of and support for addressing childhood obesity in their community. We found promising evidence that the activities can provide new insights about obesity and influence youth’s perceptions on causes of obesity and support for obesity prevention. These activities can be integrated into comprehensive obesity prevention and positive youth development curricula.

Similar to participatory systems science strategies that were conducted for the national Healthy Kids, Healthy Communities project (Brennan, Sabounchi, Kemner, & Hovmand, 2015), we found the activities created opportunities to characterize complex systems from the voice of community residents. In contrast, Healthy Kids, Healthy Communities primarily engaged adult stakeholders, whereas we gained youths’ unique perspectives about connections of multilevel factors. For example, youth indicated that patronizing a fast-food restaurant in the community was influenced by connections among environmental access and convenience, individual habituation, and social influences, which included the appeal of free WiFi that supported social media (Figure 3).

Obesity prevention studies that have employed participatory strategies with youth such as photovoice (Findholt, Michael, & Davis, 2011; Hannay, Dudley, Milan, & Leibovitz, 2013; Kramer, Schwartz, Cheadle, & Rauzon, 2013) and neighborhood mapping (Hume, Salmon, & Ball, 2005; Topmiller, Jacquez, Vissman, Raleigh, & Miller-Francis, 2015) also revealed insights about socioenvironmental influences on obesity such as underutilization of neighborhood parks (Findholt et al., 2011) and gang activity (Kramer et al., 2010). Compared with the former participatory strategies, systems science provides a unique opportunity to elucidate links between multilevel factors. The links are valuable for designing solutions that create mutually reinforcing conditions.

The youth in our study exhibited a statistically significant increase in research and action participatory behaviors, but not motivation to influence or sociopolitical skills. One youth empowerment study found modest significant improvements in these same measures following a 1-year program (Ozer & Douglas, 2013). Our participants appeared to have high levels of motivation to influence and sociopolitical skills from the outset, which likely contributed to the lack of change. In addition, our study was developmental and the intensity of sessions was low. A long-term study with additional sessions may be required for assessment of greater impact.

We found a significant change in youths’ perceptions of causal attributions of obesity, with an increase in environmental attributions. The youths’ support for obesity prevention policies and programs, specifically food access and food pricing policies, also significantly increased. Other participatory systems science studies indicate the activities can change adult participant’s perceptions about an issue and lead to more support for policy changes (Rouwette, Vennix, & van Mullekom, 2002). Systems science strategies have been used in education to improve general systems thinking skills (LaVigne, 2009); however, to our knowledge, our study is the first to evaluate the strategies for their potential to shape youth perceptions of and support for obesity prevention. The changes we observed have important implications for youth interest and willingness to advocate for social and environmental changes in their community. Youth participate in our curriculum may be better equipped and more likely to engage in community efforts to address childhood obesity and less resistant to potential environmental or policy changes (e.g., changes to school meals).

In general, the activities were acceptable to the participants. However, some of the diagramming concepts appeared challenging for the youth such as identifying the direction of relationships between variables and feedback loops. The facilitators found the allotted time did not allow for the attention needed to fully guide youth through these concepts. Increasing time for diagramming, the number of trained facilitators, or using a group format that facilitates youth to learn from each other should be considered for future implementation of the strategies.

Our findings should be considered in the context of the study limitations. We used a convenience and snowball recruitment approach and tested our curriculum with a small sample. We did not perform rigorous member checking of our qualitative analysis findings with youth, which would enhance the trustworthiness of our findings. However, we did have informal conversations with several participants who indicated agreement with our synthesis. Presurveys indicated our participants had high baseline levels of motivation to influence and sociopolitical skills, which indicates a potential selection bias of more civically minded youth. It is unclear whether the engagement activities would have similar outcomes with less civically minded youth, and without a comparison group we are limited in our ability to attribute changes to the systems science activities. The activities were also adapted and completed in a small rural African American community, which may limit generalizability to other populations. Importantly, we used a community-engaged approach to develop and adapt the strategies, which improved local relevance.

Future studies are needed that incorporate systems science participatory strategies within larger positive youth development programs. For example, we aim to continue to engage adolescent youth in this community to quantitatively research the qualitative links we explored (e.g., Figure 3), and to help them implement and evaluate multilevel solutions. In addition, as with our study, adolescents are often the focus of positive youth development efforts, but adaptations for younger children should be considered. Overall, more rigorous evaluation including quasi-experimental or cluster-randomized trials should be conducted to compare our systems science curricula with attention-control or didactic obesity prevention curricula. Evaluation measures should include individual participant (e.g., diet and physical activity behaviors) and community (e.g., policy enactment, community readiness) outcomes.

The most positive impacts on obesity prevention are not likely reaching communities with the highest prevalence and disparities (Yancey et al., 2004). More efforts need to help communities mobilize support and resources to address this complex issue. Our strategies have promise to help facilitate a systems thinking perspective, and provide a unique value of engaging young community members during critical development periods.

Acknowledgments

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The project described was supported by the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health, through Grant Award Number UL1TR001111. It was also supported in part by the National Institutes of Health under Award Numbers R01AG047869 and K24HL105493. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnotes

1

“Scripts” provide fine-grained descriptions of small group exercises that can be applied across topics, and include specific purposes of each exercise, preliminary preparations, materials needed, time required, and instructions regarding facilitative behaviors.

2

We removed items about settings not relevant to the project. For example, the scales included multiple items around the same factor applied to different settings (“If I want to improve a problem at school, I know how to … ” and “If I want to improve a problem in my city, I know how to …”). We adapted to ask these items once—referring to “in my community” in the stem.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

  1. ACT for Youth. Positive youth development resource manual. 2016 Retrieved from http://www.actforyouth.net/publications/manual.cfm.
  2. Andersen DF, Richardson GP. Scripts for group model building. System Dynamics Review. 1997;13:107–129. [Google Scholar]
  3. Arredondo E, Mueller K, Mejia E, Rovira-Oswalder T, Richardson D, Hoos T. Advocating for environmental changes to increase access to parks: Engaging promotoras and youth leaders. Health Promotion Practice. 2013;14:759–766. doi: 10.1177/1524839912473303. [DOI] [PubMed] [Google Scholar]
  4. Barry CL, Brescoll VL, Brownell KD, Schlesinger M. Obesity metaphors: How beliefs about the causes of obesity affect support for public policy. Milbank Quarterly. 2009;87:7–47. doi: 10.1111/j.1468-0009.2009.00546.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Boelsen-Robinson T, Peeters A, Beauchamp A, Chung A, Gearon E, Backholer K. A systematic review of the effectiveness of whole-of-community interventions by socioeconomic position. Obesity Reviews. 2015;16:806–816. doi: 10.1111/obr.12297. [DOI] [PubMed] [Google Scholar]
  6. Bogart LM, Cowgill BO, Elliott MN, Klein DJ, Hawes-Dawson J, Uyeda K, … Schuster MA. A randomized controlled trial of students for nutrition and exercise: A community-based participatory research study. Journal of Adolescent Health. 2014;55:415–422. doi: 10.1016/j.jadohealth.2014.03.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Brennan LK, Sabounchi NS, Kemner AL, Hovmand P. Systems thinking in 49 communities related to healthy eating, active living, and childhood obesity. Journal of Public Health Management & Practice. 2015;21(Suppl 3):S55–S69. doi: 10.1097/PHH.0000000000000248. [DOI] [PubMed] [Google Scholar]
  8. Catalano RF, Berglund ML, Ryan JA, Lonczak HS, Hawkins JD. Positive youth development in the United States: Research findings on evaluations of positive youth development programs. Annals of the American Academy of Political and Social Science. 2004;591:98–124. [Google Scholar]
  9. Corbie-Smith G, Adimora AA, Youmans S, Muhammad M, Blumenthal C, Ellison A, … Lloyd SW. Project GRACE: A staged approach to development of a community-academic partnership to address HIV in rural African American communities. Health Promotion Practice. 2011;12:293–302. doi: 10.1177/1524839909348766. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Corbie-Smith G, Bryant AR, Walker DJ, Blumenthal C, Council B, Courtney D, Adimora A. Building capacity in community-based participatory research partnerships through a focus on process and multiculturalism. Progress in Community Health Partnership. 2015;9:261–273. doi: 10.1353/cpr.2015.0038. [DOI] [PubMed] [Google Scholar]
  11. Cummins S, Flint E, Matthews SA. New neighborhood grocery store increased awareness of food access but did not alter dietary habits or obesity. Health Affairs. 2014;33:283–291. doi: 10.1377/hlthaff.2013.0512. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Devi S. Progress on childhood obesity patchy in the USA. Lancet. 2008;371:105–106. doi: 10.1016/S0140-6736(08)60088-4. [DOI] [PubMed] [Google Scholar]
  13. Dzewaltowski DA, Estabrooks PA, Welk G, Hill J, Milliken G, Karteroliotis K, Johnston JA. Healthy Youth Places: A randomized controlled trial to determine the effectiveness of facilitating adult and youth leaders to promote physical activity and fruit and vegetable consumption in middle schools. Health Education & Behavior. 2009;36:583–600. doi: 10.1177/1090198108314619. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Eskicioglu P, Halas J, Senechal M, Wood L, McKay E, Villeneuve S, … McGavock JM. Peer mentoring for type 2 diabetes prevention in first nations children. Pediatrics. 2014;133:e1624–e1631. doi: 10.1542/peds.2013-2621. [DOI] [PubMed] [Google Scholar]
  15. Findholt NE, Michael YL, Davis MM. Photovoice engages rural youth in childhood obesity prevention. Public Health Nursing. 2011;28:186–192. doi: 10.1111/j.1525-1446.2010.00895.x. [DOI] [PubMed] [Google Scholar]
  16. Forneris T, Fries E, Meyer A, Buzzard M, Uguy S, Ramakrishnan R, … Danish S. Results of a rural school-based peer-led intervention for youth: Goals for health. Journal of School Health. 2010;80:57–65. doi: 10.1111/j.1746-1561.2009.00466.x. [DOI] [PubMed] [Google Scholar]
  17. Frerichs L, Brittin J, Robbins R, Steenson S, Stewart C, Fisher C, Huang TT. SaludABLEOmaha: Improving readiness to address obesity through healthy lifestyle in a Midwestern Latino community, 2011–2013. Preventing Chronic Disease. 2015;12:E20. doi: 10.5888/pcd12.140328. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Frerichs L, Brittin J, Stewart C, Robbins R, Riggs C, Mayberger S, … Huang TT. SaludableOmaha: Development of a youth advocacy initiative to increase community readiness for obesity prevention, 2011–2012. Preventing Chronic Disease. 2012;9:E173. doi: 10.5888/pcd9.120095. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Frerichs L, Lich KH, Dave G, Corbie-Smith G. Integrating systems science and community-based participatory research to achieve health equity. American Journal of Public Health. 2016 doi: 10.2105/AJPH.2015.302944. Advance online publication. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hannay J, Dudley R, Milan S, Leibovitz PK. Combining photovoice and focus groups: Engaging Latina teens in community assessment. American Journal of Preventive Medicine. 2013;44(Suppl 3):S215–S224. doi: 10.1016/j.amepre.2012.11.011. [DOI] [PubMed] [Google Scholar]
  21. Hovmand PS. Community based system dynamics. New York, NY: Springer; 2014. [Google Scholar]
  22. Hovmand PS, Rouwette E, Andersen DF, Richardson GP, Kraus A. Scriptapedia 4.0.6. 2013 Retrieved from http://tools.systemdynamics.org/scrpda/scriptapedia_4.0.6.pdf.
  23. Huang TT, Brownson R, Esposito L, Green L, Homer C. Next steps in obesity prevention: Applying the systems approach. Child Obesity. 2013;9:11–14. doi: 10.1089/chi.2013.9102. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Huang TT, Cawley JH, Ashe M, Costa SA, Frerichs LM, Zwicker L, … Kumanyika SK. Mobilisation of public support for policy actions to prevent obesity. Lancet. 2015;385:2422–2431. doi: 10.1016/S0140-6736(14)61743-8. [DOI] [PubMed] [Google Scholar]
  25. Hume C, Salmon J, Ball K. Children’s perceptions of their home and neighborhood environments, and their association with objectively measured physical activity: A qualitative and quantitative study. Health Education Research. 2005;20:1–13. doi: 10.1093/her/cyg095. [DOI] [PubMed] [Google Scholar]
  26. Institute of Medicine. Accelerating progress in obesity prevention: Solving the weight of the nation. Washington, DC: National Academies Press; 2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. John W Gardner Center for Youth and their Communities. Youth engaged in leadership and learning. 2007 Retrieved from http://gardnercenter.stanford.edu/our_work/yell.html.
  28. Jones S, Spence M, Hardin S, Clemente N, Schoch A. Youth can! Results of a pilot trial to improve the school food environment. Journal of Nutrition Education and Behavior. 2011;43:284–287. doi: 10.1016/j.jneb.2010.10.005. [DOI] [PubMed] [Google Scholar]
  29. Kraak VI, Story M. A public health perspective on healthy lifestyles and public–private partnerships for global childhood obesity prevention. Journal of the American Dietetic Association. 2010;110:192–200. doi: 10.1016/j.jada.2009.10.036. [DOI] [PubMed] [Google Scholar]
  30. Kramer L, Schwartz P, Cheadle A, Borton JE, Wright M, Chase C, Lindley C. Promoting policy and environmental change using photovoice in the Kaiser Permanente Community Health Initiative. Health Promotion Practice. 2010;11:332–339. doi: 10.1177/1524839909341555. [DOI] [PubMed] [Google Scholar]
  31. Kramer L, Schwartz P, Cheadle A, Rauzon S. Using photovoice as a participatory evaluation tool in Kaiser Permanente’s Community Health Initiative. Health Promotion Practice. 2013;14:686–694. doi: 10.1177/1524839912463232. [DOI] [PubMed] [Google Scholar]
  32. Lapalme J, Bisset S, Potvin L. Role of context in evaluating neighbourhood interventions promoting positive youth development: A narrative systematic review. International Journal of Public Health. 2014;59:31–42. doi: 10.1007/s00038-013-0449-2. [DOI] [PubMed] [Google Scholar]
  33. LaVigne A. Systems thinking and dynamic modeling within K-12 schools: Effects on student learning. 2009 Retrieved from http://www.ppi-int.com/newsletter/SyEN-012.php#article.
  34. Lich KH, Ginexi EM, Osgood ND, Mabry PL. A call to address complexity in prevention science research. Prevention Science. 2013;14:279–289. doi: 10.1007/s11121-012-0285-2. [DOI] [PubMed] [Google Scholar]
  35. Linton LS, Edwards CC, Woodruff SI, Millstein RA, Moder C. Youth advocacy as a tool for environmental and policy changes that support physical activity and nutrition: An evaluation study in San Diego County. Preventing Chronic Disease. 2014;11:E46. doi: 10.5888/pcd11.130321. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Millstein RA, Sallis JF. Youth advocacy for obesity prevention: The next wave of social change for health. Translational Behavioral Medicine. 2011;1:497–505. doi: 10.1007/s13142-011-0060-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Niederdeppe J, Porticella N, Shapiro MA. Using theory to identify beliefs associated with support for policies to raise the price of high-fat and high-sugar foods. Journal of Health Communication. 2012;17:90–104. doi: 10.1080/10810730.2011.585694. [DOI] [PubMed] [Google Scholar]
  38. Niederdeppe J, Shapiro MA, Kim HK, Bartolo D, Porticella N. Narrative persuasion, causality, complex integration, and support for obesity policy. Health Communication. 2014;29:431–444. doi: 10.1080/10410236.2012.761805. [DOI] [PubMed] [Google Scholar]
  39. Niederdeppe J, Shapiro MA, Porticella N. Attributions of responsibility for obesity: Narrative communication reduces reactive counterarguing among liberals. Human Communication Research. 2011;37:295–323. [Google Scholar]
  40. North Carolina Department of Health and Human Services. The burden of obesity in North Carolina. 2009 Retrieved from http://www.eatsmartmovemorenc.com/ObesityInNC/Texts/OBESITY_BURDEN_2009_WEB.pdf.
  41. Ozer EJ, Douglas L. The impact of participatory research on urban teens: An experimental evaluation. American Journal of Community Psychology. 2013;51:66–75. doi: 10.1007/s10464-012-9546-2. [DOI] [PubMed] [Google Scholar]
  42. Ozer EJ, Schotland M. Psychological empowerment among urban youth: Measure development and relationship to psychosocial functioning. Health Education & Behavior. 2011;38:348–356. doi: 10.1177/1090198110373734. [DOI] [PubMed] [Google Scholar]
  43. Pitts SBJ, Smith TW, Thayer LM, Drobka S, Miller C, Keyserling TC, Ammerman AS. Addressing rural health disparities through policy change in the stroke belt. Journal of Public Health Management & Practice. 2013;19:503–510. doi: 10.1097/PHH.0b013e3182893bbb. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Quaden R, Ticotsky A, Lyneis D, Walker N. Do you want fries with that? Learning about connection circles. 2006 Retrieved from http://static.clexchange.org/ftp/documents/x-curricular/CC2010-11Shape10FriesConnexnSF.pdf.
  45. Roberto CA, Swinburn B, Hawkes C, Huang TT, Costa SA, Ashe M, … Brownell KD. Patchy progress on obesity prevention: Emerging examples, entrenched barriers, and new thinking. Lancet. 2015;385:2400–2409. doi: 10.1016/S0140-6736(14)61744-X. [DOI] [PubMed] [Google Scholar]
  46. Rouwette EA, Vennix J, Felling A. On evaluating the performance of problem structuring methods: An attempt at formulating a conceptual model. Group Decision and Negotiation. 2009;18:567–587. [Google Scholar]
  47. Rouwette EA, Vennix JA, van Mullekom T. Group model building effectiveness: A review of assessment studies. System Dynamics Review. 2002;18:5–45. [Google Scholar]
  48. Sandelowski M. Whatever happened to qualitative description? Research in Nursing & Health. 2000;23:334–340. doi: 10.1002/1098-240x(200008)23:4<334::aid-nur9>3.0.co;2-g. [DOI] [PubMed] [Google Scholar]
  49. Sandelowski M. What’s in a name? Qualitative description revisited. Research in Nursing & Health. 2010;33:77–84. doi: 10.1002/nur.20362. [DOI] [PubMed] [Google Scholar]
  50. Santos RG, Durksen A, Rabbanni R, Chanoine JP, Lamboo Miln A, Mayer T, McGavock JM. Effectiveness of peer-based healthy living lesson plans on anthropometric measures and physical activity in elementary school students: A cluster randomized trial. JAMA Pediatrics. 2014;168:330–337. doi: 10.1001/jamapediatrics.2013.3688. [DOI] [PubMed] [Google Scholar]
  51. Shenton AK. Strategies for ensuring trustworthiness in qualitative research projects. Education for Information. 2004;22:63–75. doi: 10.3233/EFI-2004-22201. [DOI] [Google Scholar]
  52. Skinner A, Skelton J. Prevalence and trends in obesity and severe obesity among children in the United States, 1999–2012. JAMA Pediatrics. 2014;168:561–566. doi: 10.1001/jamapediatrics.2014.21. [DOI] [PubMed] [Google Scholar]
  53. Sliwa S, Goldberg JP, Clark V, Collins J, Edwards R, Hyatt RR, … Economos CD. Using the community readiness model to select communities for a community-wide obesity prevention intervention. Preventing Chronic Disease. 2011;8:A150. [PMC free article] [PubMed] [Google Scholar]
  54. Smith LH, Holloman C. Comparing the effects of teen mentors to adult teachers on child lifestyle behaviors and health outcomes in Appalachia. Journal of School Nursing. 2013;29:386–396. doi: 10.1177/1059840512472708. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Sterman JD. Business dynamics: Systems thinking and modeling for a complex world. Boston, MA: McGraw-Hill; 2000. Causal loop diagrams; pp. 137–190. [Google Scholar]
  56. Story M, Kaphingst KM, Robinson-O’Brien R, Glanz K. Creating healthy food and eating environments: Policy and environmental approaches. Annual Review of Public Health. 2008;29:253–272. doi: 10.1146/annurev.publhealth.29.020907.090926. [DOI] [PubMed] [Google Scholar]
  57. Teampedia. Back-2-back drawing. 2015 Retrieved from http://www.teampedia.net/wiki/index.php?title=Back-2-Back_Drawing.
  58. Topmiller M, Jacquez F, Vissman AT, Raleigh K, Miller-Francis J. Partnering with youth to map their neighborhood environments: A multilayered GIS approach. Family and Community Health. 2015;38:66–76. doi: 10.1097/FCH.0000000000000050. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. U.S. Census Bureau. 2010 Census of population and housing. 2010 Retrieved from https://www.census.gov/prod/cen2010/cph-2-35.pdf.
  60. Utter J, Scragg R, Robinson E, Warbrick J, Faeamani G, Foroughian S, … Swinburn BA. Evaluation of the Living 4 Life project: A youth-led, school-based obesity prevention study. Obesity Reviews. 2011;12(Suppl 2):51–60. doi: 10.1111/j.1467-789X.2011.00905.x. [DOI] [PubMed] [Google Scholar]
  61. Vennix JA. Group model-building: Tackling messy problems. System Dynamics Review. 1999;15:379–401. [Google Scholar]
  62. Voinov A, Bousquet F. Modelling with stakeholders. Environmental Modelling & Software. 2010;25:1268–1281. [Google Scholar]
  63. Wallerstein N, Sanchez-Merki V. Freirian praxis in health education: Research results from an adolescent prevention program. Health Education Research. 1994;9:105–118. doi: 10.1093/her/9.1.105. [DOI] [PubMed] [Google Scholar]
  64. Wang C, Gortmaker SL, Taveras EM. Trends and racial/ethnic disparities in severe obesity among US children and adolescents, 1976–2006. International Journal of Pediatric Obesity. 2011;6:12–20. doi: 10.3109/17477161003587774. [DOI] [PubMed] [Google Scholar]
  65. Waters Foundation. The Bean game: Exploring Human interaction with natural resources. 2016 Retrieved from http://watersfoundation.org/resources/the-bean-game-exploring-human-interactions-with-natural-resources/
  66. Weiner B. Social motivation, justice, and the moral emotions: An attributional approach. Mahwah, NJ: Lawrence Erlbaum; 2006. [Google Scholar]
  67. Yancey AK, Kumanyika SK, Ponce NA, McCarthy WJ, Fielding JE, Leslie JP, Akbar J. Population-based interventions engaging communities of color in healthy eating and active living: A review. Preventing Chronic Disease. 2004;1:A09. [PMC free article] [PubMed] [Google Scholar]
  68. Youth Empowered Solutions. Creating community change. 2016 Retrieved from http://www.youthempoweredsolutions.org/

RESOURCES