Effects of Positive Action on the Emotional Health of Urban Youth: A Cluster-Randomized Trial

Abstract

Purpose

We examined the effects of Positive Action (PA), a school-based social-emotional learning and health promotion program, on the emotional health of predominately low-income and ethnic minority urban youth.

Methods

The study was a matched-pair, cluster-randomized controlled trial involving 14 Chicago public schools. Outcomes were assessed over a 6-year period of program implementation for a cohort of youth in each school, followed from grades 3 to 8. Youth reported on their emotional health (positive affect, life satisfaction, depression, anxiety) and social-emotional and character development. Growth-curve and structural-equation modeling analyses assessed overall program effects on the emotional health outcomes as well as mediation of these effects via the program’s impact on youths’ social-emotional and character development.

Results

Students in PA schools, compared to those in control schools, had more favorable change over the course of the study in positive affect (standardized mean difference effect size [ES] = .17) and life satisfaction (ES = .13) as well as significantly lower depression (ES = −.14) and anxiety (ES = −.26) at study end point. Program effects for positive affect, depression, and anxiety were mediated by more favorable change over time in social-emotional and character development for students in PA schools.

Conclusions

Results suggest that universal, school-based programs can benefit the emotional health of youth in low-income, urban settings. The modest magnitude of effects over an extended period of program implementation, however, reflects the challenges of both mounting interventions and offsetting formidable risks for mental health problems in such environments.

Introduction

Youth who grow up in low-income urban environments are at increased risk for adverse emotional health outcomes such as depression and anxiety.¹ Youth from lower socioeconomic backgrounds also fare more poorly on positive indicators of psychological well-being.² Underscoring the importance of these findings is evidence that both positive affect and life satisfaction can mitigate negative effects of stressful events and protect against the development of psychological and behavioral problems.³

School-based interventions have the potential to be an effective tool for improving the emotional health of youth in low-income, urban communities.⁴ Research suggests a reciprocal relationship between academics and the schooling environment and emotional health,⁵ such that academic challenges (e.g., poor performance) can lead to emotional and behavioral problems, and similarly, that emotional difficulties can impede the ability to learn. Additionally, although youth in these environments are most in need of mental health services, they are least likely to receive them, which makes schools the ideal setting for programming that can address emotional health. ⁴ These relationships, along with school being a common place to address challenges and issues in childhood, underscore the need to examine emotional health within the school context.

Several research gaps, however, remain to be clarified. First, the effectiveness of school-based programs for strengthening positive dimensions of emotional health (e.g., positive affect) is unclear. Second, the optimal scope of interventions requires further investigation. Despite their theoretical promise,⁶ a recent systematic review did not find evidence that school-wide programs (i.e., those delivered in the entire school) were effective for strengthening emotional health outcomes.⁷ Such programs often are relatively complex,⁷ thus potentially compromising quality of implementation and effectiveness.⁸ This consideration may be especially important within low-income, urban schools where fewer resources are generally available to support high fidelity implementation of programs.⁴ A related challenge with methodological implications for low-income, urban schools is that rates of student mobility in and out of schools is often high, making it important to examine whether any observed differences in youth outcomes between conditions are attributable to the intervention as opposed to differential characteristics of students who leave or enter treatment and control schools during the study⁹. Addressing this issue necessitates the examination of how differences in outcomes across treatment and control schools may vary across groups of students with similar mobility patterns. One final gap is that few studies have examined the mechanisms (i.e., mediation) that may account for effects of school-based programs on emotional health outcomes. Both psychological (e.g., self-esteem) and interpersonal (e.g., social support) factors are important contributors to emotional health among children and adolescents.¹⁰ As such, the development of social-emotional and character development (SECD¹¹; e.g., self-control, honesty, respect) that is likely to foster such resources could be one significant mechanism accounting for effects of programs on emotional health outcomes. To our knowledge, however, this possibility has not yet been tested.

To examine the mediating role of SECD on emotional health outcomes of students in low income, urban communities, we utilized data from the Chicago trial of Positive Action (PA). PA is a comprehensive, school-wide social-emotional learning and health promotion (SEL-HP) program grounded in theories of self-concept¹² particularly Self-esteem Enhancement Theory (SET)¹³. SET assumes that the desire to feel good about oneself is a universal need and that individuals will use a range of cognitive, affective, and behavioral strategies to help acquire and sustain feelings of worth. Positive outcomes are expected to be facilitated when people are adequately prepared and supported in satisfying their motivation for self-esteem through adaptive beliefs, values, and actions. In line with SET, PA includes a classroom-based curriculum that introduces the motivation to feel good about oneself, while teaching the skills (e.g., self-control, prosocial behavior) needed to act on this motivation in ways that are adaptive for self and others. A range of ecological supports (e.g., school climate development) also provide social reinforcement and validation for positive behaviors to encourage them in both school and non-school settings. All program components are organized around six core concepts: self-concept, positive actions for body and mind, positive actions focusing on getting along with others, and managing, being honest with, and continually improving oneself. Through promotion of feelings of self-worth, a well-established contributor to various facets of mental health (e.g., depression¹⁴) and development of skills and behaviors that are important for both sustaining rewarding social relationships and achieving personal success in areas such as school, the PA program is designed to create a foundation for positive emotional health.

The classroom component of the PA program includes 140 15-20 minute, age-appropriate and sequenced lessons per grade taught 4 days per week for grades K-6 and 70 20 minute lessons taught 2 days per week for grades 7 and 8. Other components include teacher, counselor and family training and school-wide climate development (i.e., emphasizing positive actions around the entire school with posters, assemblies, etc.). Experimental evaluations support the effectiveness of PA for improving academic and behavioral outcomes^15–18 as well as the school environment.¹⁹

To date, program effects on the emotional health of students have not been assessed. We hypothesized that, as compared to students in non-PA implementing schools: 1) students attending schools implementing PA would report more favorable emotional health in the areas of positive affect, life satisfaction, depression, and anxiety, and 2) the effects of PA on these outcomes would be mediated by relative improvements in skills and behaviors supportive of SECD. To test these hypotheses, we utilized data from the Chicago trial of PA, which included eight waves of data over six academic years, allowing us to test program effects longitudinally.

Method

Schools participating in the study were drawn from 483 K-6 and K-8 Chicago Public Schools (CPS). Sixty-eight schools met eligibility criteria (see Figure 1²⁰), of which 18 agreed to participate.²¹ Funding allowed for participation of only the 7 best-matched pairs. Seven schools were randomly assigned to PA, and 7 assigned to a control condition (business as usual). T-tests showed that PA and control schools did not differ significantly on any of the matching variables,^{20, 22} nor did the 7 pairs of schools differ significantly from the remainder of the 68 schools eligible for the study.^21,23

The trial was longitudinal with a place-focused, intent-to-treat design at the cohort level.⁹ Specifically, a cohort of students in the seven matched pairs of schools who were in grade 3 at the start of the study was assessed at study baseline (Fall 2004) and then at seven additional times (waves) over six years: Spring 2005, Fall 2005, Spring 2006, Spring 2007, Fall 2008, Spring 2009, and Spring 2010 (end of grade 8). Consistent with the study design, data were collected at each time point from all consented students in the study cohort, including those who had entered the schools since the start of the study.²⁴

Parental consent and student assent was obtained before students completed surveys; all students were consented and assented upon study entry and then re-consented and assented at Wave 6 for the second phase of funding. Parental consent for study participation was obtained for 79% of students at baseline, with consent rates ranging from 58% to 84% for students entering at later waves of the study. The total number of students enrolled in the study across all eight waves (i.e., the number with data available for at least one time point) was 1,170, of whom 53% were female; 48% were African American, 27% Hispanic and 19% other (i.e., White, Asian, Native American, and “Other”). Of the original 624 consented students in grade 3 at the beginning of the trial, 131 (i.e. 21%) remained at wave 8, reflecting the high school mobility of low-income urban students.²⁵ The average number of waves/years of participation was 3.1. The research was approved by the Institutional Review Boards at the University of Illinois at Chicago and Oregon State University.

In general, there was wide variability between schools in implementation indices (e.g., teacher description of amount and quality of PA activities in the classroom), especially in early years, with improvements over time. By the end of year 6, one school was implementing at only a moderate level, three at a moderate to high level, and three at high levels.²⁶ Students in PA schools also reported their overall satisfaction with the program at each wave. The mean rating on a 4-point scale ranged from 2.88 to 3.56 across the different waves of the trial; students tended to report somewhat lower levels of satisfaction at later waves, perhaps reflecting a more general developmental trend toward critical appraisals as youth transitioned to adolescence.

Measures

All measures were student self-report and were collected at all eight waves unless noted. We report alpha coefficients of internal consistency. We report intraclass correlations (ICCs) or median incident risk ratios (MIRRs) as indicators of variation across schools (for first measurement only) or variation across students within schools (across time points). The MIRR is more appropriate than ICC for outcomes measured as counts or rates and thus is reported for measures of this type.²⁷ An MIRR of 1.0 indicates no variation at a given level. Except where noted, each outcome was represented as the average of responses to the relevant set of items. The intended age ranges for all scales were consistent with the ages at which students were assessed and thus developmentally appropriate.

Positive affect

Positive affect was measured using a modified 6-item version of the Positive and Negative Affect Scale for Children (PANAS).²⁸ Students reported the extent to which they had experienced each type of feeling (e.g., excited, happy) in the last 2 weeks using a 4-point scale ranging from “None of the time” (1) to “All of the time” (4) (α range = .70 - .87 across time points; ICCs at the school level and across students of .02 and .32, respectively).

Life satisfaction

Life satisfaction was measured using a modified version of the Student Life Satisfaction Scale²⁹ that consisted of 3 items: “My life is just right”, “I have a good life”, and “I have what I want in life”. Students indicated how much they agreed with each statement on a 4-point scale ranging from “NO!” (1) to “YES!” (4) (α range = .71 - .84; ICCs of ≈ .00 and .30, respectively).

Depression and anxiety

Depression and anxiety were assessed using 12 items (6 per scale) from the Behavior Assessment System for Children.³⁰ Students were asked to respond either “true” or “false” to each item (example items for depression and anxiety, respectively, include “I feel depressed” and “I often worry about something bad happening to me”, respectively; for depression: α range = .70 - .79; MIRRs of 1.17 and 2.63, respectively; for anxiety: α range = .75- .81; MIRRs of 1.11 and 1.59, respectively). These outcomes were assessed only at Waves 5 through 8, and modeled as a count of endorsed items.

Social-emotional and character development

SECD was measured using the 28-item Social-Emotional and Character Development Scale (SECDS).^31,32 Students indicated how often they demonstrated each SECD-related skill or behavior, including honesty, self-control, prosocial interactions, self-development and respect, on a 4-point scale ranging from “None of the time” (1) to “All of the time” (4). Example: “I try to cheer up other kids if they are feeling sad”, “I apologize when I have done something wrong”, and “I keep my temper when I have an argument with other kids” (α range = .88 - .92; ICCs of .04 and .36, respectively).

Data Analysis

Primary study analyses were conducted using Mplus version 6.12. We first investigated overall program effects on each emotional health outcome. Since data were available from all waves for positive affect and life satisfaction, we tested the effects of condition (PA versus control) on change over time (i.e., slope) for these outcomes using growth curve modeling. For all outcomes, models were fit for waves of measurement within students within schools. As noted previously, depression and anxiety were assessed only starting at Wave 5; thus, the intercept for these outcomes was set at Wave 8, providing a test of PA versus control schools on the measure at the end of the study. To provide for a baseline control for these variables, a school-level average of student-reported levels of negative affect (PANAS) from Wave 1 (centered around the mean for all schools) was utilized as a covariate in all models that tested for program effects on depression and anxiety. Quadratic terms were tested and dropped for parsimony if non-significant.

Next, analyses tested for mediation of program effects by SECD using a structural equation model (SEM) approach. This involved testing a model that decomposed effects on outcomes into direct effects of PA on the outcomes and indirect effects via the program’s effects on growth/change over time (i.e., slope) of SECD.³³ Mediation was classified using the mediation analysis decision tree discussed by Zhao and colleagues.³⁴ Because preliminary analyses on SECD revealed a quadratic trend in change over time, scores on the measure were centered at the sample mean at each wave in order to effectively eliminate the need to model a quadratic trend and thus facilitate model interpretation.

Distributions of outcome variables were non-normal (negatively skewed for positive affect and life satisfaction, and positively skewed for depression and anxiety) so we employed bootstrap estimation with 1,000 re-samples.³⁵ Random coefficient models were estimated for all outcome variables (with the exception of life satisfaction which was a random intercept model). For the random coefficient models, slope variances were statistically significant. For all analyses, missing values were handled using full information maximum likelihood estimation. Effect sizes (ES) were calculated as standardized mean differences at study end-point using model-predicted means/counts and observed standard deviations.³⁶

Supplementary analyses

The primary analyses conducted with Mplus did not take into account clustering of data within schools. The nature of the mediation model (i.e., the use of a longitudinal mediator and non-normally distributed outcomes) necessitated the use of a program with these statistical capabilities (such as Mplus); however, the mediation models would not converge with the clustering variable included. As a sensitivity analysis relevant to this issue, we used Stata version 12 to also test the overall (primary) effects of PA on emotional health outcomes within three-level (occasion of measurement within student within school) growth curve models that did account for within-school clustering of data.

Student mobility was represented using results from a latent class analysis in which a 5-class solution (i.e., 1) stayers (average study duration of 5.72 years, N = 158), 2) temporary participants (1.30 years, only in grades 4 or 5; N=196), 3) late joiners (1.38 years; N=308); 4) early leavers (0.94 years; N=263), and 5) late leavers (3.23 years; N=287) was found to be the most appropriate fit for the data ²³ Analyses tested for a possible moderating effect of student mobility (class) on program effects (condition x time) for all outcomes (i.e., condition×time×class).

Results

Program Effects

Findings for analyses of overall program effects on emotional health outcomes are shown in Table 1. There was a trend for students in PA to have more favorable change in reported levels of positive affect as indicated by a marginally significant condition × time interaction (b = .02, P <.10; ES = .17). Although there was a general decline in positive affect over time, this decline was less pronounced among students in PA schools. There were significant linear and quadratic interactions of condition × time for life satisfaction (condition × time b = .10 P <.01; condition × time²b = −.02, P <.05), the net result of which was a notable difference at study endpoint (ES = .13) that favored students in PA schools. Additionally, students in PA schools reported significantly fewer symptoms of depression (b = −0.23, P <.05; ES = −.14) and anxiety (b = −.53, P <.001; ES = −.26) at endpoint than students in control schools. Sensitivity analyses using Stata supported these findings. There was no moderation of program effects by mobility for any measure.

Table 1:

Effects of Positive Action on Positive Affect, Life Satisfaction, Depression, and Anxiety as Mediated by Change Over Time in Social-emotional and character development (SECD) in structural equation modeling Analyses (N=1,170).

Effects	b	SE	CI
Positive Affect (PosAff)a
PA intervention → SECD	0.04***	0.01	0.02, 0.06
SECD → PosAff	0.94***	0.20	0.55, 1.33
PA intervention → PosAff	−0.01	0.01	−0.03, 0.01
Indirect Effect
PA → SECD → PosAff	0.03**	0.01	0.01, 0.05
Total Effect	0.02†	0.01	0.00, 0.04
Life Satisfaction (LifeSat)a
PA intervention → SECD	0.03***	0.01	0.02, 0.05
SECD → LifeSat Slope	2.67***	0.52	2.15, 3.69
SECD → LifeSat Quadratic	−0.24**	0.09	−0.33, −0.06
PA intervention → LifeSat Slope	0.03	0.04	−0.01, 0.11
PA intervention → LifeSat Quadratic	−0.01†	0.01	−0.02, 0.01
Indirect Effect
PA → SECD → LifeSat Slope	0.09**	0.03	0.06, 0.15
PA → SECD → LifeSat Quadratic	−0.01*	0.00	−0.01, −0.01
Total Effect-LifeSat Slope	0.12***	0.04	0.08, 0.20
Total Effect-LifeSat Quad	−0.02**	0.01	−0.03, 0.00
Depression (Dep)b
PA intervention → SECD	0.03***	0.01	0.01, 0.05
SECD → Dep	−5.69***	1.37	−8.38, −3.00
PA intervention → Dep	−0.05	0.12	−0.29, 0.19
Indirect Effect
PA → SECD → Dep	−0.19**	0.07	−0.33, −0.05
Total Effect	−0.24*	0.12	−0.48, −0.00
Anxiety (Anx)b
PA intervention → SECD	0.03***	0.01	0.01, 0.05
SECD → Anx	−5.06**	1.70	−8.39, −1.73
PA intervention → Anx	−0.37*	0.14	−0.64, −0.10
Indirect Effect
PA → SECD → Anx	−0.17*	0.08	−0.33, −0.01
Total Effect	−0.53***	0.14	−0.80, −0.26

Note: PA= Positive Action. CI=Confidence interval. CIs were calculated as b +/− (1.96*SE). Comparative Fit Index (CFI) and Root Mean Square Error of Approximation (RMSEA) for Positive Affect = .79 and .06, respectively; for Life Satisfaction: CFI= .84 and RMSEA = .05; for Depression: CFI= .87 and RMSEA = .05; for Anxiety: CFI = .86 and RMSEA = .04.

^aEffects on slope (change) of measure over time.

^bEffects on measure at study endpoint.

^†P < .10;

^*P < .05;

^**P < .01;

^***P < .001; all 2-tailed.

Mediation Analyses

Results of tests for mediation are shown in Table 1. Replicating prior results²², the PA intervention had a significant direct effect on change in SECD in a favorable direction (b = .04, P <.001). Furthermore, it can be seen that in each model the path representing the effect of change in SECD on the slope of positive affect or life satisfaction or the end point levels of depression and anxiety was significant and in the expected direction. The indirect effect of the program on change in positive affect via SECD was significant (b = .03, P <.01) as were the indirect effects on end point depression (b = −.19, P <.01) and anxiety (b = −.17, P <.01). For life satisfaction, the indirect effects of the program on linear and quadratic change in the outcome were significant and marginally significant (b = .03, P < .05, and b = −.01, P < .10, respectively), with a noteworthy indirect effect on life satisfaction at study endpoint (ES = −.58). The residual direct effects of the program on positive affect, life satisfaction, and depression were non-significant, arguing for indirect mediation,³⁴ and significant for anxiety, arguing for complementary mediation.³⁴

Discussion

Results of this study are consistent with, but also extend those of previous investigations with respect to school-related influences on emotional health.⁴ Most notably, the present findings provide one of the first demonstrations of the capacity for school-based programs to be of benefit to the emotional health of youth living in urban, low-income communities. Program effects were significant for both anxiety and depression. The limited time available for survey administration in the school context necessitated use of only subsets of the items constituting the full versions of the BASC anxiety and depression scales. Normative data, therefore, are not available to facilitate interpretation of program effects on these measures. The magnitudes of the associated effect sizes, however, especially for anxiety (ES = −.26), are in the range that can be regarded as evidence of a noteworthy degree of impact.³⁷ Reductions in psychological distress may be consequential for youth in urban, low-income environments by lessening their susceptibility to the development of clinically-significant levels of mental health concerns, which are commonplace among young persons in such settings³⁸ The similar degree of program-facilitated improvements that were evident in positive affect and life satisfaction also merit attention. Although not directly addressed by the current findings, it is noteworthy that improvements in these facets of emotional well-being have been highlighted as having the potential to serve as protective resource for youth,⁸ a benefit that could be heightened among those living in high-stress environments. Frequent positive affect also appears more generally to be a facilitator of success in multiple life domains and of improved cognitive functioning, problem-solving, and decision-making in particular.³⁹ When programs are delivered in the school setting, as is the case with PA, the transfer of improvements in emotional well-being to certain areas such as learning may be facilitated because of their salience in the day-to-day experiences of students. .

Findings of the present study also indicate that fostering development of SECD may, in turn, be an important mechanism for school-based programs to improve the emotional health of youth. These results suggest that enhanced attention to fostering social-emotional competencies (e.g., honesty, self-development) of students could be useful for increasing the ability of such programs to be of benefit for emotional health outcomes. The evidence of a program effect on at least one outcome (anxiety) independent of gains in socio-emotional skills and behaviors, however, suggests the value of exploring other potential mediators of program impacts on emotional health (e.g., improved relationships with teachers and/or peers) in future research. Furthermore, as suggested by the research noted above, the potential for gains in emotional health (e.g., positive affect) to reciprocally foster improved SECD should also be kept in mind.

Several limitations of this study should be considered. First, measures were student self-report, potentially leading to bias in estimates of program effects.⁴⁰ Second, the findings are generalizable only to schools that would self-select to participate in a trial of this nature.²¹ Such schools may be more motivated and prepared to implement a program such as Positive Action than would a broader cross-section of schools. Third, the small number of schools limited statistical power for detecting program effects. Fourth, implementation of the program may be a contributing factor in the modest effect sizes found in the present study. Future research should examine how well the program was implemented and how to improve implementation, as well as how implementation may moderate program effects. Finally, as has been seen in other studies within low income, urban school settings,²⁵ student mobility led to high turnover of students; one implication of this mobility is reduced levels of exposure to the intervention among students in the program schools, potentially weakening observed effects on outcomes. Nonetheless, our supplemental analyses revealed no moderation by mobility group. Lastly, because the present sample was necessarily limited to those youth with consent to participate in the research, the extent to which findings generalize to all youth in the cohorts that were followed in each school is not known.

Conclusion

The findings of this research provide evidence of the effectiveness of PA, a universal school-based program, for improving emotional health among low-income, urban youth. Future research should examine whether the effectiveness of PA and related interventions for this purpose can be enhanced through refinements such as more focused attention to social-emotional skill development.

Abbreviations:

CPS

Chicago Public Schools

ES

effect size

ICC

intraclass correlation

MIRR

median incident risk ratios

PA

Positive Action

PANAS

Positive Affect and Negative Affect Scale for Children

SECD

Social-Emotional and Character Development

SECDS

Social-Emotional and Character Development Scale

SEL-HP

social-emotional learning and health promotion

SEM

structural equation modeling