Using social-emotional and character development to improve academic outcomes: a matched-pair, cluster-randomized controlled trial in low-income, urban schools

Abstract

BACKGROUND

School-based social-emotional and character development (SECD) programs can influence not only SECD, but also academic-related outcomes. This study evaluated the impact of one SECD program, Positive Action (PA), on educational outcomes among low-income, urban youth.

METHODS

The longitudinal study used a matched-pair, cluster-randomized controlled design. Student-reported disaffection with learning and academic grades, and teacher ratings of academic ability and motivation were assessed for a cohort followed from grades 3 to 8. Aggregate school records were used to assess standardized test performance (for entire school, cohort, and demographic subgroups) and absenteeism (entire school). Multilevel growth-curve analyses tested program effects.

RESULTS

PA significantly improved growth in academic motivation and mitigated disaffection with learning. There was a positive impact of PA on absenteeism and marginally significant impact on math performance of all students. There were favorable program effects on reading for African American boys and cohort students transitioning between grades 7 and 8, and on math for girls and low-income students.

CONCLUSIONS

A school-based SECD program was found to influence academic outcomes among students living in low-income, urban communities. Future research should examine mechanisms by which changes in SECD influence changes in academic outcomes.

A growing body of research indicates that school-based social-emotional and character development (SECD) and SECD-like programs (eg, social-emotional learning, positive youth development) can influence health behaviors and academic achievement among low-income minority youth, a population disproportionately affected by disparities in health¹ and education. ² In their meta-analysis examining the impact of school-based mental health and behavioral programs set in low-income, urban schools, Farahmand et al³ reported a mean effect size (generally Hedges g) on academic outcomes of 0.24. Durlak et al⁴ reported a mean effect size (generally Hedges g) on academic outcomes of 0.27 in their meta-analysis on school-based social-emotional learning (SEL) programs. With respect to health-related outcomes, the Durlak meta-analysis⁴ also showed SEL programs decreased conduct problems (effect size = 0.22) and emotional distress (effect size = 0.24), and improved positive social behaviors (effect size = 0.24). Whereas these findings are encouraging, there is a need to accumulate further evidence regarding the capacity of SECD programs to promote academic outcomes, especially when implemented in low-income, urban schools. Accordingly, the primary purpose of this study was to examine the impact of one comprehensive, school-wide SECD program, Positive Action, on academic outcomes using a longitudinal cluster-randomized controlled design in low-income, urban schools.

Positive Action⁵ is grounded in theories of self-concept, ^6–8 is consistent with social-ecological theories of health behaviors such as the Theory of Triadic Influence (TTI), ^{9, 10} and proposes positive feelings, thoughts, and actions result in fewer negative behaviors and enhanced motivation to learn. The core curriculum is taught through 6 units: self-concept, positive actions for mind and body, positive social-emotional actions focusing on getting along with others, and managing, being honest with, and continually improving oneself. The sequenced classroom curriculum consists of over 140 15–20 minute, age-appropriate lessons per grade taught 4 days per week for grades K-6, and 70 20 minute lessons taught 2 days per week for grades 7 and 8. The PA program also includes teacher, counselor, family, and community training, and school-wide climate development; the school-climate kit, which was used by every school in the trial of PA under study, focuses on using curriculum lessons and school activities to promote further positive actions amongst students, the school, families, and the community. More information about PA is available at http://www.positiveaction.net.

Prior research has demonstrated that the PA program impacts a range of risk and resilience factors linked to academic outcomes, as well as academic outcomes themselves. ⁶ In an analysis of 3 longitudinal randomized controlled trials (RCT) of PA involving students aged 6 to 11 years, PA partially mitigated the decrease in number of positive behaviors endorsed by youth across time. ¹¹ In a matched-pair RCT of PA involving 20 schools in Hawai’i, PA was shown to create whole-school contextual change and improve school quality. ¹² Students in schools receiving PA were also less likely to engage in substance use, violent behaviors, or sexual activity,¹³ and PA schools had significantly higher school-level academic achievement and less absenteeism.¹⁴

Limitations in prior PA research should be addressed. For example, the academic impact of PA during the middle-school years has not yet been examined. Doing so is critical, as the adolescent years represent a key developmental period with new academic and social demands. Also, the need exists to collect academic-related data from students and teachers so that precursors of academic achievement (eg, engagement with learning) that cannot be measured by school-level archival records alone can be assessed. Lastly, the need exists for experimental designs of PA in low-income, urban settings. The present study addresses these limitations by: (1) following a cohort of students during the elementary- and middle-school years; (2) including student self-reports and teacher ratings of students; and (3) being set in a low-income, urban setting. The purpose was to test the hypothesis that academic performance across time would be better among schools and students receiving PA, than those not receiving PA.

METHODS

Participants

Participating schools were drawn from 483 K-6 and K-8 Chicago Public Schools. Schools were excluded from participation if they: (1) were non-community schools (eg, charter schools and magnet schools)’ (2) already had PA or a similar intervention; (3) had an enrollments below 50 or above 140 students per grade; (4) had annual student mobility rates over 40%; (5) had more than 50% of students who passed the Illinois State Achievement Test (ISAT); and (6) had fewer than 50% of students who received free lunch. The latter 2 criteria ensured the selection of high-risk schools. A total of 68 schools met eligibility criteria, of which 18 agreed to participate, and the 7 best-matched pairs (the N that funding would support) were selected for participation; the following variables were used in the matching process: ethnicity, percentage of students who met or exceeded criteria for passing the ISAT, attendance rate, truancy rate, percentage of students who received free lunch, percentage of students who enrolled in or left school during the academic year, number of students per grade, percentage of parents reported to demonstrate school involvement, percentage of teachers employed by the school who met minimal teaching standards, and crime rate for the neighborhood in which the school was located.^15–18 A series of t-tests revealed that the 7 pairs of schools did not significantly differ from the remainder of the 68 schools eligible for the study, and the PA and control schools were not significantly different from each other on any of the matching variables.^{15, 17} Throughout the 6 years of the study, 100% of schools were retained.

The total number of students in the analytic sample was 1170, of whom approximately 53% were girls; approximately 48% were African American, 27% Hispanic and 19% other (eg, White, Asian, Native American, or “Other”). A total of 247 teachers completed student assessments; 75% of teachers were women; 43% White, 36% African American, 13% Hispanic and 8% other (Asian and Native American).

Instruments

Student self-report measures

Disaffection with learning was assessed using 4 items from a measure of student engagement developed by Furrer and Skinner.¹⁹ Principal components factor analysis on student responses showed this measure loaded strongly onto one factor at both Wave 1 (loadings greater than or equal to 0.66) and Wave 8 (loadings greater than or equal to 0.67). Items were rated on a 4-point Likert scale (“Disagree A LOT” to “Agree A LOT”) and included “When I’m in class, I think about other things” and “When I’m in class, my mind wanders.” A mean of the items was used to create a composite score, whereby higher scores reflected having more disaffection. Cronbach’s alpha across the 8 waves of data ranged from 0.64 to 0.71. To assess the impact on academic grades, students were asked, “What grades have you been getting this school year?” with response options ranging from 1 to 9 (e.g., 1 = Mostly F’s, 4 = mix of C’s and D’s, and 9 = Mostly A’s).

Teacher ratings of students

Teachers assessed students using pre-existing measures of academic ability and motivation.^{20, 21} Each consented student was rated in the areas of reading, mathematics, academic performance, and intellectual functioning using a 5-point Likert scale (1 = Far below grade level to 5 = Far above grade level). Due to multicollinearity (ie, correlations of 0.84 and higher) between these items, a composite score was created, with higher scores indicating higher teacher ratings of students’ academic ability. Cronbach’s alpha for the composite measure ranged from 0.97 to 0.98. Academic motivation was assessed with a single-item measure, with response options ranging from “Extremely low” to “Extremely high”.

School-level archival data

Because state test data provide a policy-relevant measure of achievement, ²² archival reading and math scores of non-English Language Learners on a standardized, school-administered, statewide test (the ISAT) were gathered from the Chicago Public Schools website.²³ The website provided information on the percentages of students tested (all students, grade-specific, and demographic subgroups) whose scores fell into each category (ie, Warning, Not Meeting Standards, Meeting Standards, or Exceeding Standards). A single weighted average of the percentages of students falling into each achievement level was created for each school (ie, [[1 × % of students at Warning level] + [2 × % of students NOT meeting standards] + [3 × % of students meeting standards] + [4 × % of students exceeding standards]]) for both reading and math, overall and by demographic sub-groups.

A value-added metric index of ISAT performance was also reported by the school district.²⁴ These indices control for the prior year ISAT scores of students as well as other relevant factors (ie, grade level, gender, race/ethnicity, low income status, English Language Learner status, Individualized Education Plan status, homelessness, and mobility) and are designed to reflect the extent to which scores for a group of students improved (or declined) more than would be predicted based on these factors. Data were available for our student cohort transitioning from grades 7 to 8 (2009–10).

The school district reported average daily attendance rates for each school on a scale from 0 to 100%; these statistics were converted to a measure of average daily absenteeism by subtracting 100 from each school’s respective year-end attendance.

Procedure

The Chicago trial of PA was longitudinal (ie, 6 years and 8 waves) at the school level and used a place-focused, intent-to-treat design with a dynamic cohort at the student level.²⁵ Surveys were administered to students beginning in grade 3 (fall 2004), and at 7 additional time points (waves) over 6 years: spring 2005, fall 2005, spring 2006, spring 2007, fall 2008, spring 2009, and spring 2010 (end of grade 8).

Parental consent was obtained before students, parents, or teachers completed surveys when students were in grade 3, with students joining the study at later waves consented at the time of entry into the study. All students were re-consented for the second phase of funding at wave 6. At baseline, 79% of parents provided consent; consent rates ranged from 65% to 78% for waves 2 through 5, and from 58% to 64% for waves 6 through 8.

The total number of students in the analytic sample across all waves was 1170. Of the original 624 students in grade 3 at the beginning of the trial, only 131 (ie, 21%) remained at grade 8, reflecting the high mobility by low-income urban students. With respect to maintenance of the baseline sample size, 363 students were present at wave 8 (ie, approximately 61% of the Wave 1 sample size); the decrease in N over time is consistent with the trend among Chicago Public Schools to decrease in size during the study period, together with lower consent rates at wave 6 through 8.¹⁵

To substantiate student self-reports, teacher assessments of students and archival data were used. Student assessments were completed by teachers at all waves excepting wave 6 (the transition from one funding cycle to the next). Percentages of consented students for whom teachers completed ratings for at each wave (excepting wave 6) ranged from 72% to 93%. Archival ISAT and absenteeism data were collected for the 3 academic years prior to the baseline, as well as throughout the duration of the study.

Data Analyses

Analyses were conducted using Stata version 12.1. Preliminary analyses involved assessing distributions of each outcome and calculating intraclass correlations, Cronbach’s alphas, and correlations between the student and teacher variables at Waves 1 and 8.

Primary analyses consisted of multilevel growth-curve models to account for all observations and to model school differences. These were 3-level, time within students within schools, analyses for student-level measures, and 2-level, time within schools, analyses for the aggregated school-level data. We used Stata’s “xtmixed” command for normally distributed outcomes, and “xttobit” for outcomes with a positively or negatively skewed distribution (ie, censored below or above, respectively). ²⁶

A random-intercept model was fitted using the following equations for student- and school-level analysis, respectively:

• Ŷ_tij= β₀+ β₁(condition_j) + β₂(time_tij) + β₃(condition_j × time_tij) + ζ_j + ζ_ij + ε_tij [Student-level]

• Ŷ_tj = β_0j + β₁(condition_j) + β₂(year_tj) + β₃(year_tj × condition_j) + ζ_j + ε_tj [School-level]

Ŷ_tij and Ŷ_tj represent the estimated score on a particular outcome at a particular time t (measured as study duration, in years, for student-level models, and as academic year in school-level models). Additionally, i represents a student, j represents a school, β₀ represents the mean intercept and ζ_j is deviation of a school’s mean score from the mean score for all schools. ζ_ij is deviation of each student’s score from their school’s mean, and ε_tij and ε_tj are the residual. The original models included quadratic terms for time and the interaction of condition by time. Nonsignificant higher order terms were dropped from the model for parsimony, whereas outcomes with significant quadratic terms (eg, condition × time²) were graphed to facilitate interpretation of growth trajectories.

When applicable, analyses with student-level variables were run using both the fully reduced random-intercept and random-coefficients models, with the former model nested within the latter model. A likelihood-ratio test was performed to determine whether the random-coefficients model was a better fit for the data. ²⁶

Due to the power and sample size limitations, and because the a priori directional hypothesis was that the PA schools would have greater improvements across time, one-tailed p-values were used in tests of effects of the PA program on school-level outcomes.²⁷ In the analyses using ISAT weighted averages, 6 matched pairs were retained (for reasons discussed elsewhere); ¹⁵ all 7 matched pairs were retained for the endpoint value-added ISAT analysis and for the absenteeism growth-curve analysis. For all outcomes (student-level and school-level) analyzed using growth-curve analyses, effect sizes were calculated using the method described by Lipsey and Wilson. ²⁸

Sensitivity analyses assessed the robustness of results from the primary analyses. A first approach involved including a “pairs” variable as an additional level in each of the best-fitting models to determine whether adding a fourth level would affect findings. Second, to provide a more conservative test (from a statistical power perspective) of program effects for each outcome, the test statistic provided by Stata (which assumes a large sample size) in the primary analyses (N=14 schools) was compared to the critical value for a 2-tailed t-distribution with 12 degrees of freedom at a 95% confidence level (2.18).²⁹

For student-level data, the possible moderating effects of sex and student mobility were examined. The effect of student mobility groups was examined using results from a latent class analysis¹⁵ in which a 5-class solution was found to be the most appropriate fit for the data: (1) stayers (average study duration of 5.72 years, N=158); (2) temporary participants (present for grade 4 and/or 5 only; average study duration of 1.30 years; N=196); (3) late joiners (average study duration of 1.38 years; N=308); (4) early leavers (average study duration of 0.94 years; N=263); and (5) late leavers (average study duration of 3.23 years; N=287); stayers served as the reference group.

RESULTS

The intraclass correlations (ICCs) for the student-level measures were generally low, with none of the ICCs for student-reported and only 1 of the 14 ICCs for teacher-reported outcomes above 0.10. Scale reliabilities (reported above) were generally high, with a clear increase in Cronbach’s alphas as students aged. Table 1 shows the correlations between the student and teacher variables at Waves 1 (beginning of grade 3) and 8 (end of grade 8).

Table 1.

Youth and Teacher Reports of Academic Outcomes: Correlations at Wave 1 (above the diagonal, N=603) and Wave 8 (below the diagonal, N=335)

Variables	1	2	3	4	5	6	7
Student Self Reports
1. Disaffection with Learning	–	−0.04	−0.31**	−0.29**	−0.32**	−0.29**	−0.27**
2. Self Reported Grades	−0.23**	–	0.24**	0.17**	0.21**	0.21**	0.20**
Teacher Ratings of Students
3. Reading	−0.03	0.33**	–	0.84**	0.89**	0.93**	0.71**
4. Math	−0.06	0.37**	0.93**	–	0.84**	0.87**	0.67**
5. Intellectual Functioning	−0.01	0.29**	0.91**	0.89**	–	0.91**	0.71**
6. Academic Performance	−0.07	0.34**	0.93**	0.93**	0.92**	–	0.73**
7. Academic Motivation	−0.09	0.44*	0.67**	0.67**	0.64**	0.68**	–

^**significant at 2-tailed .01 level

Program effects (significant condition × time and condition × time² interactions) were present for disaffection with learning (Table 2). Students in PA schools started off higher than those in control schools (ie, more reported disaffection with learning). There was then an overall trend toward a net increase in disaffection with learning by the end of the study period in both PA and control schools; the pattern of change was linear in control schools and curvilinear within PA schools.

Table 2.

Multilevel Growth-curve Model Estimates for Student-level Measures (N=1170 students) and Aggregated School-level (N=14 schools) Archival Measures

Measure	Model Run	Intercept	Time	Time2	Condition (0 = Non-PA 1=PA)	Condition × Time	Condition × Time2
		B (SE)	B (SE)	B (SE)	B (SE)	B (SE)	B (SE)
Student Self Reports
Disaffection with Learning	Random Intercept	1.69 (0.06)**	0.03 (0.04)	0.01(0.01)	0.15 (0.08)*	−0.20(0.06)**	0.03 (0.01)**
Self Reported Grades	Random Intercept	7.89 (0.12)**	−0.81(0.07)**	0.11(0.01)**	0.10(0.17)	0.01(0.03)	---
Teacher Ratings of Students
Academic Performance a	Random Coefficients	2.62 (0.06)**	−0.05(0.03)*	0.02(0.005)**	−0.06(0.08)	0.02(0.02)	---
Academic Motivation	Random Coefficients	3.01(0.07)**	0.04(0.04)	−0.01(0.01)	0.05(0.10)	−0.12(0.06)*	0.03(0.01)**
School Level Archival Data b
Absenteeism	Random Intercept	6.76 (0.56)**	0.03 (0.05)	---	−0.43 (0.65)	−0.16 (0.07)*	---

^aFor the random-intercept model, the condition × time interaction is significant at the .05 level (B = 0.03, p < .05).

^bFor school level measures, time variable created using academic year, rather than time since implementation of intervention. Also, the one-tailed p-value is reported for school-level measures.

⁺p < .10;

^*p < .05;

^**p < .01

As shown in Table 2, there was evidence of a program effect on teacher ratings of student academic motivation in the form of significant condition × time and condition × time² interactions. For students in PA schools, after an initial period of modest decline there was an accelerating increase, whereas for control school students there was a gradually decreasing trend. The net result was notably higher predicted levels of teacher-rated academic motivation for students in PA schools. Sensitivity analyses at the pair level supported this finding (results not shown).

With respect to teacher-rated academic ability, a significant condition × time interaction was found in the random-intercept model. In the random-coefficients model, which provided a better fit, the condition × time interaction was not significant (B = 0.03, p < .05 in random-intercept model; B = 0.02, p > .05 in random-coefficients model). For both teacher-rating measures, there was no evidence of moderation of program effects by mobility group; gender moderation was observed for academic ability, with PA boys being rated higher by teachers than control boys.

Growth-curve analyses for the weighted composite measure of ISAT scores for all students in PA and non-PA schools did not reveal evidence of a program effect for Reading. There was, however, evidence of marginal program effects for Math (Table 3). When “pairs” was included in the random-intercept model, this finding remained marginal (results not shown). With respect to demographic subgroups, significant condition × time interactions were seen in Reading performance for African American boys (B = 0.03, one-tailed p < .05). The condition × time interaction remained significant in the pair-level analysis (results not shown). Marginal results (p-values less than or equal to .10) indicative of favorable growth in PA schools as compared to control schools, were observed for Reading performance for boys and African American students, and for Math performance for girls and students receiving free or reduced-price lunch.

Table 3.

Multilevel random-intercept growth-curve model estimates for standardized academic test scores ^a (N=12 schools)

Variables	Intercept b	Time b	Time2 b	Condition (0 = Non-PA; 1= PA)	Condition × Time	Condition × Time
	B (SE)	B (SE)	B (SE)	B (SE)	B (SE)	One-tailed p-value
Reading
All Students (Grades 3-8 Combined)	2.26 (0.07)	0.17 (0.01)	−0.02 (0.002)	0.04 (0.10)	0.01 (0.01)	0.16
Sub-Groups
Boys	2.22 (0.07)	0.16 (0.02)	−0.02 (0.003)	−0.001 (0.10)	0.01 (0.01)	0.12
Girls	2.30 (0.07)	0.17 (0.02)	−0.02 (0.003)	0.07 (0.10)	0.004 (0.01)	0.35
African Americans	2.20 (0.06)	0.15 (0.02)	−0.01 (0.003)	0.05 (0.08)	0.01 (0.01)	0.10
African American Girls	2.21 (0.05)	0.17 (0.02)	−0.02 (0.003)	0.13 (0.07)	−-0.01 (0.01)	0.23
African American Boys	2.17 (0.07)	0.16 (0.03)	−0.02 (0.005)	−0.02 (0.10)	0.03 (0.01)	0.02
Free or Reduced Price Lunch	2.25 (0.07)	0.17 (0.01)	−0.02 (0.002)	0.03 (0.09)	0.01 (0.01)	0.18
Math
All Students (Grades 3-8 Combined)	2.15 (0.08)	0.24 (0.02)	−0.03 (0.003)	0.04 (0.12)	0.01 (0.01)	0.07
Sub-Groups
Boys	2.12 (0.09)	0.24 (0.02)	−0.03 (0.004)	0.04 (0.12)	0.01 (0.01)	0.13
Girls	2.18 (0.08)	0.24 (0.02)	−0.03 (0.004)	0.04 (0.11)	0.02 (0.01)	0.09
African Americans	2.06 (0.06)	0.23 (0.02)	−0.02 (0.004)	0.06 (0.08)	0.02 (0.01)	0.11
African American Girls	2.09 (0.07)	0.23 (0.03)	−0.02 (0.004)	0.10 (0.09)	0.02 (0.01)	0.11
African American Boys	2.02 (0.07)	0.25 (0.03)	−0.03 (0.005)	0.04 (0.09)	0.02 (0.01)	0.12
Free or Reduced Price Lunch	2.15 (0.08)	0.24 (0.02)	−0.03 (0.003)	0.04 (0.11)	0.01 (0.01)	0.07

^aThe average of values from 2000/2001 through 2002/2003 was used as the estimate of baseline levels.

^bThe coefficients for Intercept, Time, and Time² were all significant at the .01 level, except the time² coefficient for African American boys, which was significant at the .05 level.

Endpoint regression analyses for our study cohort, using the value-added metric of the same standardized test, showed significant results in Reading, but not Math. As compared to students in control schools making the grade 7 to 8 transition, students in PA schools performed significantly better in reading (B=1.26, one-tailed p=0.013, effect size=0.83, results not shown).

As shown in Table 2, growth-curve analyses showed there was lower absenteeism at PA schools than control schools (B=−0.16, one tailed p=0.015). Sensitivity analyses using the pair-level variable and the adjusted degrees of freedom supported these findings (results not shown).

Table 4 shows the estimated means of our outcomes at baseline and endpoint, as well as the effect sizes for each outcome. The largest effect sizes for school-level measures were for absenteeism (effect size = −0.78) and reading performance on the ISAT for African American boys (effect size = 1.50). With respect to student-level measures, the largest effect size was observed for disaffection with learning (effect size = −0.19) and teacher ratings of academic motivation (effect size = 0.39).

Table 4.

Estimated Means and Effect Sizes for Student- and School-level Data

			Wave 1		Wave 8
Measure	Response Options	Model Run	Control	PA	Control	PA	Effect Size a
Student Self Reports
Disaffection with Learning	1 to 4	Random Intercept	1.69	1.85	2.19	2.19	−0.19
Self Reported Grades	1 to 9	Random Intercept	7.89	7.98	6.67	6.81	0.02
Teacher Ratings of Students
Academic Ability	1 to 5	Random Coefficients	2.63	2.57	2.84	2.91	0.14
Academic Motivation	1 to 5	Random Coefficients	3.01	3.06	2.80	3.24	0.39
School Level Archival Datab
Absenteeism	0 to 100	Random Intercept	6.76	6.33	6.95	5.58	−0.78
ISATs-Reading	1 to 4
All Students (Grades 3–8 Combined)	1 to 4	Random Intercept	2.26	2.29	2.64	2.72	0.22
Boys	1 to 4	Random Intercept	2.22	2.22	2.60	2.66	0.33
Girls	1 to 4	Random Intercept	2.30	2.37	2.68	2.78	0.11
African Americans	1 to 4	Random Intercept	2.20	2.25	2.62	2.74	0.50
African American Girls	1 to 4	Random Intercept	2.21	2.34	2.66	2.74	−0.54
African American Boys	1 to 4	Random Intercept	2.17	2.15	2.57	2.72	1.50
Free or Reduced Price Lunch	1 to 4	Random Intercept	2.25	2.28	2.63	2.70	0.23
ISATs-Math	1 to 4	Random Intercept
All Students (Grades 3-8 Combined)	1 to 4	Random Intercept	2.15	2.19	2.67	2.79	0.38
Boys	1 to 4	Random Intercept	2.12	2.17	2.67	2.79	0.31
Girls	1 to 4	Random Intercept	2.18	2.22	2.68	2.81	0.41
African Americans	1 to 4	Random Intercept	2.06	2.12	2.62	2.77	0.55
African American Girls	1 to 4	Random Intercept	2.09	2.19	2.61	2.80	0.69
African American Boys	1 to 4	Random Intercept	2.02	2.07	2.62	2.76	0.63
Free or Reduced Price Lunch	1 to 4	Random Intercept	2.15	2.19	2.67	2.79	0.42

^aEffect size calculations made using estimated means. Namely, the estimated mean difference at the baseline was subtracted from the estimated mean difference at the end point to obtain the difference of differences, and this value was then divided by the pooled standard deviation at baseline.

^bFor school level measures, time variable created using academic year, rather than time since implementation of the Positive Action intervention.

DISCUSSION

In the Chicago trial of PA, the intervention had a positive impact on absenteeism, mitigated a natural increase in disaffection with learning, and PA teachers rated their students as experiencing greater growth in academic motivation and ability; these findings are encouraging, as these outcomes are predictors of long-term academic achievement and school completion. ^30–32 Favorable growth was also observed with respect to ISAT Reading and Math performance, particularly for African American boys and students receiving free or reduced-price lunch. Socioeconomic background (ie, low-income), sex (ie, being male) and ethnicity (ie, African-American, Hispanic, and Native American youth) are known predictors of school drop-out, and school drop-out is associated with a multitude of negative outcomes. ³¹ As prevention programs can only influence those factors amenable to change (eg, motivation to learn), it is encouraging that this trial also demonstrated improvements in test scores for these high-risk groups.

The impact on academic-related outcomes observed in this study may be attributed to a number of factors. For example, the skills fostered by the PA program (eg, problem solving, self-control, emotional regulation, and attention), and lesson plans focusing on improving motivation to learn and do well in school, may in part explain the observed results.⁵ In addition, the promotion of positive behaviors may have resulted in less time being spent by teachers on classroom management and, subsequently, more time devoted to interactive strategies that create an intellectually stimulating environment.⁵ Moreover, the impact on academics may have been mediated through improvements in attachment to school and teachers.

This study is the first to examine the academic impact of PA in a low-income, urban setting, and supplements Snyder et al’s¹⁴ findings on the academic impact of PA in Hawai'i by including data from students and teachers of students in the elementary- and middle-school grades. The study also adds to the research of Madsen et al,³³ who evaluated the impact of a physical-activity focused, school-based, Positive Youth Development program in low-income Bay Area California schools using a quasi-experimental time series design; namely, the researchers found that each additional year of exposure to the program resulted in significantly higher scores in meaningful participation in school and academic-related goals and aspirations of youth. In the current study, for those measures with significant program effects, the effect size for disaffection with learning (effect size = −0.19) was smaller than the effect sizes for academic outcomes reported by the research teams led by Farahmand³ and Durlak.⁴ On the other hand, other measures in this study (eg, academic motivation, absenteeism, ISAT Math results) had larger effect sizes than those observed in the aforementioned studies.

Limitations

This study is not without its limitations. Student and teacher-reports on academic measures are subject to social desirability bias; this potential bias was addressed by supplementing student and teacher reports with archival measures representing the actual performance of students on standardized tests. Another possible limitation of the study is that students in the intervention group may have acted differently because they knew they were receiving the PA program, a phenomenon known as the Hawthorne effect. This limitation was addressed through the trial’s use of a control group of students and teachers who were also aware they were being observed as part of a study. With respect to external validity, the findings are generalizable only to similar schools (ie, low-income, urban schools) that would self-select to participate in a trial of this nature. The small number of pairs and schools (ie, 7 and 14, respectively) could influence statistical power; however, that significant findings were found in primary and sensitivity analyses suggest that our findings are robust. Additionally, student mobility led to high turnover of students, which is problematic as it can become difficult to determine whether observed effects can be attributed to the intervention or differential attrition.²⁵ One approach to analyzing mobility patterns is latent class analysis (LCA), ^{34, 35} and the present study contributes to the LCA literature by examining students who enter a study, not just those who exit;¹⁵ program effects were not found to differ by mobility class.

Limitations notwithstanding, the present study has several strengths. The longitudinal nature of this RCT allowed examination of school performance across 6 years, encompassing both elementary- and middle-school grades. The data from multiple sources as well as the sensitivity analyses provide confidence in study findings. In addition to standardized test performance, our study also reported on theoretically-expected mediators of academic success (eg, disaffection with learning). Moreover, this study involved a sample of students in a high-risk setting. Thus, policymakers aiming to alleviate educational disparities should use scientific data from this and other evidence-based studies to advocate for comprehensive school-based SECD programming.

Conclusions

Findings from this study reinforce prior findings that SECD-like programs can improve academic achievement as well as improve student behavior and health. Future studies should determine the mechanism by which SECD programs such as Positive Action improve academic outcomes (eg, mediation through factors that SECD programs seek to foster, such as attachment with teacher and school, improved school climate, emotional regulation, attention, executive function, and increased self-control). Future research could also supplement student and teacher reports by gathering data from parents that may influence academic performance (eg, parent’s highest level of education).

IMPLICATIONS FOR SCHOOL HEALTH

In an era where increased pressures to “teach to the test” may lead school officials to feel as though they have neither the time nor money to invest in evidence-based prevention programming,³⁶ there is an increasing need to demonstrate the impact that multifaceted prevention programs can have on academic performance and student and community wellness.³⁷ When taken together with preliminary research showing the impact of this trial on health behaviors,³⁸ results from this study demonstrate the possibility of addressing the proverbial “2 birds” (ie, health and academics) with “one stone” (ie, school-based social-emotional and character development programs).

Human Subjects Approval Statement

The research presented herein was approved by the institutional review boards of Oregon State University and the University of Illinois at Chicago, the Research Review Board at Chicago Public Schools and the Public/Private Ventures Institutional Review Board for Mathematica Policy Research Inc.