Active learning improves on-task behaviors in 4^th grade children

Abstract

While increased opportunities for physical activity (PA) are a critical, public health need for children, school-based interventions often place teachers in the position to choose between PA and time spent on academic lessons. Active learning is designed to overcome this by combining PA with academic material. Moreover, teachers are likely to be more responsive to change in academic-related outcomes than in PA. This study utilizes a large, cluster randomized control trial in which student attention, or time on task (TOT) and accelerometer-based PA is assessed in conjunction with active learning. Participants were 2,716 children (46% male, 46% white) from 28 elementary schools in Central Texas that were assigned to either: 1) active learning (math n= 10; spelling n=9); or 2) traditional, sedentary academic lessons (n=9). PA was measured with accelerometers. TOT was measured through a momentary time sampling protocol. A series of three-level (student, classroom, school) regression models estimated the effect of the intervention. The intervention lead to significantly increased TOT. Moreover, the dose of PA (steps) during the intervention was positively associated with the increase in TOT. In contrast, a greater dose of PA was associated with reduced TOT for students in control schools. Race, gender, and SES did not moderate these effects. Planned PA - as a part of an active, academic lesson - positively impacted TOT. In contrast, a traditional, sedentary lesson was associated with lower TOT. This differential impact offers intriguing possibilities to better understand the relationship between PA and academic performance.

Introduction

Given the paucity of children who meet recommendations for daily physical activity (PA) ¹, and the decline in PA that occurs during the elementary years ², there has been a concerted effort to increase opportunities for PA during the elementary school day. Schools are natural target for intervention. During the week, children spend the majority of their waking time in school and the majority of that time – up to 73% - is spent sedentary ³. In response, there are a number of interventions that seek to increase opportunities for PA in school. While these have successfully been applied to PE ⁴ and recess ⁵, the impact of these interventions are undermined by the fact that they target time when many children are already active. In contrast, PA interventions will be maximized to the extent that they can substitute PA for what would traditionally be seated time. While this may be a natural and acceptable trade-off for a public health specialist, it is far from obvious for teachers and school administrators who are unlikely to sacrifice time for academic pursuits to increase child PA. The health and academic missions of schools ⁶ can come into conflict as administrators seek additional academic time – especially given the growing emphasis on standardized test performance. For example, schools across the country have chosen to reduce time in recess and PE to expand time spent on academic lessons ⁶. It is imperative that public health researchers seek to counter this trend and to do so with outcomes that are relevant to the academic mission of schools.

Fortunately, there is strong evidence that PA can enhance academic outcomes in children. While the impact of fitness and PA on overall academic performance is well known ⁷, there is also support for the impact of acute bouts of PA, especially for student focus. Students have been shown to respond to brief periods of PA with enhanced attention and impulse control ^8–10. PA during recess has been shown to improve classroom focus on teacher assigned tasks and reduce the amount of disruptive, verbal interaction between students ¹¹. Likewise, students aged 9–11 years who engaged in even four minutes of MVPA improved standardized assessment of attention and concentration, ¹² which, in turn, predict academic achievement ^13,14. The attention data are in line with research that demonstrates the benefits of acute bouts of PA for executive functioning ^15,16, concentration¹⁷, and academic performance ^20,21. Moreover, these acute responses may be particularly impactful when advocating for change. Teachers are sensitive to challenges with children’s attention control ¹⁸ and PA interventions that can be shown to modify this outcome may be attractive to schools.

The challenge is to develop and test PA interventions that would occur during the normally, sedentary class times without sacrificing academic time. Physically active academic lessons, or active learning, are designed to do just this. They integrate acute bouts of PA with academic material as a means to increase MVPA throughout the school day without sacrificing academic time ¹⁹. Active learning interventions are often brief, 5–15 minute lessons that range from teacher-designed and implemented lessons to video-based systems and have been associated with improved academic performance. Mullender-Wijnsma and colleagues (2016) conducted a two-year intervention in which math and language lessons were taught through acute 10–15 minute physically active lessons ²⁰. Results indicated significant improvement in both mathematics and spelling ²¹. In addition, while each bout of active learning is of a relatively short duration, they consistently result in acute improvement in attention and impulse control as assessed by direct observation of time focused on teacher-assigned tasks (TOT). Mahar and colleagues (2006) found that implementing acute 10-minute bouts of active learning improved TOT by more than 10% for students aged 8–11 years old ¹⁰. Grieco et. al, (2009) showed similar results in teacher-implemented active learning in 3^rd – 5^th grade children ⁹. It may be that the long-term improvement in academic performance was due, in part, to the accumulated benefit of daily, acute improvement in attention and impulse control following each active lesson.

Recently, Szabo-Reed and colleagues (2017) showed an improvement in TOT that was correlated with the dose of activity ²². They assessed MVPA through observation via SOFIT (System for Observing Fitness Instruction Time) and found that the acute change in TOT was correlated with the level of MVPA during the lesson. These stand in contrast to the only experimentally-manipulated assessment of the dose response. Grieco and colleagues (2016) compared low and moderate intensity active lessons with a traditional sedentary lesson and an enjoyable, though sedentary game ⁸. Results indicated that TOT only increased following both physically active lessons, but with no difference between low and high intensity versions ⁸. In addition to the differences in design (experimental vs correlational) there are a number of other differences between these studies. The correlational study (Szabo-Reed, et al., 2017) failed to include a control condition and examined children in groups of 5 across multiple days to achieve the full class assessment over time rather than a single assessment/class. Moreover, their assessment of PA intensity was based on observation rather than accelerometer-based assessment. To address these limitations, the present study recruited 28 elementary schools, with 149 teachers to implement the lessons, and 2,716 4^th grade students as participants. This is a sample that provides the power to conduct the appropriate hierarchical model for data nested within classrooms and to test for potential moderating effects for demographic sub-group. Research staff conducted TOT observations for all participants in full class sessions in conjunction with active or traditional sedentary control lessons. PA was assessed with GT3X+ accelerometers to accurately assess intensity. As a result, this is the first study to provide the power and appropriate design to conduct a hierarchical analysis of the moderating role of both PA intensity and demographic sub-groups on the impact of active learning on TOT.

Methods

Participants

Students and parents were recruited from 28 schools (n=19 intervention; n=9 control) with 149 teachers (n=99 intervention; n=50 control) to participate in the parent study, Texas Initiatives for Children’s Activity and Nutrition (I-CAN!) program over the years 2012–2015. The I-CAN! program focuses on implementing active lessons in the classroom for 10–15 minutes for 5 days during the school week ²³. Schools were recruited for study participation and agreed to be randomly assigned to one of 3 possible conditions: control, intervention language arts, or intervention math. Once schools were recruited for the next year of the study, they were then grouped into sets of 3 based on similar demographics and size. Once the sets of 3 were created, the 3 schools within each set were then randomly assigned to conditions. All students in the 4^th grade from these schools were recruited to participate in I-CAN!. For more details on the design see Bartholomew et al., 2017 ²³. Both parents and students provided informed consent prior to the start of the study. The institutional review boards for each of the four school districts as well as The University of Texas at Austin approved this study.

I-CAN! active lessons include a series of general games (e.g. freeze tag, relay races) focused on either math or language arts academic material. These active lessons are designed to engage students in moderate-intensity aerobic PA, but the resulting level of activity is dependent upon the specific lesson chosen and implemented by the teacher. Results for the impact of I-CAN! on child PA can be found in Bartholomew et al., 2018 ²⁴. The current study examined the impact of a single, acute bout of PA during the active lesson and its impact on TOT compared to a traditional academic lesson. A traditional academic lesson consists of students seated at their desks and listening to the teacher present the material or students are seated completing a teacher assigned task.

Student demographic information (i.e. sex, age, race/ethnicity, eligibility for free/reduced lunch) was obtained through school records. Table 1 presents the demographic information for schools and students participating in the study. Schools were 9.5% Black, and 46.3% White, with 32.0% Hispanic. In each school, an average of 21.0% of students were eligible for free or reduced-priced lunch. To establish baseline equivalence, students were compared in the treatment and control conditions on the characteristics presented in Table 1. There were no statistically significant differences (p-values ranged from .09 to .51). Moreover, there were no demographic differences at the school level (p-values ranged from .74 to .98), suggesting that randomization successfully produced comparable groups of schools at baseline.

Table 1.

Demographic Characteristics of Schools and Students

	Overall	Treatment	Control
School characteristics	j = 28	j = 19	j = 9
Mean % economically disadvantaged	34.31	35.43	31.69
Mean race
% Hispanic	31.99	33.06	29.51
% African American	9.51	9.73	8.97
%White	46.26	46.54	45.60
Student characteristics	n = 2716	n = 1903	n = 813
Male	45.90	49.00	43.90
Free/reduced priced lunch	21.70	23.60	19.70
Race
Hispanic	22.9	24	22.9
American Indian/Alaska Native	1.1	1.4	0.5
Asian	6.3	4.4	11.4
African American	7.7	8.3	7.1
Native Hawaiian	0.1	0.1	0.2
White	45.8	49.2	42.7
Multi	4.7	4.8	4.9
Missing	11.5	7.9	10.2

Attrition

Attrition and attrition-related bias in cluster randomized-control trials depend on attrition at both the cluster and the case levels. In the current sample, there was no school-level attrition. At the student level, overall attrition for TOT measurements was 20%. That is, 80% of the students in the parent study completed the TOT measures. Differential attrition was 4%, suggesting no difference between condition and that student-level attrition posed a minimal threat to the study’s internal validity (i.e., very low bias due to differential attrition between conditions) based on guidelines provided by the What Works Clearinghouse ²⁵.

Procedures

Moderate to vigorous physical activity (MVPA) and step count were assessed during data collection week through accelerometers, starting on a Monday and ending on a Friday. Students wore an Actigraph GT3X+ monitor in a belt around their waist, positioned on their right hip during the entire school day for an average of 6 hours per day. The Actigraph GTX3X+ is widely considered the most valid and reliable accelerometer ^26,27. Evenson et al., (2008) cut points were used to determine percent time in MVPA and step-counts ²⁸. These data were downloaded onto a computer and analyzed with ActiLife software.

On-task behaviors were observed using five-second momentary time sampling for 15-minutes prior to a physically active lesson and for 15-minutes immediately following the active lesson. TOT was assessed for half of the schools in the Fall semester, and the other half of the schools in the Spring semester, within their sets of 3. Students were observed either before or after lunch depending on the class schedule. For the intervention schools, 73% of the observations were done in the morning while 43% of observations were conducted in the morning in the control schools. Time of day TOT was not found to be a significant predictor of on-task behaviors. These procedures were the same in the control schools, on-task behaviors were observed 15-minutes prior to the traditional academic lesson and 15-minutes post traditional academic lesson. To increase validity of pre-post comparisons for TOT ratings, researchers observed the same students pre- and post- lesson. This method has been well established in the literature to assess on and off-task behaviors ²⁹. Research staff was trained in this TOT protocol prior to data collection. Once training was complete, the lead researcher and research staff practiced TOT observations in an elementary classroom not involved in the study to establish inter-rater reliability (IRR) between observations of the same students. IRR greater than 90% was considered acceptable agreement.

TOT Measurement Protocol

Two members of the research staff would split the total number of students in the classroom such that each researcher was observing only half of the students. They each observed a separate set of students. Each researcher would sketch out the layout of the room including the identification numbers of the students who they were observing. Researchers listened to a 15-minute recording of five-second “beeps” simultaneously. During this time, each researcher would spend five seconds on each student and mark whether they were on (1)- or off (0)-task. The sound of the beep indicated moving to the next student. Students considered on-task were quietly engaged and following teacher’s instructions or paying attention to the teacher for the entire five seconds. Those considered off-task were not following teacher’s instructions by talking, walking around, laying their head on the desk, or gazing off during the 5 seconds of being observed. On average, each student was observed twenty-two times. This range depended on the number of students in the classroom. A percent TOT score was calculated for each student based on the number of observations and the amount of times the student was noted as on-task.

Data Analytic Strategy

To estimate the main effect of the I-CAN! intervention on TOT, a series of three-level regression models were fit ³⁰. Multilevel modeling corrects for the statistical dependence that characterizes nested data, yielding correct standard errors and, ultimately, permitting unbiased significance tests ³¹. Students were nested within classes and classes within schools, therefore we estimated the effect of treatment at level three (the school-level). To control for differences between districts, they were entered as covariates (three dummy variables) at level three. Effect sizes were calculated as a ratio, with the coefficient for the relevant model parameter in the numerator and the pooled standard deviation at post-test in the denominator ³². The moderating effects of gender, race, SES, and PA were evaluated during I-CAN! lessons in separate multilevel regression models by including cross-level interactions (the product of student-level values for the potential moderators and school-level assignment to treatment) as well as a between-school level three interaction terms (the product of school means on the potential moderators and assignment to treatment) to control for contextual effects ³³. For significant interaction effects involving continuously-measured moderators, strategies were used recommended by Bauer & Curran, (2005) to identify values of the moderator for which the interaction effect differed significantly from zero ³⁴. All analyses were conducted using Mplus ³⁰.

Results

Analyses

Table 2 presents means and standard deviations across the two treatment groups for the outcome variable. Skewness and kurtosis values, residual plots, and histograms to assess normality, linearity, homoscedasticity, and outliers were all examined. All variables were within acceptable ranges ³⁵. Plots and histograms indicated that the statistical assumptions underlying our models were reasonable. To evaluate clustering in the data, we fit the unconditional (i.e., no predictors) three-level model, which partitions total variance into its student-level, class-level, and school-level components ³¹. Variance in the random effects (Table 3) differed significantly from zero, although the coefficients were increasingly smaller at higher-levels of the model (from student to class to school levels). About 77% of the variation in TOT occurred between students, 14% was between classes, and close to 9% occurred between schools. Because there was no difference in TOT between spelling and math schools (p=.79) schools were combined to provide a direct comparison between intervention and control schools.

Table 2.

Means (M) And Standard Deviation (SD) For Time on Task (TOT) Outcome and Moderate to Vigorous Physical Activity (MVPA)

	Condition	n	M	SD
% TOT at pretest	Control	562	85.89	16.83
	I-CAN!	1488	84.32	16.64
% TOT at posttest	Control	644	82.29	18.77
	I-CAN!	1562	86.30	16.26
% MVPA during day	Control	734	6.01	1.99
	I-CAN!	1750	6.69	2.14
% MVPA during lesson	Control	686	3.32	22.47
	I-CAN!	1668	5.89	12.11

Table 3.

Fixed And Random Effects For The Main Effect Analysis

	Unconditional model			Conditional model
	Fixed effects
Predictor	Coefficient	SE	p-value	Coefficient	SE	p-value	Effect size
Intercept	84.42	1.19	0.00	81.36	1.93	0.00
Pretest (Level 1)				0.36	0.04	0.00
Pretest (Level 2)				0.67	0.15	0.00
Pretest (Level 3)				0.96	0.24	0.00
Treatment				5.53	1.73	0.00	.32
District 1				0.77	1.83	0.67
District 2				0.43	1.61	0.79
District 3				−1.53	1.71	0.37
	Random Effect
	Estimate	p-value	ICC	Estimate	p-value	ICC
Level 1	231.31	0.00	0.77	225.71	0.00	.51
Level 2	41.89	0.00	0.14	202.14	0.00	.45
Level 3	26.30	0.01	0.09	18.67	0.00	.04

Main effect of intervention

Students in treatment schools spent significantly more TOT than their counterparts in control schools (β = 5.53, SE =1.73, p < .001); (d = .32) (Table 3). PA during I-CAN! lesson moderated I-CAN!’s effect on TOT (Table 4). The significant cross-level interaction between student-level step count and school-level assignment (β =.11, SE =.04, p =.01) suggests that TOT scores were positively associated with step counts in intervention schools, but negatively associated with step counts in control schools. PA dose when indicated by MVPA did not moderate this relationship (p = .85). Race, gender, and socioeconomic status did not moderate I-CAN!’s effect on TOT.

Table 4.

Fixed And Random Effects For Time On Task Moderation Analysis

			Fixed effect
Measure	Moderator	Predictors	Estimate	SE	p-value
TOT	Step count	Intercept	83.03	3.27	0.00
		Level 1
		Pretest	0.36	0.02	0.00
		Step count	−0.07	0.03	0.03
		Level 2
		Pretest	0.66	0.12	0.00
		Step count	−0.07	0.04	0.09
		Level 3
		Pretest	0.97	0.22	0.00
		Step count	0.11	0.08	0.17
		Treatment	3.44	3.39	0.31
		Cross-level interaction	0.11	0.04	0.01
		Between-school interaction	−0.10	0.10	0.31
			Random effects
TOT	Step count	Level 1 (Step count)	499.71	15.02	0.00
		Level 1 (TOT)	201.40	6.53	0.00
		Level 2 (Step count)	374.78	53.57	0.00
		Level 2 (TOT)	22.18	5.24	0.00
		Level 3 (Step count)	−0.43	0.56	0.45
		Level 3 (TOT)	0.00	0.06	0.98

Discussion

This study was designed to overcome limitations of existing research regarding TOT following a period of active learning ^8–10. This study: (1) utilized objective, accelerometer-derived measures of PA for the entire sample; (2) drew from a large and diverse set of schools; (3) with teacher implemented lessons; and (4) used the appropriate, hierarchical analysis. Results indicated a main effect for the intervention across all participants, with TOT of greater than 85% following active learning. This aligns with findings from similar interventions ²⁹. Moreover, a higher dose of PA was associated with improved TOT during the experimental conditions. The effect only held when it was modeled as a function of the total number of steps during the lesson rather than the time spent in MVPA during the lesson, and applied across both math and spelling content.

A primary aim of this study was to test the potential moderating effect of demographic variables on the effects for TOT. This requires both a large and a diverse sample. In this case, with more than 2,700 participants, 34% were economically disadvantaged, with large groups of Hispanic (32%), White (46%), and African American (9%) sub-groups. In fact, with 9% of the full sample, there were nearly as many African American participants in this study (n=258) as there were in the majority of existing studies of active learning and TOT. Additionally, this analysis is important as schools and teachers differ widely in their practices for addressing student behavior ³⁶. In fact, we found that 14% of the variance in TOT was explained by differences in class-level effects, and 9% was explained by differences in school-level effects. Thus, the present study provides the clearest assessment of student TOT in response to PA to date and results indicated that the change in TOT was consistent across demographic categories (SES, sex or race/ethnicity). Thus, while girls had higher TOT than boys overall, both groups had similar responses to the physically active learning or sedentary lesson and our finding for the positive impact of PA on on-task behavior appears to be robust across student sub-groups. This robust, common response strengthens the argument to use active learning in schools. Not only do these lessons increase MVPA while addressing academic material, they improve attention across all groups of children.

The interaction between condition for PA and TOT was another key finding and it mirrors, in part, the findings of Szabo-Reed and colleagues (2017). There was a positive relationship between PA and the acute change in TOT in the intervention condition. That is, a higher dose of PA during the lesson (number of steps/lesson) were associated with greater levels of TOT following the active lesson, but a reduction in TOT following the sedentary lesson. However, unlike the earlier study²², this interaction did not occur when the data were modeled with time spent in MVPA. It may be that the hip placement of the accelerometer is registering fidgeting or other small movements during the sedentary control lesson as an increase in steps. If so, it is not surprising that “steps” in the control lesson were associated with reduced TOT. Thus, the higher level of measured PA may reflect some level of impulsivity in these children. While speculative, this would explain why higher levels of PA in the control, sedentary condition were associated with lower TOT and it provides an interest direction to research on PA and attention. It may also be that the activity for controls students was in response to low levels of TOT as teachers may seek to manage off-task students with a change in seating or movement to a self-control area. However, such an event is unlikely as it would have been noted in our observations. Such efforts were not reported in our observations and in the majority of cases off-task behavior reflected distracted rather than disruptive students. That said, this relationship requires further study. The positive relationship with dose and TOT for active learning has implications for the design of these interventions. The failure to find a relationship with MVPA runs counter to the recent work of Szabo-Reed and colleagues (2017)²². However, the former study utilized observations of activity intensity (SOFIT) in which walking was included as a form of MVPA. This mixes low intensity with moderate intensity movement. In contrast, the present study utilized an objective measure (accelerometer). Thus, it may be that both studies were reflecting the benefit of dose at lower intensities. While intriguing, there is a clear need for randomized control trials that vary the intensity of active learning to tease apart the dose response.

Although the present findings extend the existing literature, some limitations should be noted. The TOT observations were conducted before and immediately after the 10–15 minute lesson. The benefit beyond 15 minutes is not clear. Additionally, we did not collect data on diagnosis of attention deficit or other learning challenges. Given the findings within our control condition, this would be an important component of future research. Third, while the research staff were not present during the active learning lesson, they were not blinded to condition. Given the magnitude of the study and the school-level assignment to condition, this was not possible. While this opens the possibility of reactance on the part of the research staff, the similarity of these findings to the existing literature – much of which used research staff that were blinded – suggests that this had little impact on these data. Finally, this study was not designed to assess potential mechanisms for the impact of activity on TOT. Given the consistency of results across studies, future research should begin to include tests of possible mechanisms for the effect.

Despite these limitations, this study breaks new ground by being the first adequately powered study to investigate active learning and TOT across critical sub-groups with an objective measure of PA. In addition, this study utilized teacher-implemented lessons rather than researcher-implemented lessons in an effort to increase the generalizability of these results. This design provides the strongest test of this relationship to date and supports the robust nature of improved TOT following periods of active learning. The positive relationship between the number of steps and TOT supports efforts to implement school policy to incorporate more opportunities for PA during the elementary school day.

HIGHLIGHTS.

• Drew from a large and diverse set of elementary schools

• Utilized objective, accelerometer-derived measures of physical activity

• Teachers implemented the active lessons in the classrooms

• More physical activity during the active lessons was associated with improved on-task behaviors