Measuring Preschool Learning Engagement in the Laboratory

Abstract

Learning engagement is a critical factor for academic achievement and successful school transitioning. However, current methods of assessing learning engagement in young children are limited to teacher-report or classroom-observation, which may limit the types of research questions one could assess about this construct. The current study investigated the validity of a novel assessment designed to measure behavioral learning engagement among young children in a standardized laboratory setting and examined how learning engagement in the laboratory relates to future classroom adjustment. Preschool-aged children (N=278) participated in a learning-based Tangrams task and Story sequencing task and were observed based on 7 behavioral indicators of engagement. Confirmatory factor analysis supported the construct validity for a behavioral engagement factor comprised of six of the original behavioral indicators: attention to instructions, on-task behavior, enthusiasm/energy, persistence, monitoring progress/strategy use, and negative affect. Concurrent validity for this behavioral engagement factor was established through its associations with parent-reported mastery motivation and pre-academic skills in math and literacy measured in the laboratory, and predictive validity was demonstrated through its associations with teacher-reported classroom learning behaviors and performance in math and reading in kindergarten. These associations were found when behavioral engagement was observed during both the nonverbal task and verbal story-sequencing tasks and persisted even after controlling for child minority status, gender, and maternal education. Learning engagement in preschool appears to be successfully measurable in a laboratory setting. This finding has implications for future research on the mechanisms that support successful academic development.

Introduction

Children’s learning behaviors play an important role in promoting successful academic and school outcomes (Appleton, Christenson, & Furlong, 2008; Kagan, Moore, & Bredekamp, 1995). Learning engagement in early childhood may be particularly critical, as early engagement may predict both future levels of engagement (Ladd & Dinella, 2009) and trajectories of academic growth through elementary school (Bulotsky-Shearer & Fantuzzo, 2011; Li-Grining, Votruba-Drzal, Maldonado-Carreño, & Haas, 2010; McClelland, Morrison, & Holmes, 2000). Understanding the development of learning engagement and the mechanisms that support it may in turn provide important information about processes of school adjustment. In order to assess these mechanisms, it is necessary to have instruments that can validly measure learning engagement prior to school entry in diverse contexts and through diverse means. However, current methods of measuring learning engagement in early childhood are primarily restricted to teacher-report or classroom observation, each of which may be influenced by the reporter or the context and may be restrictive for investigators conducting research within a laboratory setting. A laboratory measure of learning engagement may therefore help broaden our understanding of the individual psychological processes and mechanisms supporting this construct. The goal of this study is to examine the validity of a novel measure designed to assess behavioral learning engagement in young children and investigate whether laboratory measured learning engagement in preschool is associated with classroom measures of school adjustment in kindergarten.

Engagement during learning, variously labeled by such terms as school engagement, approaches to learning, and learning engagement, can be conceived of as a multi-dimensional construct that operates at the affective, cognitive, and behavioral levels, although most work has focused on the behavioral aspects (Fredricks, Blumenfeld, & Paris, 2004). Affective engagement describes how much a child likes school and is interested in learning, whereas cognitive engagement describes a child’s effort and investment as well as deep, strategic thinking. These levels of engagement tend to be internal processes that are difficult to observe.

In contrast, behavioral engagement broadly refers to observable actions, particularly those that denote active participation and focused involvement. In early childhood, these behaviors may be characterized by focused on-task behavior, attention during instruction, rule adherence, and the contribution of questions or observations at appropriate times. As children advance through school, initiating active involvement, participating in extra curricular learning activities, and remitting work punctually may also become important indicators of behavioral engagement (Mahatmya, Lohman, Matjasko, & Farb, 2012). Although these three dimensions of learning engagement may be distinct theoretically, they are often difficult to tease apart empirically, as behavioral engagement can also encompass the behavioral manifestations of engagement at the affective and cognitive levels in the form of expressed enjoyment, enthusiasm, and strategic behavior. Behavioral engagement is therefore the broadest and most readily measured aspect of engagement and may be particularly useful when studying the development of learning processes in children.

Learning engagement is often compared to motivation. These two processes are related but distinct constructs, with motivation conceived of as more abstract and engagement as more concrete (Appleton et al., 2008; Finn & Zimmer, 2012; Newmann, Wehlage, & Lamborn, 1992). In other words, whereas motivation refers to internal drives, engagement is the behavioral, cognitive, and affective result of these drives (Reeve, 2012). As Appleton and colleagues (2008) conclude, motivation is necessary but not sufficient for engagement to occur. As such, one would expect children’s mastery motivation, defined as the psychological drive to independently master a skill or solve a problem (Morgan, Harmon, & Maslin-Cole, 1990), to be associated with their engagement in the learning process. Children who are more driven to learn and complete challenging tasks should be more likely to derive pleasure from the task, exert strategic effort, and focus their behavior into productive, task-focused engagement.

Learning engagement is also a critical factor in children’s successful transition to school. According to The National Education Goals Panel, a child’s approach toward learning is one of the five key abilities most important for early learning and development (Kagan et al., 1995). It is necessary for children to adopt appropriate attitudes and habits that will allow them to benefit from their cognitive competencies and actively apply their acquired skills. Behavioral engagement specifically may increase the amount of time children interact with learning materials in meaningful ways and provide more opportunities for learning to occur. Behaviorally engaged children may also develop more positive relationships with teachers, who may in turn provide greater support and instruction. Behavioral learning engagement is therefore an important process for researchers to understand and accurately measure during early childhood.

Behavioral engagement may also help reinforce subsequent engagement. For example, behavioral engagement may result in greater opportunities for success, which may in turn promote feelings of pleasure, pride, and social camaraderie with teachers and classroom peers (affective engagement) as well as more effortful focus and challenge-seeking (cognitive engagement). Positive feelings and increased effort may in turn facilitate greater behavioral engagement. In support of these hypotheses, Ladd and Dinella (2009) found that behavioral and affective engagement fostered growth in one another through early elementary school. This is particularly important given that behavioral and affective engagement have both been associated with a greater school success throughout the later school years (Connell, Spencer, & Aber, 1994; Furrer & Skinner, 2003; Marks, 2000; Reschly & Christenson, 2006; Voelkl, 1997). Thus, early childhood may be a particularly important developmental period in which to examine learning engagement, as children’s engagement at or before their transition to school may determine not only their initial adjustment to school but also initiate cycles of future engagement and academic success.

Given the importance of learning engagement, it is critical that we have a variety of appropriate methodologies to properly and thoroughly measure it. Currently, the most common method of measuring learning engagement in early childhood is through teacher-report questionnaires. One commonly used set of questionnaires is The Learning Behavior Scale (LBS; McDermott, Green, Francis, & Stott, 1999) and the related Preschool Learning Behavior Scale (PLBS; McDermott, Green, Francis, & Stott, 2000). The PLBS has demonstrated robust correlations between leaning behaviors and achievement outcomes within the preschooler year (McDermott, Rikoon, Waterman, & Fantuzzo, 2012; Vitiello, Greenfield, Munis, & George, 2011) as well as longitudinally through early elementary school (McDermott, Rikoon, & Fantuzzo, 2014). A more recent measure from the same group of researchers is the Learning-to-Learn Scales (McDermott et al., 2011), which was designed to more accurately measure developmental change and better distinguish different aspects of engagement. This measure uncovered seven subscales—strategic planning, effectiveness motivation, interpersonal responsiveness in learning, vocal engagement in learning, sustained focus in learning, acceptance of novelty and risk, and group learning. These subscales conceptually fall within the broad dimensions of behavioral, cognitive-behavioral, and affective-behavioral engagement and were empirically associated with better academic skills.

Several other teacher-rating scales exist to measure learning engagement and related constructs, which have also demonstrated positive relations with achievement. For example, the work related skills subscale of the The Cooper-Farran Behavioral Rating Scales measured during kindergarten demonstrated predictive associations with academic achievement through the second (McClelland et al., 2000) and sixth grades (McClelland, Acock, & Morrison, 2006). Another method of assessing engagement is through classroom observations conducted by trained researchers. Research using these observational methodologies have also supported the association between engagement and success. For example, using The Child Observation in Preschool Protocol, Nesbitt, Farran, and Fuhs (2015) found that learning behaviors in the spring and winter of preschool not only predicted later math and reading, but also mediated the effects of executive functions on achievement.

Both teacher questionnaires and classroom observations provide important information about children’s behavioral engagement within the ecologically-relevant classroom setting. However, each method has its own set of constraints. Specifically, teacher-ratings are subject to some amount of bias and may more accurately assess teacher’s perceptions of children. This may be particularly problematic if all other study variables, including measures of academic performance, are also derived from the same teacher’s report. Although experimenter ratings are subject to bias as well, a multi-informant framework may help eliminate systematic error associated with a single rater’s responses. Moreover, for researchers whose work is laboratory based, adding a teacher questionnaire or classroom observation may be challenging. This may be particularly true for researchers who work with broad community samples that would require working in many schools at once. Finally, when assessing learning engagement among preschool children specifically, these methods may not work well for children who do not attend a formal pre-kindergarten program. Even among children who do attend preschool or daycare, the great variation in academic rigor among these programs may make it difficult to compare teacher reports or classroom observations across children from different programs.

A laboratory assessment would help minimize the potential problems associated with teacher-report and classroom observations and may provide a nuanced version of engagement that is different from what is measured in the classroom. For example, measuring engagement in the lab may limit the influence of any positive or negative associations that a child may have with his current classroom. As children may be engaged in school (e.g., enjoy participating and being social) while unengaged in academic tasks or vice versa (Stipek, 2002), a laboratory measure may isolate task learning engagement from classroom engagement more generally and may help the field draw more precise conclusions about the mechanisms that support school adjustment and achievement. Finally, a laboratory measure of learning engagement would help further research investigating the relations between engagement and other constructs more easily measured in a lab or home setting, such as parenting behavior, executive functioning, and emotion regulation. Investigating how these processes may influence engagement may eventually guide the development of early interventions to help support early engagement and subsequent school success.

Extant laboratory tasks measuring engagement-related processes such as mastery motivation and on-task behavior are generally insufficient in providing a rich depiction of children’s engagement. For example, motivation in young children is classically operationalized by persistence on a laboratory task, often measured by total or proportion of time spent occupied with that task (Deci, Driver, Hotchkiss, Robbins, & Wilson, 1993; Frodi, Bridges, & Grolnick, 1985; Morgan et al., 1990). Although time on-task during a challenging laboratory task at age 3 was predictive of future academic achievement in kindergarten (Mokrova, O’Brien, Calkins, Leerkes, & Marcovitch, 2013), measuring on-task behavior in this way obfuscates any differences in engagement when task difficulty changes and therefore ignores potentially important variation in effort and reaction to challenge. This methodology also fails to capture children’s emotional responses to success or the energy with which children conduct their actions. Furthermore, these simple coding systems may overlook instances of perseveration, whereby a child may be casually task focused but engaged in repetitive or thoughtless behaviors.

Observable Indicators of Learning Engagement

In order to contend with these issues, we developed a laboratory-based, observational learning engagement assessment. The behavioral indicators we selected were derived by consulting current empirical literature on learning behaviors in young children and broad theories of learning engagement (e.g., Appleton et al., 2008; Birch & Ladd, 1997; Connell & Wellborn, 1991; Fredricks et al., 2004), as described below. The final coding structure included seven indicators of engagement: Attention to instruction, on-task behavior, persistence, monitoring and strategy use, enthusiasm and energy, positive affect, and negative affect.

Attention to instructions

Paying attention to an instructor while instructions are being given is both a prosocial behavior and a necessary step for the successful completion of a task. However, this may be particularly challenging for young children: Over 45% of kindergarten teachers reported that children entered their classroom with difficulty following directions (Rimm-Kaufman, Pianta, & Cox, 2000). Paying attention requires children to recruit several regulatory skills that are rapidly developing during this stage of development (Blair & Diamond, 2008). Specifically, they must inhibit any impulse to interact with task materials before instructions are given, speak over the instructor, or attend to other task-irrelevant stimuli while sustaining focus on the instructor and task relevant stimuli. These attentional skills are crucial for scholastic and academic success (Duncan et al., 2007; McClelland, Acock, Piccinin, Rhea, & Stallings, 2013). Additionally, attentiveness to one’s instructor is a sign of compliance, which in itself is discussed as an indicator of behavioral engagement among both older (Finn, Pannozzo, & Voelkl, 1995) and younger children (Birch & Ladd, 1997). Compliant and cooperative behavior is a clear demonstration of willingness or ability to engage in the learning task. As such, paying attention to instructions is an important indicator of learning engagement in young children.

On-task behavior

On-task behavior is perhaps the most basic form of behavioral engagement: A child who is on-task is inherently engaged, even if the task-directed behavior is only mildly effortful. During independent learning tasks, remaining on-task requires many of the same skills needed to successfully attend to instructions, such as attentional focus and behavioral compliance, but involves less social salience. As working independently was another prevalent problem reported by kindergarten teachers (Rimm-Kaufman et al., 2000), remaining on-task during these learning contexts seems to be a similarly challenging task for young children and may therefore be an important source of variability in behavioral engagement at this developmental period.

Persistence

Persistence may be conceived as the maintenance of on-task behavior and effort even when a task is particularly challenging or boring. As such, it has been conceptualized as a component of effortful control (Eisenberg et al., 2004) as well as a way of operationalizing mastery motivation in young children (Morgan et al., 1990). Persistence in a learning context is a conceptually integral element of engagement and an important facilitator of successful academic outcomes (Andersson & Bergman, 2011; McClelland et al., 2013; McDermott et al., 2014). In order to learn, children must persevere when challenged and remain focused even when the educational task is no longer inherently enjoyable. As such, this is a key element of behavioral engagement.

Monitoring and strategy use

Strategically and flexibly selecting strategies, asking meaningful questions, and accepting help when necessary are all indicators of cognitive engagement or self-regulated learning (Cleary & Zimmerman, 2012; Pintrich & De Groot, 1990). According to Self-Regulated Learning Theory, children must be able to self-monitor, self-evaluate, and respond to their progress while working on learning tasks (Zimmerman, 1990). Although the cognitive processes that underlie such strategic planning become much more advanced during the school years (Diamond, 2013; Zelazo, 2015), younger children are also capable of engaging in self-regulated learning (Pintrich & Zusho, 2002). Indeed, behavioral manifestations of these cognitive processes are reflected in some preschool questionnaires, such as the Learning Strategy subscale of the PLBS (McDermott et al., 2012) and the Strategic Planning subscale of the LTLS (McDermott et al., 2011). The observable manifestations of a planful and flexible approach to learning tasks should therefore add important information about a child’s level of engagement, as it highlights the quality of children’s learning behavior when on-task.

Enthusiasm and energy

Enthusiasm has been defined as an important element of affective engagement in several classic conceptualizations of engagement (Connell & Wellborn, 1991; Stipek, 2002), as it connotes a level of enjoyment on the part of the child. However, enthusiasm may also be a sign of behavioral engagement as well: Signs of enthusiasm such as sitting up straight rather than slouching and expressing eagerness rather than reluctance are all behavioral indicators of engagement. Furthermore, as enthusiasm may be more observable than other forms of affective engagement, such as interest, it may be a particularly useful metric upon which to evaluate a child’s level of affective-behavioral learning engagement.

Positive affect

A child’s positive emotional reaction to learning and general attitude while completing learning tasks is also an important indicator of affective-behavioral engagement (Connell & Wellborn, 1991; Jimerson, Campos, & Greif, 2003). Developing positive interactions with learning is an important task during early childhood (Mahatmya et al., 2012), and adopting a favorable rather than negative attitude about school is an early and important aspect of engagement (Ladd, Buhs, & Seid, 2000). Although enjoyment and interest may be internal, behaviorally manifested positive affect while engaged in a task, such as excitement when presented with a novel puzzle or pride following a small success, can inform researchers about these processes. However, despite the positive associations demonstrated between children’s school-liking and school adjustment (Ladd et al., 2000; Ladd & Dinella, 2009), expressed positive affect may have a more complicated relation with overall engagement, as it may also disrupt focus and impede on-task behavior. For example, Denham and colleagues (2012) found that more emotionally positive 4-year-old children were rated by their teachers as concurrently less engaged and suggest that highly positive behaviors may be considered boisterous and off-task. Thus, it is less clear whether observed positive affect will cohere with the other indicators of observed learning engagement.

Negative affect

Some engagement theorists consider affective engagement to encompass all affective reactions to learning, including negatively valanced emotions such as sadness, anxiety, and frustration (Connell & Wellborn, 1991). Additionally, behavioral manifestations of negative affect may also be directly conflictive with behavioral engagement. For example, the expression of frustration may include disruptive behaviors, such as pushing task materials away, which are in themselves considered indicative of poor behavioral engagement (Finn et al., 1995). Negative affect may also take on other disengaged manifestations, such as folding one’s arms or complaining. Although functional theories of emotion argue that negative emotions like anger can be motivating (Keltner, Haidt, & Shiota, 2006), more intense, expressed negative affect may be counterproductive for adaptive engagement with learning.

Learning Engagement Tasks

In addition to a more nuanced coding scheme, appropriate laboratory tasks that mimic learning contexts are also needed. Although some researchers have developed more complex coding schemes to measure children’s engagement, the tasks during which children were observed were not learning based. For example, Berhenke, Miller, Brown, Seifer, and Dickstein (2011) measured children’s emotions and task behaviors while children were engaged during impossible puzzles. As these puzzles were not solvable in the allotted amount of time or designed to include a learning component, they are not analogous to educational contexts and therefore do not measure engagement with learning specifically.

We designed tasks to center around a mathematics-oriented and literacy-oriented learning goal to mimic activities that typically occur in a classroom. The Tangrams task, which may help teach children about spacial relationships, taps children’s visuo-spacial and problem solving skills. Children must be able to match shaped blocks into pictures of shapes on a template, recognize when a shape does not correctly fit, identify when a shape must be turned or flipped, and reason how to put two or more shapes together in order to make a bigger shape. Story-sequencing is used in classrooms to help teach reading comprehension, story telling, and temporal ordering. Children must pay close attention to the pictures presented and find clues to help them determine the order of story events (e.g., see that a glass is full in one picture but empty in another). Finally, children must be able to articulate the basic premise of the story by focusing on key elements of each story card. Both the Tangrams and Story tasks require children to pay close attention to task materials, think critically, and monitor their performance, just as they would need to in a classroom setting. Children must also be able to attend to and retain information, as each task involves an initial teaching component, similar to classroom instruction, that provides children with information to help them solve subsequent trials (e.g., how to flip a parallelogram, how to identify key elements in story cards). Finally, as each task involves increasingly difficult trials, children should eventually get to a task that poses a challenge and therefore requires greater effort and persistence.

This cross-domain paradigm is another benefit of the current design, as many current measures of engagement do not specify the target or source of engagement but rather ask broad questions about school in general. This does not allow the respondent or observer to specify differences in engagement across academic domains (i.e., mathematics, literacy) (Fredricks & McColskey, 2012). By investigating engagement across two tasks, we hoped to determine whether observable learning engagement operates similarly across academic domains.

Current Study

The goal of the current study was to investigate a novel laboratory measure of learning engagement and assess both the concurrent and predictive validity of this measure among preschool aged children by assessing associations with motivation, academic achievement, and learning behaviors measured in the classroom. We hypothesized that within task, behavioral learning engagement would cohere into a single behavioral engagement factor. We also compared the fit of a single, cross-domain factor model of learning engagement (i.e., indicators from both tasks loading onto a single factor) to a domain specific, two-factor model in which the learning engagement in Story factor and and learning engagement in Tangrams factor were allowed to correlate. Furthermore, we hypothesized that preschool learning engagement, during both the Tangrams and Story tasks, would be concurrently related to children’s pre-academic skills and parent’s report of children’s mastery motivation and longitudinally associated with children’s classroom learning behaviors and academic performance in both math and reading during kindergarten. We specifically tested kindergarten academic performance and learning behaviors through teacher report to compare scores from this laboratory measure with children’s functioning in a real-world learning setting.

Method

Participants

This study was a part of a larger longitudinal study, the School Transition and Readiness (STAR) Project, investigating trajectories of early academic success. Children (N = 278) and their primary caregivers from a mid-sized southeastern city of the USA participated. During the first wave of data collection, children ranged in age from 46 to 70 months (M = 56.38, SD = 4.69), were approximately split on sex (54.9% female), and were racially diverse (30% African American, 59.2% European American, 1.8% Asian, 9% multi-racial; 6.9% Hispanic). Primary caregivers (267 mothers, 10 fathers) were on average 35 years of age (SD = 6.35), and 28.8% had completed a graduate degree, 31.8% completed a 4-year college degree, 10.8% completed a 2-year college degree, 18.1% completed some college, 10.5% completed a high school degree or less. Average income-to-needs ratio, calculated by dividing the total family income by the appropriate poverty threshold, was 2.11 (SD = 1.41). Approximately one year following their first lab visit, 249 families (M = 79.8 months old, SD = 3.86) returned for follow-up assessments.

During this second wave of data collection, 243 parents provided permission to contact their children’s teachers. One hundred and fifty-five teachers completed questionnaires for 222 children. The majority of teachers reported on one or 2 children (90%); 16 teachers (10%) reported on 3–5 children. Sensitivity analyses indicated that nesting within classroom did not affect teacher ratings of individual children. Two children were held back in school for one year, so kindergarten teacher data was collected for these children two years after their initial laboratory assessment.

Procedure

Families were scheduled for laboratory visits that lasted approximately 2 hours in length and included several tasks assessing self-regulation and social-cognitive understanding. During the visit, children were videotaped completing several tasks with an experimenter while mothers completed a series of questionnaires. Families were invited back to the lab approximately one year after the initial visit. At both visits, children were first administered The Woodcock Johnson III scales followed by Tangrams and then Story. Between The Woodcock Johnson III and the learning engagement tasks, children were asked to put on a set of heart rate stickers and an electroencephalogram net as well as complete two short baseline tasks (i.e., a two minute neutral video and a one minute statue game). During the second point of data collection, parents were also asked permission to contact children’s kindergarten teacher. Teachers were contacted via email and asked to complete a series of questionnaires using Qualtrics in the spring semester of the kindergarten year. Families were compensated $50 for their time at the pre-kindergarten visit and $75 at the kindergarten visit, and teachers were compensated $75. Children also selected a small toy to take home at the completion of each visit.

Preschool Measures

Demographics

Primary caregivers provided information about their family, including family income, mother’s highest level of education, and child’s gender, race and ethnicity. Child minority status was coded as white non-Hispanic children (N = 164) and all other children (N = 113).

Learning engagement

Children were observed during two learning-focused tasks, a Tangrams task and a Story-sequencing task, designed to mimic a classroom experience. Each task comprised a short tutorial followed by a sequence of increasingly difficult trials administered by an experimenter. During Tangrams, which lasted 10 minutes or until the most difficult puzzle was completed, children were shown how to fit wooden shapes into 2-dimensional pictures of shapes on a laminated piece of paper, how to appropriately flip a parallelogram, and how to combine these blocks to make larger shapes in the absence of internal guiding lines. Children were then presented with puzzles of increasing difficulty (i.e., missing an increasing number of internal lines) and instructed to ask for help if needed. The Story task followed a similar format whereby the experimenter demonstrated how to put picture cards in order from beginning, middle, to end in order to make a complete story and then instructed the child to complete subsequent stories on his or her own. Stories increased in difficulty in the following order: one story composed of three cards, two stories mixed together composed of three cards each, and one story composed of four cards. This task ended after 8 minutes elapsed or until the most difficult story had been completed. For both tasks, children were expected to work autonomously, and experimenters only provided minimal help when requested or if child appeared to be struggling for greater than 15 seconds. Redirection was provided if the child was off-task for over 15 seconds or tried to get out of his or her seat. All children took the full 10 minutes on the Tangrams task, as no child made it through the last puzzle. Approximately 65% of children finished all Story trials at or within 8 minutes.

Children’s behavior was coded on seven dimensions, each rated on a Likert scale ranging from 1 (no indication of behavior) to 5 (high indication of behavior) (see Appendix for full coding scheme). Attention to instruction measured how attentive children were during the initial description of task and other interactions with the experimenter. A low scoring child might handle the task materials in task-irrelevant ways while the experimenter was speaking or refuse to answer the experimenter’s questions. A high scoring child would focus attention on the experimenter and relevant materials and actively respond to the experimenter by nodding. On-task behavior was based on children’s maintained focus on task materials, task-relevant actions, and the amount of time children remained task-oriented. Energy/enthusiasm assessed how interested or eager children appeared to be in the task and how energetic versus passive they were while engaging. Low scorers might move sluggishly or work while slumped in seat or with head in hand whereas high scorers might eagerly begin each new trial or express interest (e.g., “Can I try?”). Persistence measured whether children maintained engagement even when the task became demonstrably difficult. Persistent children tried to figure out the problem themselves without deferring to the experimenter and remained fully committed throughout. Monitoring/strategy use assessed how flexible children were in their strategy use and how aware they appeared to be of specific problems or progress. Strategic behaviors included, focusing on specific problems rather than aimlessly moving task materials around the table, checking work before claiming completion, asking for help on a specific problem, and using feedback in constructive ways. Finally, positive affect assessed the amount and intensity of physical and verbal cues of pleasure and enjoyment, whereas negative affect assessed frustration, anger annoyance, and sadness. Behavioral coding was conducted independently for the Tangrams and Story.

Two trained experimenters, a graduate student and a research assistant, completed all coding. Coders were trained under the initial supervision of a principal investigator by watching 4 Story and 6 Tangrams videos together and coming to a consensus on all codes. To remain consistent, coders reconvened to jointly code 4 additional videos each of Story and Tangrams throughout the duration of coding. Reliability was calculated on 42 double rated cases. For Tangrams, intraclass correlation coefficients (ICC’s) for individual ratings ranged from .70 to .84 (mean ICC = .79). For Story, ICC’s ranged from .78 to .91 (mean ICC = .85). (See Table 1).

Table 1.

Descriptive Statistics

	N	ICC	Min	Max	Mean	SD	Skew (SE)	Kurtosis (SE)
Learning Engagement: Tangrams
Attention to instructions	278	.70	1	5	4.22	0.86	−0.95 (.15)	0.38 (.29)
On-task behavior	278	.77	1	5	4.28	0.87	−1.09 (.15)	0.57 (.29)
Enthusiasm/Energy	277	.79	1	5	3.31	0.77	0.17 (.15)	−0.07 (.29)
Persistence	278	.84	1	5	3.87	1.06	−0.72 (.15)	−0.17 (.29)
Monitoring progress/Strategy use	278	.83	1	5	3.86	0.87	−0.52 (.15)	0.00 (.29)
Positive Affect	277	.82	1	5	2.35	0.92	0.62 (.15)	0.39 (.29)
Negative Affect	277	.82	1	5	1.77	0.89	1.10 (.15)	0.76 (.29)
Learning Engagement: Story
Attention to instructions	278	.83	1	5	4.23	0.89	−1.15 (.15)	1.12 (.29)
On-task behavior	278	.90	1	5	4.30	0.94	−1.35 (.15)	1.28 (.29)
Enthusiasm/Energy	277	.81	1	5	3.14	0.81	0.11 (.15)	0.23 (.29)
Persistence	278	.91	1	5	3.95	1.23	−0.81 (.15)	−0.59 (.29)
Monitoring progress/Strategy use	277	.91	1	5	3.96	1.08	−0.68 (.15)	−0.52 (.29)
Positive Affect	277	.78	1	5	2.17	0.94	0.85 (.15)	0.59 (.29)
Negative Affect	277	.82	1	5	1.59	0.95	1.73 (.15)	2.52 (.29)
Mastery Motivation	275	n/a	2.06	3.78	3.00	0.29	0.08 (.15)	0.34 (.29)
Pre-math skills	278	n/a	72	149	109.17	11.99	−0.07 (.15)	0.27 (.29)
Pre-literacy skills	278	n/a	70	183	105.84	13.93	1.42 (.15)	6.74 (.29)
Classroom Learning Behaviors	222	n/a	−0.38	1	0.64	0.31	−1.30 (.16)	1.22 (.33)
Math Performance	222	n/a	1	5	3.61	.94	−0.19 (.16)	−0.25 (.33)
Reading Performance	222	n/a	1	5	3.43	1.20	−0.29 (.16)	−0.78 (.33)

Mastery motivation

Mothers completed the Instrumental Competence Scale for Children (COMP; Adler & Lange, 1997), an 18-item questionnaire designed to assess child motivation and mastery orientations. Items indicated the degree to which a child exhibited specific behaviors (e.g., “finishes tasks and activities,” “actively uses resources for help and information,” and “shows exploratory behavior”) on a 4-point Likert scale ranging from 1 (Strongly Disagree) to 4 (Strongly Agree). Prior maternal reports on this measure demonstrated good test-retest reliability over a six-week interval (r = .77, p < .001) (Lange, Mackinnon, & Nida, 1989), and a teacher version of this scale predicted children’s reading and math achievement in kindergarten (Howse, Calkins, Anastopoulos, Keane, & Shelton, 2003). Total scores were calculated by averaging items, with higher scores reflecting greater mastery motivation. In the current study, Cronbach’s α for the total score was .75.

Pre-academic skills

Two scales of The Woodcock Johnson III (WJ III; Woodcock, McGrew, & Mather, 2001) were administered to measure academic skills before the start of formal schooling. Pre-literacy was measured by The Letter-Word Identification subscale, which assesses symbolic learning and reading identification skills, and pre-math was measured by the Applied Problems subscale, which measures analytic and mathematic problem solving. Problems of increasing difficulty were administered until both basal and ceiling levels were obtained. Normed standard scores were calculated based on child’s age at administration (Woodcock Johnson III Compuscore, v. 2.0; Schrank & Woodcock, 2003).

Kindergarten Measures

Classroom learning behaviors

The Learning Behavior Scale [LBS: (McDermott, 1999; Rikoon, McDermott, & Fantuzzo, 2012) was used to measure children’s approaches to learning (e.g., competence motivation, discipline/persistence, cooperation, emotion control). Kindergarten teachers indicated the prevalence of learning-related behaviors through 29 items, measured on a 3-point Likert scale. The scale has demonstrated good internal consistency, convergent and divergent validity, and predictive validity regarding children’s school adjustment (McDermott, 1999; Rikoon et al., 2012). In the current study, the total mean score was calculated (Cronbach’s α = .91) and reversed such that higher scores indicated more adaptive classroom behavior.

School performance

Teachers evaluated children’s school performance on The Mock Report Card (MRC; Pierce, Hamm, & Vandell, 1999). Teachers were asked to report on children’s reading, oral language, written language, math, social studies, and science performance on a 5-point scale (1=below grade level, 5=excellent/above grade level). Previous research has identified math and reading as the best indicators of academic functioning in early elementary school (Pierce et al., 1999) and has focused exclusively on these two domains (Pierce, Bolt, & Vandell, 2010). Math and Reading scores on the MRC have also demonstrated large correlations with standardized measures of achievement (Pierce et al., 2010). As the math and reading domains were also most analogous to our preschool measures of achievement on the WJIII, we narrowed our focus to these two domains in the current study.

Analyses

Focal analyses were conducted using Mplus (Version 7.31; Muthén & Muthén, 2015). Parameters were estimated using full information maximum likelihood (FIML) to account for missing data. Three primary caregivers failed to complete the COMP scale, and one child did not receive scores for enthusiasm/energy, positive affect, negative affect, and strategy use due to a problem with video recording, and kindergarten teacher data was absent for 58 children.

Confirmatory factor analysis (CFA), fit to include ordinal indicators, was used to test the construct validity of the behavioral learning engagement factor during the Tangrams and Story tasks. Model fit was evaluated by examining the root mean square error of approximation (RMSEA), and the comparative fit index (CFI). RMSEA values less than or equal to .08 are generally considered reasonable, whereas values under .5 are considered good. Values equal to or above .95 is considered excellent for the CFI. To test whether each task represented a distinct dimension of learning engagement, we compared a two-factor model (i.e., Tangrams and Story as separate, but correlated factors) with a one-factor model combining both tasks into one learning engagement construct.

We then estimated a series of structural models to test concurrent and predictive validity. Each model included learning engagement during preschool with pathways to endogenous preschool (concurrent validity) or kindergarten (predictive validity) outcomes. Analyses were conducted separately for Tangrams and Story in order to assess whether each task was useful in predicting child outcomes on their own. In all models, endogenous variables were controlled for basic demographic information, including maternal education, child sex, and minority status. These models were evaluated by the same fit indices used to assess the CFA models.

Results

Bivariate Correlations

Descriptive statistics for all study variables can be viewed in Table 1. Correlations among the eight observed learning engagement behaviors were small to moderate in size. All significant associations were positive, except associations with negative affect, which were all negative. In Tangrams, positive affect was not significantly correlated with persistence or negative affect. Significant correlations ranged from .16 to .73, and the strongest associations were between persistence and both on-task behavior and monitoring/strategy use (see Table 2). Associations among learning engagement indicators were all significant during the Story paradigm (min |r| = 0.15, max |r| = 0.76, p < .05, See Table 2). Similar to the Tangrams task, the strongest associations were between on task behavior and both persistence and monitoring/strategy use, and the weakest correlations were between positive affect and both on-task behavior and negative affect.

Table 2.

Correlations Among All Study Variables

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22
1 Gender	–
2 Maternal education	−.02	–
3 Minority status	.02	−.25**	–
4 Attention to instructions (S)	.18**	.21**	−.23**	–
5 On-task behavior (S)	.13*	.20**	−.28**	.61**	–
6 Enthusiasm/Energy (S)	.17**	.18**	−.17**	.52**	.57**	–
7 Persistence (S)	.10	.27**	−.25**	.60**	.76**	.60**	–
8 Monitoring/Strategy use (S)	.10	.29**	−.33**	.57**	.72*	.56**	.82**	–
9 Positive affect (S)	.17**	.01	.03**	.28*	.15**	.55	.24**	.26**	–
10 Negative affect (S)	−.15*	−.14*	.12	−.38**	−.54**	−.47**	−.66**	−.54**	−.18**	–
11 Attention to instructions (T)	.11	.19**	−.21**	.47**	.40**	.31**	.39**	.45**	.12*	−.28**	–
12 On-task behavior (T)	.15*	.14*	−.11	.41**	.54**	.30**	.46**	.43**	.11	−.33**	.50**	–
13 Enthusiasm/Energy (T)	.07	.07	−.08	.26**	.23**	.50**	.31**	.33**	.35**	−.27**	.35**	.38**	–
14 Persistence (T)	.12*	.11	−.07	.35**	.44**	.28**	.48**	.45**	.11	−.45**	.45**	.73**	.45**	–
15 Monitoring/Strategy use (T)	.11	.17**	−.18**	.34**	.40**	.25**	.42**	.45**	.09	−.29**	.48**	.57**	.41**	.67**	–
16 Positive affect (T)	.14*	−.09	−.05	.15*	.02	.29**	.06	.15*	.53**	−.02	.16**	.02	.48**	.13*	.18**	–
17 Negative affect (T)	−.08	−.05	−.05	−.22**	−.29**	−.18**	−.34**	−.26**	−.16**	.44**	−.32**	−.41**	−.34**	−.49**	−.39**	−.11	–
18 Mastery motivation (Pre-k)	.12	.21**	−.11	.28**	.24**	.20**	.29**	.28**	.11	−.20**	.17**	.18**	.18**	.21**	.26**	.07	−.10	–
19 Pre-math skills (Pre-k)	.07	.35**	−.33**	.38**	.29**	.27**	.34**	.44**	.10	−.15*	.30**	.32**	.21**	.30**	.40**	.10	−.10	.26**	–
20 Pre-literacy skills (Pre-k)	.12*	.39**	−.07	.26**	.23**	.33**	.28**	.30**	.14*	−.23**	.20**	.20**	.24**	.16**	.16**	.07	−.09	.23**	.52**	–
21 Classroom learning behaviors (K)	.18**	.19**	−.21**	.25**	.32**	.28**	.37**	.38**	.09	−.19**	.15*	.24**	.17*	.24**	.29**	.01	−.11	.23**	.33**	.23**	–
22 Math performance (K)	−.06	.20**	−.19**	.27**	.23**	.18**	.23**	.29**	.05	−.09	.14*	.12	.16*	.19**	.19**	.09	−.04	.29**	.40**	.39**	.31**	–
23 Reading performance (K)	.06	.32**	−.12	.25**	.29**	.32**	.32**	.33**	.10	−.14*	.19**	.18**	.19**	.18**	.22**	.06	.00	.32**	.46**	.55**	.39**	.69**

Note. S = Story; T = Tangrams; Pre-k = preschool; K = kindergarten.

^*p < .05.

^**p < .01.

During both the Tangrams and Story tasks, attention to instructions, enthusiasm/energy, persistence, and monitoring/strategy use significantly correlated with all preschool and kindergarten variables as expected. On-task behavior during the Story task was also significantly positively correlated with all outcome variables, whereas on-task behavior during the Tangrams task was positively correlated with all outcomes except math performance in kindergarten. During the Story task, positive affect was significantly positively associated with preschool literacy and negative affect was significantly negatively correlated with all outcome variables except kindergarten math performance. Indicators of positive and negative affect during the Tangrams task were not correlated with any outcome variable.

There were also significant correlations among the learning engagement indicators and the three potential covariates, gender, maternal education, and minority status. In Story, attention to instructions, on-task behavior, and enthusiasm/energy were significantly associated will all three covariates, such that higher scores were related to being female, having a mother with higher education, and being a non-minority. Higher scores on Persistence and monitoring progress/strategy use in Story were also associated with higher maternal education and non-minority status, and positive affect was associated with being female. In contrast, greater negative affect was associated with being male and lower maternal education. In Tangrams, higher scores on attention to instructions and monitoring progress/strategy use were associated with higher maternal education and non-minority status, greater on-task behavior was related to being female and higher maternal education, and greater persistence and positive affect was associated with being female. Neither enthusiasm/energy nor negative affect during Tangrams were associated with any of the covariates.

Confirmatory Factor Analysis

Tangrams

Overall fit for the one-factor model including all seven indicators was poor (χ²(14) = 126.831, p < .001; RMSEA = .170, 90% CI [.144, .198], CFI = .942). Standardized factor loadings were acceptable for all indicators except positive affect (λ = .29), indicating potential misidentification. Given this small factor loading and the low correlations between positive affect and the other observed indicators, analyses were repeated, this time omitting positive affect from the model. Fit for this model was good (χ²(9) = 15.684, ns; RMSEA = .052, 90% CI [.000, .093], CFI = .996). (See Table 3).

Table 3.

CFA Factor Loadings

	Factor Loading	Standard Error	Standardized loading
Tangrams
Persistence	1.00	.00	.94
Attention to instructions	.66	.05	.61
On-task behavior	.90	.04	.84
Monitoring progress/Strategy use	.85	.04	.80
Enthusiasm/Energy	.62	.06	.58
Negative affect	−.61	.05	−.57
Story
Persistence	1.00	.00	.97
Attention to instructions	.74	.04	.72
On-task behavior	.90	.02	.88
Monitoring progress/Strategy use	.93	.02	.91
Enthusiasm/Energy	.78	.03	.76
Negative affect	−.76	.04	−.74

Story

Identical analyses were conducted on learning engagement indicators from the Story task. As with data from Tangrams, the full one-factor model including all seven indicators fit the data poorly (χ²(14) = 129.339, p < .001; RMSEA = .172, 90% CI [.146, .200], CFI = .974), and the standardized factor loading of positive affect on learning engagement was relatively small (λ = .45). When analyses were repeated without positive affect in the model, overall fit was good (χ²(9) = 22.571, p < .01; RMSEA = .074, 90% CI [.036, .112], CFI = .997). (See Table 3).

Based on these results, positive affect was removed from the model, and the remaining six indicators, attention to instructions, on-task behavior, enthusiasm/energy, persistence, monitoring/strategy use, and negative affect were used to create the learning engagement construct in both the Tangrams and Story tasks. This construct demonstrated good internal consistency during the Tangrams (Cronbach’s α = .84) and Story (Cronbach’s α = .90) tasks.

Dimensionality Across Tasks

To test whether the two tasks represent distinct dimensions, the multidimensional invariance model was compared to a unidimensional model wherein all 12 indicators (six from each task) explained a single learning engagement factor. The multidimensional model (χ²(47) = 75.217, p < .001; RMSEA = .046, 90% CI [.025, .065], CFI = .995) fit the data better than the unidimensional model (χ²(48) = 464.303, p < .001; RMSEA = .177, 90% CI [.162, .191], CFI = .927), suggesting that the tasks should remain as two separate scales. The two factors correlated with one another by .61. Given that this study was primarily interested in the validity of each task on its own rather than their comparative influence on a set of outcomes, behavior during Tangrams and Story were analyzed separately in subsequent analyses.

Concurrent Validity

To assess the associations between behavioral learning engagement in both Tangrams and Story with concurrent pre-academic skills and mastery motivation in preschool, a set of SEM models was fit to the data. Overall fit for the Tangrams model was acceptable (χ²(41) = 76.406, p < .001, RMSEA = .056, 90% CI [.036, .075] CFI = .982), and all hypothesized effects were significant. As reported in Figure 1 and Table 4, learning engagement predicted mastery motivation, pre-math, and pre-literacy skills, such that greater engagement during Tangrams was associated with greater motivation and better pre-math and pre-literacy skills. Mothers with higher education had children who demonstrated greater mastery motivation and better pre-math skills, pre-literacy skills. Minority status was associated with lower pre-math skills and females tended to outperform males on pre-literacy. Key pathways remained significant over and above the effects of these covariates, and strongest effect sizes were found for pre-math and pre-literacy.

Figure 1. Tangrams Learning Engagement and Preschool Outcomes.

Structural equation model testing concurrent validity of the learning engagement factor during Tangrams task. Values are standardized coefficients. * p < .05. ** p < .01. *** p < .001.

Table 4.

SEM Path Coefficients

	Tangrams			Story
Path	Estimate	SE	Standardized Estimate	Estimate	SE	Standardized Estimate
Preschool Outcomes
LE➔Mastery motivation	0.07***	0.02	.23	0.08***	0.02	.28
LE➔Pre-math skills	4.36***	0.79	.34	3.75***	0.74	.30
LE➔Pre-literacy skills	2.57**	0.77	.17	4.58***	0.71	.32
Maternal education➔Mastery motivation	0.03**	0.01	.16	0.02*	0.01	.14
Maternal education ➔Pre-math skills	1.68***	0.34	.24	1.55***	0.34	.22
Maternal education ➔Pre-literacy skills	3.03***	0.43	.37	2.64***	0.43	.32
Minority status➔Mastery motivation	−0.02	0.03	−.04	0.01	0.04	.01
Minority status ➔Pre-math skills	−5.30***	1.20	−.22	−4.19**	1.26	−.17
Minority status➔Pre-literacy skills	1.24	1.48	.04	3.14*	1.50	.11
Gender➔Mastery motivation	0.05	0.03	.09	0.04	0.04	.07
Gender➔Pre-math skills	0.64	1.18	.03	0.48	1.23	.02
Gender➔Pre-literacy skills	2.91*	1.34	.10	1.95	1.38	.07
Kindergarten Outcomes
LE➔Classroom learning behaviors	0.07**	0.03	.22	0.10***	0.03	.30
LE➔Math performance	0.17*	0.07	.17	0.24**	0.07	.25
LE➔Reading performance	0.23*	0.10	.18	0.39***	0.09	.32
Maternal education➔ Classroom learning behaviors	0.02	0.01	.11	0.01	0.01	.08
Maternal education ➔Pre-math skills	0.07*	0.03	.13	0.06	0.04	.10
Maternal education ➔Pre-literacy skills	0.20***	0.05	.28	0.16***	0.04	.23
Minority status➔ Classroom learning behaviors	−0.10*	0.04	−.15	−0.06	0.04	−.10
Minority status ➔Pre-math skills	−0.25	0.13	−.13	−0.15	0.13	−.08
Minority status➔Pre-literacy skills	−0.07	0.16	−.03	0.09	0.16	.04
Gender➔ Classroom learning behaviors	0.09*	0.04	.15	0.08	0.04	.13
Gender➔Pre-math skills	−0.14	0.12	−.08	−0.19	0.12	−.10
Gender➔Pre-literacy skills	0.09	0.15	.04	0.01	0.15	.00

Note. LE = learning engagement.

^*p < .05.

^**p < .01.

^***p < .001.

To test this model using learning engagement data collected during the Story task, a second model was fit (χ²(41) = 66.314, p < .01; RMSEA = .047, 90% CI [.025, .067], CFI = .994). As with Tangrams, greater learning engagement during Story predicted greater mastery motivation and better pre-math and pre-literacy skills. Furthermore, maternal education was positively associated with pre-math skills, pre-literacy skills, and mastery motivation, and being a minority was negatively associated with pre-math but positively associated with pre-literacy skills. (See Figure 2 and Table 4).

Figure 2. Story Learning Engagement and Preschool Outcomes.

Structural equation model testing concurrent validity of the learning engagement factor during Story task. Values are standardized coefficients. * p < .05. ** p < .01. *** p < .001.

Predictive Validity

Another set of SEM models was run to analyze the associations between learning engagement in Tangrams and Story with subsequent academic performance and classroom learning behaviors in kindergarten. The Tangrams predictive model (χ²(41) = 61.451, p < .05; RMSEA = .042, 90% CI [.017, .063], CFI = .988) revealed significant pathways between learning engagement and teacher-reported math performance, reading performance, and classroom learning behaviors. The three covariates in the model also influenced kindergarten outcomes. Higher maternal education predicted better math and reading performance, and both being female and being a non-minority predicted better learning behaviors. (See Figure 3 and Table 4).

Figure 3. Tangrams Learning Engagement and Kindergarten Outcomes.

Structural equation model testing predictive validity of the learning engagement factor during Tangrams task. Values are standardized coefficients. * p < .05. ** p < .01. *** p < .001.

Similar results were found in the Story model (χ²(41) = 58.177, p < .05; RMSEA = .039, 90% CI [.009, .060], CFI = .996). Learning engagement during Story demonstrated direct effects on math and reading performance and classroom learning behaviors, such that greater engagement predicted better math and reading and more engaged classroom behaviors. Furthermore, higher maternal education was associated with better reading performance. Minority status and gender were not related to any kindergarten outcome in this model. (See Figure 4 and Table 4).

Figure 4. Story Learning Engagement and Kindergarten Outcomes.

Structural equation model testing predictive validity of the learning engagement factor during Story task. Values are standardized coefficients. * p < .05. ** p < .01. *** p < .001.

Discussion

Learning engagement is important for the successful transition to school (Kagan et al., 1995; Ladd & Dinella, 2009), and having a wide range of valid measures of engagement that may best suit various kinds of research is needed. A laboratory assessment of engagement offers an alternative to current methodologies and may allow for novel ways to examine the mechanisms that underlie both the development of early engagement and the role of engagement in promoting achievement throughout the school years. The current study has established construct, concurrent, and predictive validity for this new method of measuring behavioral learning engagement through laboratory observation during two semi-structured tasks.

Among the seven behavioral indicators selected to best summarize behavioral learning engagement in young children, attention to instructions, on-task behavior, enthusiasm/energy, persistence, monitoring/strategy use and negative affect, demonstrated strong associations with one another. CFA conducted with these six indicators revealed good to adequate fit for a one-factor model of learning engagement, measured during both the Tangrams and Story paradigms. This finding supports the hypothesis that behavioral learning engagement can effectively be observed in the laboratory and indicates that it can successfully be captured by a single construct.

These six indicators each provide important information about learning engagement and together create a complex construct that can inform researchers about how children behave while participating in learning tasks. This measure is thus an improvement over existing laboratory assessments of engagement that consider a smaller scope of behaviors. Furthermore, the design of this measure allows for more specific operationalization of learning engagement. First, the learning and problem-solving demands of the tasks used in the current study help establish that this measure is assessing engagement with learning, not task engagement more broadly. Second, observing engagement in the laboratory setting helps isolate children’s engagement with learning tasks from engagement within their classroom, which may be influenced by preexisting social experiences.

In contrast to the original hypotheses, positive affect was not strongly associated with many of the other observable engagement behaviors, and the CFA models that included positive affect yielded poor fit for both the Tangrams and Story paradigms. One reason for this may have been that the current tasks were largely non-social and therefore may not have encouraged the expressions of positive emotion. It is possible that some children experienced task enjoyment, pride, and interest internally, but did not manifest these emotions behaviorally. Additionally, the expression of positive affect may have a more nuanced relation with behavioral engagement than originally hypothesized: Although positive affect is an indicator of task enjoyment, certain positive emotions such as overexcitement may actually disrupt focused involvement (Pekrun & Linnenbrink-Garcia, 2012). Accordingly, greater positive affect has been associated with poorer effortful control (Kochanska, Murray, & Harlan, 2000) and greater behavior problems in children (Eisenberg et al., 1996). Thus, in order for a child to be fully behaviorally engaged in the moment, positive affect may need to be down-regulated or delayed. Additional research is needed to further understand the association between positive affect and observed learning engagement among preschoolers.

It is notable that the pattern of associations among these engagement indicators was similar across the Tangrams and Story tasks. This suggests that behavioral learning engagement operates similarly across verbal and nonverbal domains during this developmental period and that behavioral learning engagement may best be conceptualized as a general approach to learning rather than something more content specific. However, a test of multidimensionality did suggest that the two tasks represent different dimensions that are best analyzed separately rather than combined into a single scale. As such, considering the task in which learning behaviors were measured may be important. Whether this multi-dimensionality is due to differential academic domain or other measurement differences between the tasks warrants further investigation. In either case, the generalization of learning engagement across tasks suggests that this construct is not solely dependent upon the specifications of a single task.

The construct of behavioral learning engagement during both the Tangrams and Story tasks demonstrated good concurrent and predictive validity, even after controlling for parent and child demographics including child gender, minority status, and maternal education. As hypothesized, behavioral learning engagement during preschool was positively associated with children’s concurrent pre-literacy and pre-math skills as well as parents’ report of children’s mastery motivation. These associations between the current measure of learning engagement and both academic skill and motivation are consistent with prior literature and theory (Appleton et al., 2008; Finn & Zimmer, 2012; Newmann et al., 1992; Reeve, 2012).

Behavioral learning engagement in preschool was also predictive of children’s kindergarten math and reading performance and classroom learning behaviors, as reported by the child’s teacher. Not only do these findings demonstrate the predictive validity of the current measure and the potential long-term effects of behavioral learning engagement, it also establishes the relation between learning engagement measured in a laboratory setting with learning behaviors and academic performance in the classroom. Although learning may occur in many different contexts in early childhood, the classroom is important, as it is an environment specifically dedicated to formal learning activities. Thus, the cross-context associations between the current measure of learning engagement and both achievement and classroom learning engagement provide evidence that our measure is assessing qualities that generalize beyond the laboratory context.

Given these results, the current measure is a promising new way of assessing behavioral learning engagement. As an alternative to classroom measures of engagement and simpler laboratory assessments, this measure may provide opportunities for researchers to examine the construct in novel ways and in relation to a broader range of other factors.

Limitations & Future Directions

Some limitations of this study must be acknowledged. First, although our sample was racially and economically diverse and matched the demographics of the city in which assessment took place, follow-up studies testing this measure among lower-income children specifically may be particularly useful, as the transition to school may be uniquely challenging for low-income children (Janus & Duku, 2007). Second, although structured laboratory measures allow more control and standardization than classroom observations, they are less naturalistic. The laboratory environment differs in many ways from a classroom. For example, classroom learning is often done within group settings among other peers and with greater distractions. Although children worked while seated next to an experimenter, the current tasks were primarily nonsocial and may have influenced children’s behaviors. For example, more social tasks may have elicited more expressive positive affect from children who appeared neutrally content to independently work during the current tasks. Additionally, in a laboratory, children may feel uncomfortable in an unfamiliar setting with a novel adult experimenter. Laboratory measures are thus less ecologically valid than both teacher report or classroom observations. However, this study demonstrates that engagement in the laboratory is longitudinally associated with children’s behaviors in the classroom, as reported by teachers. This suggests that the current measure may have some degree of ecological validity.

It is also important to consider aspects of learning engagement that may not be easily observed, such as engagement at the affective and cognitive levels. As the current measure was based on observation, it only assesses behavioral manifestations of learning engagement. However, we may also consider affective and cognitive engagement through children’s self-report on their experiences with a learning task, thoughts about school, and preferences for challenge. Moreover, certain aspects of internal engagement, such as effort, stress, and pleasure, may be assessed through physiological or neural measures. Although the current measure incorporates behavioral manifestations of affective (enthusiasm, positive/negative affect) and cognitive (monitoring/strategy use) engagement, assessing these processes specifically may improve our understanding of children’s learning engagement as a whole.

Further research is also needed to demonstrate the longitudinal stability and relevance of this learning engagement factor. Studies should investigate whether behavioral learning engagement in preschool maintains a similar structure in kindergarten, first grade, and beyond, and whether it continues to correlate with classroom measures of achievement and engagement. Assuming that this construct does hold together in future years of assessment, this measure may also help researchers elucidate the mechanisms that foster strong levels engagement in young children. Understanding the development of learning engagement may in turn help inform prevention and intervention programming targeted at high risk children who may demonstrate low engagement with learning.

Conclusion

The current study suggests that learning engagement can successfully be measured in a laboratory environment and supports the important role early engagement with learning has on subsequent achievement and learning related behaviors. The new measure established by this study provides a novel method of measuring learning engagement and offers an alternative to teacher-report questionnaires or classroom observation. Thus, this measure may be particularly useful for researchers who want to concentrate resources on laboratory assessments and may be sampling children who attend educationally diverse preschool programs. The current study provides evidence for the construct validity of this measure and demonstrates its concurrent and predictive validity. Based on these results, these behavioral learning engagement laboratory tasks and coding scheme appear to be a strong assessment tool that will give researchers new ways to investigate the predictors and consequences of learning engagement in early childhood. Given the importance of learning engagement for both early and later school success, this measure may facilitate translational research that will help broaden our understanding of school readiness and achievement and inform educators, program officers, and policy makers on how best to prepare young children to be active participants in their own learning.

Highlights.

• A novel laboratory assessment of behavioral learning engagement was investigated.

• Confirmatory-factor analysis supported an engagement factor with six indicators.

• Learning engagement predicted concurrent mastery motivation and pre-academic skills.

• Engagement predicted kindergarten classroom behaviors and school performance.

• Associations persisted even when child demographics were controlled.

Appendix: Learning Engagement Coding Scheme

1. Attention to Instructions

This code concerns the child’s attention to the examiner and materials while the task is initially being described.

1. Child does not attend to examiner during instructions but fidgets and/or looks away throughout most of the time. When asked a question, child does not respond. Child appears to pay little or no attention to examiner’s directions.

2. During initial instructions, child appears bored or makes efforts to handle the materials in his/her own way without regard for the task being described by the examiner. Child may not respond to questions or may respond but without much interest.

3. During initial instructions, child is sometimes involved and appears to be listening but is sometimes looking away or handling materials in his/her own way. Child does not ask questions or indicate verbally or nonverbally (nodding) that he or she is following the examiner’s directions, but wants to handle the materials (focus is on the materials rather than the examiner). If examiner asks questions, child responds to at least some of them.

4. Child listens attentively to most of the examiner’s instructions and watches the examiner’s demonstration, but may lose interest somewhat toward the end if instructions go on very long. Child answers questions or nods to indicate understanding but does not initiate conversation (does not ask questions or make independent comments about the task or materials).

5. Child pays close attention and participates actively while instructions are given (asks questions, makes comments about the task or materials, engages with examiner). Child shows he or she understands by nodding, answering questions about the task, and/or manipulating materials appropriately to demonstrate understanding. In story task, child may solve the task while the instructions are being given.

2. On-task Behavior

This code concerns the extent to which the child maintains focus on and active, productive engagement with the task throughout the session. The code is largely based on the amount of task time that the child is focused and involved. Focus involves visual attention to materials and to the examiner if he/she is talking or the child is requesting help. Engagement with the task involves manipulation of materials in a way that would result in completing the task (putting the shapes into the template as opposed to standing them up on their sides; examining the picture cards as opposed to turning them over or putting them into a design that is not related to the story). The child does not have to be successful to be on-task; he or she must simply be working with some level of concentration and effort, even if his/her solutions are incorrect.

1. Child does not focus on or engage with the task. Child may handle materials but does his/her own thing with them rather than working to solve the task presented.

2. Child’s focus on and engagement with the task is minimal and attention may be fleeting. There is little sustained involvement. Child may alternate focus on the task with doing his/her own thing with the materials.

3. Child’s focus on and engagement with the task is evident for about half the session, or the child is repeatedly distracted and has to be redirected on several occasions.

4. Child focuses on and engages with the task for most of the time. There may be brief distractions or short periods of the time when the child’s attention appears to wander, but he or she returns to the task quickly.

5. Child is focused and engaged for the entire time or essentially all of the time.

3. Enthusiasm/Energy

This code describes the quality of the child’s involvement with the task. A child who is enthusiastic and energetic appears eager to try each problem as presented, leans forward, makes positive comments or verbally expresses interest. A child may solve the task or portions of the task successfully but show little energy or enthusiasm – just seems to be going through the motions.

1. Child does not engage with the task and seems largely bored and uninterested. Child is passive.

2. Child’s energy is low; he/she manipulates the materials but with little interest. Movements are slow; there may be long pauses between actions while the child looks at the table or focus is elsewhere.

3. Child participates in tasks but does not appear enthusiastic; child appears to be following instructions or being compliant but without a lot of energy. Or, child may initially seem interested and energetic but then interest wanes during the last half of the session.

4. Child shows an active interest in task when materials are first presented; reaches for materials or may express interest verbally (“Can I try that?”; “I can do that”). Child actively participates in solving the tasks. Child maintains involvement and enthusiasm, and may only show some lowering of energy if task becomes very difficult.

5. Child’s interest in task appears strong and he/she may make positive comments (“This is fun.” “I like doing this.”). Child begins efforts to solve each problem immediately and appears to be eager and ready for the next challenge once he/she has solved a problem. Child is an active and eager participant, even when tasks are difficult. (The difference between 4 and 5 is qualitative – a child scoring 5 is consistently highly enthusiastic.)

4. Persistence

This code comes into play when the child has some difficulty with a task – it can be an easy part of the task but one the child has some trouble with (placing the two triangles into the square box so the lines don’t show, for example), or a more difficult component of the task (dividing two sets of picture cards into two stories). Persistence is coded at whatever level or stage of the task that is needed by the child. If child never has difficulty (completes all segments of the task correctly), score 5.

1. Child never makes an effort to solve the task or gives up as soon as a task becomes at all difficult. May switch to manipulating materials in own way, trying to talk with the examiner about something other than the task, whining (“I can’t do it”), asking to do something else, or simply sitting back and doing nothing.

2. Child is satisfied with an incorrect solution and appears to have no interest in correcting errors or trying again. May shrug shoulders if told solution is incorrect or just take the whole thing apart (remove all the Tangrams from the template, push the story cards away) in haste.

3. Child sometimes shows efforts to fix incorrect solution or solve a tough problem, but gives up and appears to make no effort on at least one task until examiner redirects. May try to get examiner to solve the problem for him or her rather than using suggestions or trying on his/her own.

4. Child makes repeated attempts to solve a difficult task or consistently remains focused when task is hard. May briefly appear to be discouraged on a task but reengages quickly without examiner intervention. If child asks for help on a difficult problem, he or she may disengage slightly, waiting for the examiner to solve the problem.

5. Child maintains focus throughout tasks, even when they are difficult. May seem to increase concentration when task is difficult, or simply takes the difficulty in stride. Works carefully and is clearly trying to solve the problem. Child may ask for help but continues working at the same time (doesn’t rely on examiner to solve the problem).

5. Monitoring Progress/Strategy Use

This code involves the extent to which the child is aware of his/her own progress toward solving the task, can recognize when there is a problem that is preventing a correct solution, is able to use problem-solving strategies, and can use examiner’s suggestions effectively.

1. Child makes minimal or no effort to solve the task. Comments or suggestions by the examiner make the child confused or cause him/her to withdraw.

2. Child moves materials about but seems either not to understand what a solution would involve or is not interested enough to work on a solution. May not appear to be concerned about whether a solution is correct or not. (Moves picture cards around without really examining them; places Tangram blocks on template carelessly.) Or child may appear to be trying but seems to be unaware of how to solve the problem (on Tangrams, places shapes on the page but doesn’t appear to recognize they do not fit in the lines; arranges picture cards in some configuration that does not fit the template). Comments or suggestions by the examiner are not attended to or used by the child to solve the task. Child may not seek help or may say “No” when asked if he/she needs help.

3. Child appears to be trying to solve the task, but if his/her initial solution does not “work,” he or she takes the whole thing apart and then appears to start over completely rather than identifying the specific problem and solving it. May keep doing the same thing over several times (continuing with the same error, as not placing the parallelogram correctly; putting the picture cards back in the same order), even when this does not solve the task. If asks for help, is not able to identify the specific problem where help is needed. When the examiner makes comments or suggestions the child may appear upset and may seem reluctant to keep trying (leans back in chair away from task; simply looks at examiner or materials without trying to fix the problem).

4. Child recognizes when a problem exists and focuses his/her attention on the portion of the task that is the problem (flipping or rotating a shape that does not fit; picking up a picture card to look at it more closely; in Tangrams, checking to see which shapes do not fit inside the lines). Child appears to be strategic in solving problems (does parallelogram first or last; spreads story cards out on table rather than stacking them). If child runs into a problem and takes task apart to start over, he/she solves it quickly on the second try, making it appear that the child has identified the problem and was able to solve it. If asks for help, can identify where help is needed (by pointing to the troublesome shape or card, or asking more specifically what he/she needs help with). Responds to examiner’s comments or suggestions by trying to fix the problem (in story task, if told cards are not in the right order, reorders them even if the new solution is not correct; in Tangrams, can point to the shape that is not correct). May occasionally “check-in” with examiner, by looking toward the examiner, as if to see if his/her own assessment of progress is reflected in the examiner’s response.

5. Child is clearly aware of errors (stops activity, shows change of facial expression, looks closely at the portion of the task that is creating a problem, may look questioningly at the examiner) and makes a focused attempt to correct the error. As in (4), child can identify where help is needed. Throughout the task, child tries alternative solutions rather than repeating errors. If child finds a solution that works for one part of the task (flipping the parallelogram, identifying the end card or beginning card first), he/she applies that same solution in a later part of the session. If a child works effectively throughout to solve problems without ever getting to a point where there is an obvious error, score 5.

6. Positive Affect

The positive emotions to be coded are pleasure or enjoyment and interest. Pleasure or enjoyment is shown by a relaxed face and upturned mouth or smile. Interest is detected by widened eyes and focused attention. Sometimes a slight frown also indicates interest and positive concentration, but if the frown deepens and the eyes narrow, this indicates a transition into negative affect. A low score on positive affect does not require expressions of negative affect, but simply the lack of emotional indicators of enjoyment and interest. This code incorporates both intensity and frequency of positive affect.

1. No clear emotional indicators of enjoyment or interest are shown during the session.

2. Child shows interest or enjoyment briefly or a few times but they are not intense and are not evident throughout the session. Child may smile briefly at completion of one segment of the task, for example, but then lapse back into a neutral expression.

3. Child shows interest and/or enjoyment during about half the session but is neutral for about half the time. May share positive feeling of success with examiner once or twice by looking at the examiner with a positive expression or saying “I’m done” or “I did it” in a bright and positive way.

4. Child is largely positive; shows interest and/or enjoyment during most or all of the session. Child smiles often. Shares success or positive affect with examiner more than once or twice. Affect is largely moderate rather than intense.

5. Child is highly positive throughout the entire session and shares positive feelings with examiner (“I didn’t need help,” “I’m good at this.”). Child smiles broadly and clearly enjoys success.

7. Negative Affect

Negative affect may include frustration, anger, annoyance, sadness, and boredom. Negative affect is shown through facial expressions involving frowning, narrowing of the eyes, a dejected look, or by negative arousal and agitation. A low score on negative affect does not require expressions of positive affect, but simply the lack of emotional indicators of anger, frustration, or sadness. This code incorporates both intensity and frequency of negative affect.

1. No negative affect is displayed.

2. Child may have one or two episodes of negative affect about the task but they are brief and low in intensity.

3. Child shows several separate episodes of low-intensity negative affect, or mild negativity persists throughout a portion of the session. Negative affect is about the task, not about the situation, the physio equipment, or the examiner’s comments. Child may express a desire to stop the task (“I don’t like this.” “When can we play a game?”)

4. Child appears annoyed, angry, or upset throughout much of the session. Or may appear very bored or sad. Child frowns deeply, whines, and/or gets frustrated or upset easily and quickly, even early in the session when the task is not difficult. Child’s requests to stop the task are repeated more than once.

5. Child’s negative affect is intense for at least part of the time and continues throughout the session. Child whines throughout or continuously asks to stop or do something else.