Emotional competence is a key component of young children’s school readiness (Blair, 2002; Denham, 2006). School readiness includes the ability to sustain effort, or persist, even in the face of the frustration that arises when trying to learn new, challenging information is challenging (Denham, Zahn-Waxler, Cummings, & Iannotti, 1991; Martin, Ryan, & Brooks-Gunn, 2013). Persistence contributes to children’s later academic achievement, above and beyond their vocabulary skills and demographic characteristics (Mokrova, O’Brien, Calkins, Leerkes, & Marcovitch, 2013). As children approach formal school entry, it is believed that they also gain the ability to harness their emotions in ways that help them tackle challenges, e.g., to harness their frustration and persist in trying to tackle a problem. However, evidence in support of this link is minimal. Research usually focuses on the problematic aspects of young children’s emotions, rather than on their functional benefits. How emotions motivate behaviors such as persistence is important, however, because reduced engagement and effort can have important, long-lasting educational implications (Fredricks, Blumenfeld, & Paris, 2004; Mokrova et al., 2013). Thus, understanding the adaptive function of emotion to motivate and organize action, such as persistence during a challenge, is an important empirical question (Skinner, Pitzer, & Brule, 2014).
A substantial body of research has highlighted the problematic aspects of children’s negative emotion. For example, problems with anger expression are associated with conduct problems that can result in expulsion as early as preschool (Gilliam & Shahar, 2006). However, being able to experience, express, and utilize the full range of human emotions is a central component of models of emotional competence (Denham, 1998; Halberstadt, Denham, & Dunsmore, 2001; Saarni, 1998). These models adopt the functionalist perspective on emotional development (Barrett & Campos, 1987), which emphasizes the adaptive nature of emotions. In this framework, anger reflects appraisal that a goal is blocked and readiness to act with increased effort to overcome obstacles to a goal. Although anger can be problematic when children’s goals conflict with rules or adult demands or directions, this functional view of anger contends that it reflects readiness to engage in increased and sustained effort to overcome a goal. Thus, modest anger should help children achieve goals that are acceptable, such as persisting at a difficult task. In contrast to anger, sadness reflects appraisal that a goal is lost and cannot be recovered and readiness to relinquish effort, i.e. give up. In the context of needing to tackle a hard problem, sadness can interfere with persistence, whereas anger can be harnessed to achieve goals that adults want children to tackle (Cole et al., 2011). Despite a general acceptance of the functionalist perspective in research on children’s emotional development, tests of the adaptive functions of emotions are few (Keltner & Gross, 1999; Lench, Tibbett, & Bench, 2016).
A small corpus of evidence suggests that anger is associated with appropriate effort toward overcoming a blocked goal in infants, preschoolers, and young adults (Dennis, Cole, Wiggins, Cohen & Zalewski, 2009; He, Xu, & Degnan, 2012; Lench & Levine, 2008; Tan & Smith, 2018). When preschool age children’s ability to retrieve a toy was blocked, those who displayed anger more frequently and quickly spent more time persisting (He et al., 2012). Anger was also associated with a wider range of appropriate efforts by preschool age children to retrieve a toy in a locked box (Dennis et al., 2009). In contrast to anger, infants’ sadness was unrelated to persisting at regaining a goal (Lewis, Alessandri, & Sullivan, 1990; Lewis, Sullivan, & Kim, 2015) and preschool age children’s sadness predicted fewer appropriate efforts at retrieving a toy in a locked box (Dennis et al., 2009). Furthermore, another study showed that 4- and 5-year-olds persisted longer when they showed more anger than sadness (Tan & Smith, 2018). Based on the functional perspective on emotional development and the available evidence, we predict that anger should be associated with longer persistence, whereas sadness should not be.
The functionalist perspective also underscores that emotional expressions reflect an ongoing and continuous underlying psychological process indicating how the individual relates to the environment (Barrett & Campos, 1987). As a situation unfolds, appraisal and action readiness change. Therefore, this perspective urges the study of emotions as temporal processes (Cole, Martin, & Dennis, 2004; Cole & Hollenstein, 2018; Keltner & Gross, 1999). However, most studies of children’s emotion measure the presence or absence of an emotion expression or other behavior within relatively brief time intervals and aggregate across time to create summary variables to use in subsequent analyses. This approach obscures information that may be important for understanding children’s behavior. For example, consider two children with the same summary score for on-task behavior. One child is off-task occasionally but consistently transitions back to on-task behavior. The second child is on-task initially but abandons any effort later in the task. The first child’s behavior is likely to be viewed as reflecting persistence at the task more than the latter child’s behavior; however, this distinction is lost if only using a summary score approach. Evidence suggests that children’s anger can contribute to their task persistence (e.g., Dennis et al., 2009), but the process by which this unfolds and, specifically, direction of the relation between children’s emotion and the duration of their on-task behavior has not been tested directly. Children may express frustration as a reaction to unsuccessful persistence or, as the functionalist perspective argues, anger may precede and motivate sustained effort.
In the current study, we use two contemporary analytic approaches to study the relation between children’s anger and sadness and their on-task behavior in order to test the hypotheses that young children’s anger is associated with a lower likelihood of abandoning effort, and that sadness is associated with a higher likelihood of abandoning effort. Additionally, we investigate children’s transitions between states defined by anger and sadness and on-task and off-task behavior. The functionalist perspective, which asserts both adaptive aspects of an emotion like anger and its dynamics as a process, supports the prediction that the ability to transition between anger and calm on-task behavior should improve with age.
Emotional Development in the Preschool Years
There are adaptive changes in the frequency and intensity of emotional expressions in early childhood (Izard, 1977). For example, anger expressions increase as young children become more capable of acting upon their environments and overcoming obstacles (Campos, Campos, & Barrett, 1989). After the onset of crawling, infants display more anger for several weeks (Zachry et al., 2015), which appears linked to increased willfulness (Biringen, Campos, Emde, & Appelbaum, 2008) and greater reactivity to their action being restricted (Roben et al., 2012). Anger further increases during the second year; as children’s self-awareness and language skills develop, they assert themselves even when they recognize their goals are at odds with those of caregivers and can say “No” (Goodenough, 1931). Such increases in anger are also observed in the preschool years and may be related to sense of self and agency in one’s environment, though less is known about changes in sadness (Abe & Izard, 1999). Further, the preschool years are thought to mark the acquisition of the ability to modulate emotions (Kopp, 1989), likely offering children the opportunity to harness emotions in the services of adaptive functions, such as overcoming a blocked goal, rather than be derailed by intense emotions. Thus, it may be that by 48 months, children have more angry appraisals of situations given that they are older and have a sense that they should be able to do more. Additionally, by this age, children may be able to harness anger in order to capitalize on its adaptive functions. Moreover, there is evidence to suggest that temporal dynamics of emotion may change with age in preschool-aged children. For example, one longitudinal study reports that the timing of anger and other behaviors in reaction to a blocked goal changes between 18 and 48 months (Cole et al., 2011). However, that study did not examine temporal associations between anger and behavior.
The Present Study
In the present study, we observe children’s anger and sadness, and their on- and off-task behavior, when a small toy is in a locked box that the child cannot open because they were given the wrong key (Goldsmith, Reilly, Lemery, Longley, & Prescott, 1995). In this task, young children generally display mild to moderate anger and sadness. Thus, we examine whether these expressions are related to how long children try to open the box before they abandon effort. We use survival analyses to examine age-related change in how anger is related to the maintenance of on-task behavior and also use hidden Markov models to examine age-related changes in transitions between angry on-task and calm on-task behavior.
Survival analysis is used to examine event times, e.g., how long until the event of interest occurs (Lougheed, Benson, Cole, & Ram, 2019). In this study, we use survival analysis to test the hypothesis that angry expressions are associated with longer time until children completely stop on-task behavior, and the hypothesis that sad expressions are associated with shorter time until children completely stop on-task behavior. Moreover, we hypothesize that the association between anger and longer persistence of on-task behavior increases between 36 and 48 months of age, indicating age-related improvement in the adaptive function of anger.
Whereas survival models provide for examination of the time until a specified event occurs, hidden Markov models (HMM) provide more fluid and descriptive examination of transitions between two or more hidden/unobserved states. Here, we used HMM to test the hypothesis that the probability of transitioning between angry on-task and calm on-task states is greater at age 48 months than at age 36 months.
Method
Participants
Data for these analyses come from a larger longitudinal study of emotion regulation (Development of Toddlers Study; Cole, Crnic, Nelson, & Blair, 2000). In that study, 124 economically-strained families living in rural and semi-rural communities in a Mid-Atlantic region of the United States of America participated in a series of lab and home visits starting at child age 18 months. By age 48 months, 120 families remained enrolled and did not differ from withdrawn participants on demographic characteristics. Children were identified by their mothers as white (93.3%) or biracial (6.7%) and were mostly from two-parent households (97.6%). Most mothers (67.3%), and about half of fathers (44.8%), had at least some college education. Families’ parent-reported household income levels, assessed when children were 42 months old, averaged $49,016.47 (SD = $22,377.01). Eligibility for study participation required income greater than the poverty level but below the national median income for their family size.
At the visit when children were 36 months, three families did not attend the lab visit and three other children did not complete the locked box task (e.g., due to difficulty separating from parent), resulting in N = 114 participants. At 48 months, three families did not attend the lab visit and one child did not complete the locked box task, resulting in N = 116 participants. Children attended the lab visit within two weeks before or after their birthday.
Procedure
The Transparent Box procedure (Goldsmith et al., 1995) was administered at both the 36 and 48 month lab visits. The task elicits anger and sadness in young children (Buss & Kiel, 2004; Day & Smith, 2013; Dennis, Hong, & Solomon, 2010; Jahromi & Stifter, 2008). A trained research assistant (RA) invited the child to choose a small toy to take home from a selection of three small toys: Diecast cars, Care Bears, or Block Buddies. The RA put the preferred toy into a clear Plexiglass box with a padlock, locked it, and taught the child to open the box with a key. Once the child unlocked the box independently, the RA then handed the child a set of keys; unbeknownst to the child, this set of identical keys would not unlock the box. The RA said while leaving, “I’ll be back in a little bit. I will let you work on that for a while. When you open the box, you can play with the toy inside.” The child was alone for 2.5 minutes (150s) at 36 months and 3 minutes (180s) at 48 months. At the end of the time, the RA returned, acknowledged the wrong keys were given accidentally, gave the child the correct keys and the child opened the box and retrieved the toy. To standardize the analytic frame across ages, we analyzed behaviors and emotions observed during the first 2.5 minutes (150s) of the task.
Measures
Two independent trained teams coded nonverbal emotion expressions or behaviors in 1s intervals from video records. Coders in each team first achieved 90% accuracy with a master coder. Inter-rater reliabilities for emotion and for task behavior was estimated from 15% of cases that were double coded at each age point.
On-Task Behavior.
Persistence was defined by appropriate on-task behavior, i.e., trying to open the box without engaging in disruptive behavior (e.g., trying to break open the box). Children’s attempts to open the box with the keys or to make appropriate alternative attempts (e.g., seeing if the box opened at the hinges) were coded as on-task behavior. Inter-rater reliability for the behavioral coding system was good, Cohen’s κ = .88 and percent agreement = 94%.
The 150s on-task behavior stream was used in two different ways. For the survival analysis, we defined an event time variable as the number of seconds until the child “gave up,” i.e., the time until a child was off-task and never resumed on-task behavior. Proportion of time on-task and time until giving up were correlated at r = .84 at 36 months and r = .79 at 48 months. For the HMM, the second-by-second binary codes (indicating presence or absence) of on-task behavior were analyzed as a 150s time-series.
Emotion Expressions.
Nonverbal facial, vocal, gestural, and postural cues defined expressions of anger and sadness (Cole et al., 1994). Cues that defined anger included harsh or hostile vocal quality, furrowed brows, eye narrowing, jaw clenching, or lip pressing in the face, and hands formed into fists in posture. Cues that defined sadness included whining or whimpering vocal quality, pulled down or pouted lips, drooped eyes or eyebrows in the face, and slumped body posture. Coders rated emotion expressions as absent (0) or present (1). If present, coders created an additional rating of emotional intensity, from low (1) to high (3). Because intensity scores were predominantly low intensity (>90% at both ages), only the absent and present codes were used. If children left the camera frame or turned away from the camera, and did not vocalize, the expression variable was treated as missing. Inter-rater reliability for the emotion coding system was good as indicated by Cohen’s (1968) weighted kappa (in order to allow for partial agreement, e.g., one coder indicating that anger and sadness occurred, but the other indicating only anger occurred would receive a .75 weight; κ = .85). Percent agreement was 89% and 80% for anger and sadness, respectively.
For the survival analysis, we calculated anger and sadness scores for each child as the proportion of the 150s of the task that the child displayed angry or sad expressions, respectively. Specifically, if a child gave up 130s into the task, the anger and sadness scores indicated the proportion of each observed during those 130 seconds. Thus, the anger and sadness variables were limited proportions of emotions expressed prior to the permanent cessation of on-task behavior. For the HMM, the second-by-second emotion codes were compiled into a 150s binary time-series indicating absence or presence of anger and sadness in each second.
Age.
Child age was treated as a binary variable indicating whether target variables were observed at 36 months (= 0) or 48 months (= 1).
Data Analysis
Two types of stochastic models, survival analysis and HMM, were used to examine age-related differences in (a) the influence of anger and sadness on the risk of transitioning from on-task behavior to stopping on-task behavior; and (b) how children transitioned among states defined by on/off task behavior and calm/negative emotions.
Survival Analysis.
To test hypotheses that more anger was associated with later abandonment of effort, and that more sadness was associated with earlier abandonment of effort, particularly as children age, we used a form of survival analysis, the Cox proportional hazards model (Cox, 1972) with a frailty term for random effects (Lougheed et al., 2019; Therneau, Grambsch, & Pankratz, 2003). Specifically, the model was constructed as,
where the log hazard of child i’s giving up at time t, logh(time in taskit), is modeled as a function of the baseline hazard, αt, that applies for all children, a random effect (frailty) term, vi, that accommodated individual differences, and a set of parameters, β1 to β6, that indicated how the hazard differs systematically (and proportionally) with age, anger, and sadness. The interaction terms of interest indicated how the influence of anger on the log hazard was moderated by age. Because of prior literature, anger × sadness interaction was included (Tan & Smith, 2018).
The survival model was estimated using the coxme package in R, which allowed for the addition of a random intercept to account for dependencies (Therneau, 2015; R Core Team, 2015). Per guidelines for Cox regression, left-censored cases, where children (n = 14) never engaged in on-task behavior, were not included in the analysis (Lougheed et al., 2019; Singer & Willett, 2003). Children (n = 89) who never stopped on-task behavior completely by the end of the task were considered to be right-censored and included in the analysis. Proportion of time on-task for the 89 censored children ranged from .21 to 1.00, with 88% of these children spending more than half of the time on-task. After main analyses, diagnostic tests were used to evaluate viability of the proportional hazard assumption (i.e., that the hazard of children stopping on-task behavior was proportional across different levels of predictors).
Hidden Markov Models.
To investigate how children transitioned between different latent emotion/task engagement states, we used HMM (Rabiner, 1989). Specifically, the model was constructed to test age-related differences in in how children transitioned among a set of “hidden” or unobserved states. Assuming that the current state (St) is only dependent on the prior state (St - 1), the dynamics of the system are described as a Markov process, where P(St|S1, …, St-1) = P(St|St-1) is captured analytically by a matrix of transition probabilities and a matrix of initial state probabilities.
In the present study, the 150-occasion time-series data for anger, sadness, and on-task behavior were described using an HMM. Formally (following Visser & Speekenbrink, 2010), the joint likelihood of observations O1:T and latent states S1:T, given model parameters Ɵ and covariates z1:T, is modeled as
where St is an element within a set of i = 1 to n latent sates, πi(z1) gives the probability of state i at time t = 1 with covariate z1, aij(zt) gives the probability of transitioning from latent state Si in one second to latent state Sj in the next second when the covariate is zt, and is a vector of observation densities specifying how k = 1 to m observed variables, , are linked to latent state j = 1 to n. With the present data, the response distributions for each of the binary observed variables (anger, sadness, and on-task behavior), , was specified as a multinomial distribution function with an identity link function.
Maximum likelihood estimates of model parameters are obtained using the forward part of the forward-backward algorithm, as implemented in the depmixS4 R package (Baum & Petrie, 1966; Rabiner, 1989; Visser, 2011; Visser & Speekenbrink, 2010). To determine the number of “hidden” (latent) states needed to describe changes in anger, sadness, and on-task behavior across the task we compared the fit of models with between 2 and 10 latent states (150 sets of random starting values for each model). After selecting the number of latent states, the model was re-estimated to obtain age-specific parameter estimates, and a standard Viterbi (1967) algorithm was used to derive the implied time series of latent states for each child at each age (see Visser, 2011 for estimation details). We then examined these latent time series using plots. Additionally, we calculated the number of each type of transition for each child at each age, and used a dependent t-test to examine whether the average number of transitions of each type differed with age (see Hollenstein & Lewis, 2006).
Results
Survival Analysis.
Cox regression models with a random intercept were used to examine how anger and sadness were related to the time until children completely stopped on-task behavior. Results of the full model are presented in left side of Table 2. The age × anger interaction term was significant, β4 = −0.05, exp(β4) = .95, p = .02, but the other interaction terms (age × sadness; anger × sadness) were not. Results from a trimmed model are presented in the right side of Table 2. As seen in Figure 1, at age 36 months (left panel), greater anger was associated with greater likelihood of giving up earlier in the task, β2 = 0.03, exp(β2) = 1.03, p < .05, while at age 48 months (right panel), greater anger was marginally associated with decreased likelihood of giving up earlier in the task, β2 + β4 = −0.03, exp(β2 + β4) = .97, p = .05.
Table 2.
Results from Cox Regression Mixed Model Examining Relations Emotion Expressions and Time until Giving up, Moderated by Child Age
| Full Model | Trimmed Model | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fixed effects: | Estimate | SE | HR | p | 95% CI (Lower, Upper) |
Estimate | SE | HR | p | 95% CI (Lower, Upper) |
||
| Age (0 = 36 months) | .328 | .251 | 1.389 | .190 | −.164, | .821 | .326 | .241 | 1.386 | .180 | −.146, | .798 |
| Anger | .019 | .019 | 1.019 | .320 | −.018, | .056 | .026 | .012 | 1.026 | .035 | .002, | .050 |
| Sadness | −.001 | .010 | .999 | .960 | −.021, | .020 | .002 | .007 | 1.002 | .790 | −.012, | .016 |
| Age × Anger | −.053 | .023 | .948 | .022 | −.099, | −.008 | −.058 | .021 | .944 | .005 | −.099, | −.017 |
| Age × Sadness | −.004 | .015 | .996 | .800 | −.032, | .025 | ||||||
| Anger × Sadness | .001 | .001 | 1.001 | .620 | −.002, | .003 | ||||||
| Random effect: | SD | SD | ||||||||||
| Intercept | .510 | .491 | ||||||||||
| Model fit: | ||||||||||||
| Penalized Log Likelihood | χ2(30) = 65.05, p <.001 | χ2(27) = 61.58, p <.001 | ||||||||||
Note. CI = Confidence Interval; SE = standard error; HR = hazard ratio; SD = standard deviation; N = 216 after removal of left-censored cases
Figure 1.
Survival Function of Time until Giving Up at Low and High Anger at Ages 36 and 48 months.
Note. Proportion of anger plotted at one standard deviation below and above median; n = 114 (left panel), n = 116 (right panel).
Diagnostic tests using Schoenfeld residuals confirmed the viability of the proportionality assumption (i.e., hazard of children stopping on-task behavior was proportional across different levels of age, anger, and sadness) for all variables except the age × anger interaction, (χ2 = 5.11, p = .02). This suggested that the age difference in how anger was related to persistence was larger later in the task. To account for this possibility, we followed up with a model that included the three-way age × anger × time interaction. Results were similar, suggesting that any violation of proportionality was not substantial, and that results from the more parsimonious model were robust.
Hidden Markov Model.
HMMs (Rabiner, 1989) were used to investigate how children transitioned between different latent states defined by joint probabilities of anger, sadness, and on-task behavior. We first determined how many “hidden” (latent) states were needed to describe the time series of anger, sadness, and on-task behavior. Fits and stability of models with between 2 and 10 latent states (150 sets of random starting values for each model) suggested that a 4-state model provided the most parsimonious and substantively meaningful representation of the data (minAIC2state = 42297, minAIC3state = 34379, minAIC4state = 30033; models with more than four states were increasingly unstable, often with less than 5 of 150 models converging).
As shown in Table 3, patterns of anger, sadness, and on-task behavior observed in each second of the task were described parsimoniously by four latent states: A Calm/On-task state (State 1) was characterized by no anger (1.000) and very high probability of on-task behavior (0.999); an Anger/On-task state (State 2) characterized by a combination of anger (0.969) and on-task behavior (0.990); a Sadness/Limited On-task state (State 3) characterized by high probability of sadness (0.975) and a bit of on-task behavior (0.262); and a Calm/Off-task state (State 4) characterized by lack of anger (0.951), sadness (0.999) and on-task behaviors (1.000).
Table 3.
Results from 4-state HMM: Probability of Behavior and Emotions in each of the Four Latent States (invariant across ages 36 and 48 months)
| Probability of Observed Emotions and Behavior | ||||||
|---|---|---|---|---|---|---|
| Anger (0 = no) |
Anger (1 = yes) |
Sadness (0 = no) |
Sadness (1 = yes) |
On-task (0 = no) |
On-task (1 = yes) |
|
| State 1, Calm/On-task | 1.000 | 0.000 | 0.997 | 0.003 | 0.001 | 0.999 |
| State 2, Anger/On-task | 0.031 | 0.969 | 0.992 | 0.008 | 0.010 | 0.990 |
| State 3, Sadness/Limited On-task | 0.933 | 0.067 | 0.025 | 0.975 | 0.738 | 0.262 |
| State 4, Calm/Off-task | 0.951 | 0.049 | 0.999 | 0.001 | 1.000 | 0.000 |
Note. N = 117 (n = 114 at 36 months, n = 116 at 48 months); 0 = behavior (anger, sadness, or on-task) absent at that observation (i.e., 1-second epoch); 1 = behavior present in at that observation.
Age differences in initial state probabilities and transitions among these states are shown in Table 4. Initial state probabilities (top row of Table 4) indicate that at both 36 and 48 months, children were most likely to begin the task in the Calm/On-task state (π1 = 0.607 and 0.782 for 36 months and 48 months, respectively). Although these initial probabilities did not differ with age, there were significant age differences in how children transitioned among the four states, χ2(12) = 122.953, p < .001. These differences are descriptively evident from the implied latent state time series plots in Figure 2. Each row in the stack depicts how a child transitioned among states as they completed the task at age 36 months (left panel) and 48 months (right panel). For example, the bottom row of the left panel of Figure 2 depicts a 36-month-old child who spent the entire task in the Calm/Off-task state (State 4), shown as solid gray bar running from t = 0 to t = 150. In contrast, the top row in the left panel of Figure 2 depicts a 36-month-old child who spent the entire length of the task in a Calm/On-task state (State 1; green bar). Other children transitioned into and out of multiple states, shown as bars changing color as one moves from left to right. Some age-related differences can also be noticed in the plots. For example, there are more instances of yellow bars, the Anger/On-task state (State 2), at 48 months (right panel) than at 36 months (left panel).
Table 4.
Results from 4-state HMM: Age-related Differences from Ages 36 and 48 months in Initial State Probabilities and Transition Probabilities between Four Latent States.
| Calm/On-task | Anger/On-task | Sadness/Limited On-task | Calm/Off-task | |||||
|---|---|---|---|---|---|---|---|---|
| 36 months | 48 months | 36 months | 48 months | 36 months | 48 months | 36 months | 48 months | |
| Initial State Probability | 0.607 | 0.782 | 0.009 | 0.036 | 0.018 | 0.000 | 0.366 | 0.182 |
| Transition Probabilities (from row state to column state) | ||||||||
| Calm/On-task | 0.947 (M = 71.37, SD = 42.83) | 0.926 (M = 69.68, SD = 36.56) | 0.014 (M = 0.97, SD = 1.34) | 0.042 (M = 3.06, SD = 2.53) | 0.006 (M = 0.43, SD =1.06) | 0.004 (M = 0.34, SD = 0.81) | 0.034 (M = 2.53, SD = 1.77) | 0.027 (M = 2.10, SD = 1.54) |
| d = .07 | d = 1.08 | d = −.09 | d = .43 | |||||
| Anger/On-task | 0.226 (M = 0.96, SD = 1.39) | 0.270 (M = 2.83, SD = 2.49) | 0.731 (M =3.12, SD = 8.30) | 0.696 (M =7.59, SD = 14.56) | 0.015 (M = 0.06, SD = 0.28) | 0.006 (M = 0.06, SD = 0.30) | 0.029 (M = 0.11, SD = 0.35) | 0.028 (M = 0.28, SD = 0.64) |
| d = .96 | d = .39 | d = 0 | d = .33 | |||||
| Sadness/Limited On-task | 0.033 (M = 0.31, SD = 0.94 | 0.023 (M = 0.20, SD = 0.56) | 0.003 (M = 0.04, SD = 0.18) | 0.003 (M = 0.03, SD = 0.21) | 0.883 (M = 9.40, SD = 23.53) | 0.894 (M = 6.66, SD = 17.07) | 0.081 (M = 0.77, SD = 1.43) | 0.080 (M= 0.59, SD = 1.27) |
| d = .15 | d = .05 | d = .13 | d = .13 | |||||
| Calm/Off-task | 0.051 (M = 2.42, SD = 1.78) | 0.038 (M = 2.05, SD = 1.65) | 0.003 (M = 0.13, SD = 0.41) | 0.002 (M = 0.10, SD = 0.36) | 0.014 (M = 0.72, SD = 1.42) | 0.010 (M = 0.53, SD = 1.11) | 0.932 (M = 51.44, SD = 39.92) | 0.950 (M = 50.93, SD = 37.91) |
| d = .22 | d = .08 | d = .15 | d = .01 | |||||
Note. N = 117 (n = 114 at 36 months, n = 116 at 48 months; Parameters estimates are the initial probabilities (π matrix) and transition probabilities (a matrix) from HMM with age as a covariate; M = mean number of transitions, SD = standard deviation of number of transitions, as derived from most likely state sequences (Vertebi 1967) obtained for each individual at each age, d = effect size; Bold indicates significant age difference (p < .05) in number of specific state transitions.
Figure 2.
State sequence implied by 4-state HMM model with age differences in initial state probabilities and transition probabilites (left panel = 36 months, right panel = 48 months).
Note. State 1 = Calm/On-task, State 2 = Anger/On-task, State 3 = Sadness/Limited On-task, State 4 = Calm/Off-task. Anger/On-task (yellow) is more interspersed throughout the time series at 36 months (left panel) compared to 48 months panel (right panel); n = 114 (left panel), n = 116 (right panel).
Age-specific transition probabilities and transition counts (means and standard deviations) are shown in the bottom portion of Table 4 (significant dependent t-tests are bolded). Results indicated that children were more likely to transition from a Calm/On-task to Calm/Off-task state at 36 months than at 48 months (a14@36 = 0.034, a14@48 = 0.027; M36 = 2.53, M48 = 2.10); that children were more likely to remain in an Anger/On-task state at 36 months than at 48 months (a22@36 = 0.731, a22@48 = 0.696; M36 = 3.12, M48 = 7.59). Most interesting from a functionalist perspective, children were more likely to transition from Calm/On-task to Anger/On-task (a12@36 = 0.014, a22@48 = 0.042; M36 = 0.97, M48 = 3.06), and from Anger/On-task back to Calm/On-task (a21@36 = 0.226, a22@48 = 0.270; M36 = 0.96, M48 = 2.83) at 48 months than at 36 months. These differences can be seen in Figure 2 when looking at how the intermittent patches of yellow (Anger/On-task, State 2) are more prevalent in the right panel (48 months) than in the left panel (36 months).
Discussion
The present study used dynamic analytic methods to test relations between preschoolers’ emotion expressions and persistence at a frustrating task, and how these relations change during an important period for the development of emotional competence. Most research on young children’s anger is aimed at understanding detrimental influences on behavior in early childhood. We adopted the functionalist perspective (Barrett & Campos, 1987), which posits that anger entails increased effort at achieving a blocked goal, and sadness entails relinquishing the goal. We tested the possibility that anger can contribute to longer persistence during a blocked goal task, while sadness can contribute to giving up earlier. Moreover, we posited that this benefit should increase with age because during this age range children are thought to develop sense of agency and skill at regulating emotion (Abe & Izard, 1999; Kopp 1989).
In order to test these functional relations, we capitalized on time series data and used two dynamic modeling approaches. Survival analysis showed that although children’s time until completely stopping on-task behavior was similar at 36 and 48 months, two aspects of their behavior changed with age: the amount of anger displayed and its influence on persistence. At age 48 months, children displayed more anger and their anger appeared to facilitate their persistence. Hidden Markov models further revealed that at 48 months, children were more likely to transition from being angry to calm, and calm to angry, while staying on task compared to 36 months. We found no support in either model for predictions about sadness.
Our findings are consistent with another study that reported an increase in anger frequency in a similar task between ages 3 and 4 years (Chaplin, Klein, Cole, & Turpyn, & 2017), but that study did not examine the influence of anger on behavior. In the infancy literature, several studies indicated increased anger as infants began to crawl (Campos, Kermoian, & Zumbahlen, 1992; Roben et al., 2012; Zachry et al., 2015). An interpretation is that as infants gain experience being able to locomote, they are more readily angered when their movement is blocked. However, infant anger can often lead to less desirable behavior, such as distress or tantrums. In our study, while at age 36 months, greater anger was indeed associated with a greater likelihood of giving up earlier in the task, at 48 months, anger was associated with a lower likelihood of giving up earlier in the task. Though this association was small, our findings suggest that although children were angrier at age 48 months than at 36 months, their anger at 48 months contributed to more desirable and functional behavior. This finding extends prior research showing that anger is correlated with adaptive outcomes such as persistence in this type of task (e.g., Dennis et al., 2009; He et al., 2012), first, by suggesting directionality in this relation and, second, by providing the first longitudinal test of a functional perspective on anger. Although this finding requires replication with longer tasks and additional age groups, it also shows promise for a dynamic method like survival analysis in testing hypotheses regarding the development of the functionality of emotion.
The possibility that when children are younger, anger derails persistence, but when they are older, anger enhances persistence merits additional research, including potential explanations. For example, perhaps there is age-related change in the appraisals children make in task-specific contexts. The functionalist perspective defines emotion as both appraisal and action readiness (Frijda, 1986). Possibly, as children age during the preschool period, they acquire an increasing sense of agency and autonomy (Abe & Izard, 1999) such that, by 48 months, they may believe they can and should be able to unlock a box and get their toy; in that way, their anger aids persistent behavior. However, also important to note is that children’s anger in this task was predominantly low in intensity. That is, children displayed nonverbal expressions of low-level anger such as muttering with angry prosody, furrowing their brows, and pressing their lips. No children yelled or had a tantrum in this non-risk sample. At higher intensities, children’s persistence may be derailed regardless of age and may lead to more disruptive behavior (Cole et al., 1994; Potegal & Davidson, 2003). However, even this low-level anger had a detrimental influence on persistence at age 36 months, suggesting that the beneficial influence of anger for persistence may emerge as children develop.
Hidden Markov models provided additional information about preschoolers’ anger in relation to their task persistence. This method revealed four latent states: Anger/On-task, Calm/Off-task, Sadness/Limited On-task, and Calm/On-task. At both ages, most children were calm and engaging in effort at opening the box at the outset of the task. However, descriptively, the overall proportion of on-task behavior was lower at 36 months compared to 48 months. Age-related differences in children’s transitions between these four latent states appeared to explain this difference. At age 36 months, children were more likely to remain in the Anger/On-task state, or transition from Calm/On-task to Calm/Off-task, relative to their transitions at 48 months. In contrast, at 48 months, there was a higher probability that children transitioned from Anger/On-task to Calm/On-task and vice versa, relative to their transitions at 36 months. That is, by 48 months, children appeared better able to transition from anger to calm emotion displays, and back to anger, while staying on task. This finding supports the functionalist perspective that anger entails increased effort to overcome a blocked goal (Barrett & Campos, 1987) and further supports the claim that functionality may emerge as a function of age. At 48 months, children may have greater capacity to regulate and even harness their anger (Cole et al., 2011; Kopp, 1989), thereby facilitating their maintenance of persistence over the course of the task. Future research would benefit from examining behavioral strategies, emotion, and their influence on a task goal in order to test this inference directly (see Cole, Bendezú, Chow, & Ram, 2017; Gilliom et al., 2002).
Our predictions regarding sadness were not supported. The Hidden Markov modeling, however, revealed a latent state for sadness with limited on-task behavior, indicating that although sadness did not predict giving up, sadness co-occurred with periods of limited persistence. The locked box task is designed to elicit anger and frustration (Goldsmith et al., 1995) and children were shown how to unlock the box, such that the task may predispose children to appraise the situation as a blocked goal that can be overcome. There may be moments of sad appraisal, however, when children questioned whether they would open the box. An adequate test of the development of the functionality of sadness for appropriate goal relinquishment would require a different task.
Together, these findings also highlight the importance of analyzing behavior over time, in addition to studies that aggregate across time (Cole & Hollenstein, 2018). By employing survival analysis, we documented age-related changes in the relation between emotion prior to stoppingon task behavior and the timing of the “survival” of their persistence. HMM added to these results, utilizing the full time series, and further revealed transitional patterns that may underlie how children improve in maintaining this persistence at a blocked goal.
Limitations and Outlook
The results of the current study are preliminary but suggest that the development of functionality of emotion is an important area for future research to pursue. We discuss several limitations from this study that can be used to guide future work. First, although our study was longitudinal, the study of development is better when there are more than two age points. Because models of emotion regulation suggest important developments in the third year of life (Kopp, 1989) and the ability to manage anger is essential for school readiness (Denham et al., 1991), longitudinal studies that capture this age range could be particularly interesting. It could also be that at 48 months, children are only beginning to harness anger for functional behaviors, and that later ages could reveal stronger associations between anger and behaviors such as persistence.
Second, the original lock box task is designed to be 150 seconds long. It may be that a longer task would better assess the extent to which young children may resume trying after a long period of being off-task or give up permanently. In longitudinal studies, there is always the challenge of making tasks both comparable and age-appropriate. Future studies should consider the length of a task designed to investigate the influence of emotion on task persistence. In our study, at age 48 months, the research assistant checked with children at the 150 second mark and then encouraged them to keep trying, in case children needed more time to become frustrated. We had not anticipated that at 48 months children would be angrier than they had been at age 36 months. To compare these two ages, we only compared the first 150 seconds, and cannot address what would have happened if we had observed longer at age 36 months.
Third, although particularly useful in early childhood research, emotions cannot be conceptualized solely by facial expressions, and there may be other indicators of emotionality that this study did not measure (Barrett & Campos, 1987). Further studies with multiple measures, e.g., heart rate variability as an index of physiological regulation, (Calkins & Fox, 2002) and assessment of emotion regulation attempts, could capture processes that were not addressed in this study. Relatedly, emotion expressions are embedded within cultural norms and values. The extent to which these findings, obtained from a sample of economically-strained families living in rural and semi-rural communities in a Mid-Atlantic region in the United States, generalize to other samples is unknown and warrants further testing.
Finally, this study capitalized on two dynamic methods, which offer both new insights and limitations to be considered in future research. One limitation was that due to either features of the data, features of the depmixS4 package, or both, we were not able to directly test which cells of the transition matrix differed by age and were only able to test whether the whole transition matrix differed with age. A Bayesian estimation approach could avoid this problem.
Conclusions
Because most related research addresses the important issue of how anger interferes with children’s appropriate behavior, there are few investigations of anger’s functionality and how this functionality may emerge at a crucial developmental period for children’s emotional competence. In the current study, anger contributed to giving up on persisting at a blocked goal at 36 months, but enhanced children’s persistence at 48 months. We have posited that this change may reflect growing agency and an appraisal that 48-month-olds should be able to do the task. Further, we documented age-related change in the process of maintaining longer overall persistence. At 36 months, children were more likely to transition from calm on-task states to calm-off task ones compared to at 48 months. More notably, at 48 months, children were more likely to move flexibly between calm and angry on-task states compared to 36 months. We posited that this may reflect a growing ability to modulate and harness anger, suggesting that anger is integral to a 48-month-old’s maintenance of persistence. We highlight that by using dynamic modeling approaches, we were able to operationalize persistence in meaningful ways (i.e., when a child abandons on-task behavior completely), and to examine how emotion and persistence unfold together time. These dynamic approaches extend our understanding of functionalist theories of emotional development.
Table 1.
Descriptive Statistics for Study Variables
| 36 months M (SD) |
48 months M (SD) |
Observed Range | p-value | Effect size, r | |
|---|---|---|---|---|---|
| Proportion of Anger | .06 (.10) | .10 (.12) | 0-.87 | .001* | .33 |
| Proportion of Sadness | .06 (.16) | .04 (.11) | 0-.95 | .155 | .10 |
| Time until Giving up | 113.28 (47.3) | 119.44 (42.08) | 1–150 | .420 | .09 |
| Proportion of On-Task Behavior | .52 (.30) | .59 (.28) | 0–1 | .042* | .19 |
| N | 114 | 116 |
Note. Time measured in seconds; M = Mean, SD = Standard Deviation;
denotes p < .05 in Wilcoxon test of mean differences across age
Acknowledgments
The research reported here was supported in part by a training grant from the Institute of Education Sciences (R305B090007), awarded to the first author; a research grant from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01-HD076994), awarded to the third author; and a research grant from the National Institute of Mental Health (HD076994), awarded to the third and fourth authors. Opinions expressed are those of the authors and do not necessarily represent the granting agencies.
Contributor Information
K. Ashana Ramsook, Department of Psychology, The Pennsylvania State University;.
Lizbeth Benson, Department of Human Development and Family Studies, The Pennsylvania State University;.
Nilam Ram, Department of Human Development and Family Studies, The Pennsylvania State University..
Pamela M. Cole, Department of Psychology, The Pennsylvania State University;
References
- Abe JAA, & Izard CE (1999). The developmental functions of emotions: An analysis in terms of differential emotions theory. Cognition & Emotion, 13, 523–549. doi: 10.1080/026999399379177 [DOI] [Google Scholar]
- Barrett KC, & Campos JJ (1987). Perspectives on emotional development II: A functionalist approach to emotions. In Osofsky JD (Ed.), Handbook on infant development (pp. 555–578). New York, NY: Wiley. [Google Scholar]
- Biringen Z, Campos JJ, Emde RN, & Appelbaum M (2008). Development of autonomy: Role of walking onset and its timing. Perceptual and Motor Skills, 106, 395–414. doi: 10.2466/pms.106.2.395-414 [DOI] [PubMed] [Google Scholar]
- Blair C (2002). School readiness: Integrating cognition and emotion in a neurobiological conceptualization of children’s functioning at school entry. American Psychologist, 57, 111–127. doi: 10.1037//0003-066x.57.2.111 [DOI] [PubMed] [Google Scholar]
- Buss KA, & Kiel EJ (2004). Comparison of sadness, anger, and fear facial expressions when toddlers look at their mothers. Child Development, 75, 1761–1773. doi: 10.1111/j.1467-8624.2004.00815.x [DOI] [PubMed] [Google Scholar]
- Calkins SD, & Fox NA (2002). Self-regulatory processes in early personality development: A multilevel approach to the study of childhood social withdrawal and aggression. Development and Psychopathology, 14, 477–498. doi: 10.1017/S095457940200305X [DOI] [PubMed] [Google Scholar]
- Campos JJ, Campos RG, & Barrett KC (1989). Emergent themes in the study of emotional development and emotion regulation. Developmental Psychology, 25, 394–402. doi: 10.1037/0012-1649.25.3.394 [DOI] [Google Scholar]
- Chaplin TM, Klein MR, Cole PM, & Turpyn CC (2017). Developmental change in emotion expression in frustrating situations: The roles of context and gender. Infant and Child Development, 26, e2028. doi: 10.1002/icd.2028 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cohen J (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220. doi: 10.1037/h0026256. [DOI] [PubMed] [Google Scholar]
- Cole PM, Crnic K, Nelson K, & Blair C (2000). Development of Toddlers Study. Cole PM, & Hollenstein T (Eds.). (2018). Emotion regulation: A matter of time. New York, NY: Routledge. [Google Scholar]
- Cole PM, Martin SE, & Dennis TA (2004). Emotion regulation as a scientific construct: Methodological challenges and directions for child development research. Child Development, 75, 317–333. doi: 10.1111/j.1467-8624.2004.00673.x [DOI] [PubMed] [Google Scholar]
- Cole PM, Tan PZ, Hall SE, Zhang Y, Crnic KA, Blair CB, & Li R (2011). Developmental changes in anger expression and attention focus: learning to wait. Developmental psychology, 47, 1078–1089. doi: 10.1037/a0023813 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Day KL, & Smith CL (2013). Understanding the role of private speech in children’s emotion regulation. Early Childhood Research Quarterly, 28, 405–414. doi: 10.1016/j.ecresq.2012.10.003 [DOI] [Google Scholar]
- Denham SA (1998). Emotional development in young children. New York, NY: Guilford Press. [Google Scholar]
- Denham SA (2006). Social-emotional competence as support for school readiness: What is it and how do we assess it? Early education and development, 17, 57–89. doi: 10.1207/s15566935eed1701_4 [DOI] [Google Scholar]
- Denham SA, Zahn-Waxler C, Cummings EM, & Iannotti RJ (1991). Social competence in young children’s peer relations: Patterns of development and change. Child psychiatry and human development, 22, 29–44. doi: 10.1007/BF00706057 [DOI] [PubMed] [Google Scholar]
- Dennis TA, Cole PM, Wiggins CN, Cohen LH, & Zalewski M (2009). The functional organization of preschool-age children’s emotion expressions and actions in challenging situations. Emotion, 9, 520–530. doi: 10.1037/a0016514 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fredricks JA, Blumenfeld PC, & Paris AH (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59–109. doi: 10.3102/00346543074001059 [DOI] [Google Scholar]
- Frijda NH (1986). The emotions: Studies in emotion and social interaction. Paris: Maison de Sciences de l’Homme. [Google Scholar]
- Gilliom M, Shaw DS, Beck JE, Schonberg MA, & Lukon JL (2002). Anger regulation in disadvantaged preschool boys: Strategies, antecedents, and the development of self-control. Developmental Psychology, 38, 222–235. doi: 10.1037/0012-1649.38.2.222 [DOI] [PubMed] [Google Scholar]
- Gilliam WS, & Shahar G (2006). Preschool and child care expulsion and suspension: Rates and predictors in one state. Infants & Young Children, 19, 228–245. doi: 10.1097/00001163-200607000-00007 [DOI] [Google Scholar]
- Goldsmith HH, Reilly J, Lemery KS, Longley S, & Prescott A (1995). Laboratory temperament assessment battery: Preschool version. Unpublished manuscript. [Google Scholar]
- Goodenough FL (1931). Anger in young children. Oxford, England: University of Minnesota. [Google Scholar]
- Halberstadt AG, Denham SA, & Dunsmore JC (2001). Affective social competence. Social development, 10, 79–119. doi: 10.1111/1467-9507.00150 [DOI] [Google Scholar]
- He J, Xu Q, & Degnan KA (2012). Anger expression and persistence in young children. Social Development, 21, 343–353. doi: 10.1111/j.1467-9507.2011.00622.x [DOI] [Google Scholar]
- Hollenstein T, & Lewis MD (2006). A state space analysis of emotion and flexibility in parent-child interactions. Emotion, 6, 656–662. doi: 10.1037/1528-3542.6.4.656 [DOI] [PubMed] [Google Scholar]
- Izard CE (1977). Human Emotions. New York, NY: Plenum Press. [Google Scholar]
- Jahromi LB, & Stifter CA (2008). Individual differences in preschoolers’ self-regulation and theory of mind. Merrill-Palmer Quarterly, 54, 125–150. doi: 10.1353/mpq.2008.0007 [DOI] [Google Scholar]
- Keltner D, & Gross JJ (1999). Functional accounts of emotions. Cognition & Emotion, 13, 467–480. doi: 10.1080/026999399379140 [DOI] [Google Scholar]
- Kopp CB (1989). Regulation of distress and negative emotions: A developmental view. Developmental Psychology, 25, 343–354. doi: 10.1037//0012-1649.25.3.343 [DOI] [Google Scholar]
- Lench HC, & Levine LJ (2008). Goals and responses to failure: Knowing when to hold them and when to fold them. Motivation and Emotion, 32, 127–140. doi: 10.1007/s11031-008-9085-1 [DOI] [Google Scholar]
- Lench HC, Tibbett TP, & Bench SW (2016). Exploring the toolkit of emotion: What do sadness and anger do for us? Social and Personality Psychology Compass, 10, 11–25. doi: 10.1111/spc3.12229 [DOI] [Google Scholar]
- Lewis M, Alessandri SM, & Sullivan MW (1990). Violation of expectancy, loss of control, and anger expressions in young infants. Developmental Psychology, 26, 745–751. doi: 10.1037/0012-1649.26.5.745 [DOI] [Google Scholar]
- Lewis M, Sullivan MW, & Kim HMS (2015). Infant approach and withdrawal in response to a goal blockage: Its antecedent causes and its effect on toddler persistence. Developmental psychology, 51, 1553–1563. doi: 10.1037/dev0000043 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lougheed JP, Benson L, Cole PM, & Ram N (2019). Multilevel survival analysis: Studying the timing of children’s recurring behaviors. Developmental Psychology, 55, 53–65. doi: 10.1037/dev0000619 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Martin A, Ryan RM, & Brooks-Gunn J (2013). Longitudinal associations among interest, persistence, supportive parenting, and achievement in early childhood. Early Childhood Research Quarterly, 28, 658–667. doi: 10.1016/j.ecresq.2013.05.003 [DOI] [Google Scholar]
- Mokrova IL, O’Brien M, Calkins SD, Leerkes EM, & Marcovitch S (2013). The role of persistence at preschool age in academic skills at kindergarten. European Journal of Psychology of Education, 28, 1495–1503. doi: 10.1007/s10212-013-0177-2 [DOI] [Google Scholar]
- Potegal M, & Davidson RJ (2003). Temper tantrums in young children: 1. Behavioral composition. Journal of Developmental & Behavioral Pediatrics, 24, 140–147. doi: 10.1097/00004703-200306000-00002 [DOI] [PubMed] [Google Scholar]
- Rabiner LR (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77, 257–286. doi: 10.1109/5.18626 [DOI] [Google Scholar]
- Roben CKP, Bass AJ, Moore GA, Murray-Kolb L, Tan PZ, … & Teti LO (2012). Let me go: The influences of crawling experience and temperament on the development of anger expression. Infancy, 17, 558–577. doi: 10.1111/j.1532-7078.2011.00092.x [DOI] [PubMed] [Google Scholar]
- Saarni C (1998). Issues of cultural meaningfulness in emotional development. Developmental Psychology, 34, 647–652. doi: 10.1037/0012-1649.34.4.647 [DOI] [PubMed] [Google Scholar]
- Skinner E, Pitzer J, & Brule H (2014). The role of emotion in engagement, coping, and the development of motivational resilience. In Pekrun R, & Linnenbrink-Garcia L (Eds.), International Handbook of emotions in education (pp. 331–347) Routledge/Taylor & Francis Group: New York, NY. [Google Scholar]
- Tan L, & Smith CL (2018). Function of child anger and sadness in response to a blocked goal. Journal of experimental child psychology, 170, 190–196. doi: 10.1016/j.jecp.2018.01.005 [DOI] [PubMed] [Google Scholar]
- Therneau TM, Grambsch PM, & Pankratz VS (2003). Penalized survival models and frailty. Journal of computational and graphical statistics, 12, 156–175. doi: 10.1198/1061860031365 [DOI] [Google Scholar]
- Therneau TM, & Therneau MTM (2015). Package ‘coxme’. R package version, 2. [Google Scholar]
- Visser I (2011). Seven things to remember about hidden Markov models: A tutorial on Markovian models for time series. Journal of Mathematical Psychology, 55, 403–415. doi: 10.1016/j.jmp.2011.08.002 [DOI] [Google Scholar]
- Visser I, & Speekenbrink M (2010). depmixS4: an R package for hidden Markov models. Journal of Statistical Software, 36, 1–21. [Google Scholar]
- Viterbi A (1967). Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13, 260–269. doi: 10.1109/TIT.1967.1054010 [DOI] [Google Scholar]
- Zachry AH, Chappell LH, Cox VH, Lopez EH, Mitchell AW, & Woodard L (2015). Differences in the angry behaviors of precrawling and crawling infants. Journal of Education and Human Development, 4, 67–72. doi: 10.15640/jehd.v4n2_1a7 [DOI] [Google Scholar]


