Contextual Influences on Concordance between Maternal Report and Laboratory Observation of Toddler Fear

Abstract

Emotion and temperament researchers have faced an enduring issue of how to best measure children’s tendencies to express specific emotions. Inconsistencies between laboratory observation and parental report have made it challenging for researchers to determine the utility of these different forms of measurement. The current study examined the effect of laboratory episode characteristics (i.e., threat-level of the episode, maternal involvement) on concordance between maternal report and laboratory observation of toddler fear. The sample included 111 mother-toddler dyads who participated in a laboratory assessment when toddlers were approximately 24-months-old. Toddler fear was assessed both via maternal report and observation from a number of laboratory episodes that varied in their level of threat and whether mothers were free or constrained in their involvement in the task. Results indicated that maternal report related to the observed fear composites for low-threat, but not high-threat episodes. On the other hand, maternal involvement in the laboratory episodes did not moderate the relation between maternal report and laboratory observation of fear. These results suggest that the threat-level of laboratory episodes designed to elicit fear, but not maternal involvement in these episodes, may be important to take into consideration when assessing their relation to maternal report of fear and fearful temperament.

Individual differences in the biologically-based proclivity to express particular emotions in response to the environment have been theorized to represent temperament (Buss & Plomin, 1984; Goldsmith & Campos, 1982; Rothbart, 1989; Shiner et al., 2012). The study of emotions from a temperament framework has faced a longstanding issue of how best to measure these dispositions, with parental (most often maternal) report providing an efficient and economical but potentially biased assessment, and laboratory observation providing a seemingly more objective but also more costly and intensive assessment. The small to moderate relations that tend to be found between these two methods (Durbin & Wilson, 2012; Olino, Durbin, Klein, Hayden, & Dyson, 2013; Seifer, Sameroff, Barrett, & Krafchuk, 1994; Stifter, Willoughby, & Towe-Goodman, 2008) have prompted a number of investigations attempting to discern the nature of the discrepancies. Most of this prior research has focused on maternal characteristics of personality and psychopathology that relate to maternal perceptions or affect the extent to which maternal report relates to laboratory observation (e.g., Bates & Bayles, 1984; Durbin & Wilson, 2012; Leerkes & Crockenberg, 2003). However, laboratory observations of emotion vary as well, so characteristics of the nature of those procedures may also shed light on the association with maternal report. It has been suggested that mothers and observers should use similar situations to rate child behavior to maximize concordance (Rothbart & Bates, 2006). Contextual influences on concordance may be of particular interest in the realm of temperamental fear, which has been the focus of many studies of both typical and atypical development (e.g., Buss, 2011; Fox, Henderson, Marshall, Nichols, & Ghera, 2005; Kagan, Reznick, Clarke, Snidman, & Garcia-Coll, 1984; Kochanska, 1995). Understanding how characteristics of laboratory procedures influence the strength of the relation between observation and maternal report of fear could add clarity to the assessment of temperament and thus impact temperament research and theory more broadly. We review literature on fearful temperament and issues of concordance and informant discrepancies to frame the current study.

Fearful temperament (also, behavioral inhibition when studied as a discrete category of children) is conceptualized as wariness, hesitance, and distress in response to novel and uncertain situations (Kagan et al., 1984). Toddlerhood is a common developmental period for the assessment of fearful temperament because the motor and language developments that begin to give toddlers some independence in their environments also highlight individual differences in self-regulation in the face of challenge (Calkins, 2007). Whether studied dimensionally as fearful temperament or categorically as behavioral inhibition, high levels of fear in toddlerhood predict social withdrawal across childhood and adolescence, as well as risk for the development of anxiety and related problems (e.g., Biederman et al., 1990; Gartstein et al., 2010; Hudson, Dodd, & Bovopoulos, 2011; Kagan, Snidman, Zentner, & Peterson, 1999; Kiel & Buss, 2011; Rapee & Coplan, 2010). Given the implications of fearful temperament for children’s development, there exists a substantial literature on its phenomenology, predictors, and consequences (see Buss & Kiel, 2013; Fox et al., 2005; Kagan, 2012 for reviews).

Studies use both standardized laboratory procedures and parent-report questionnaires to assess temperamental fear. Laboratory paradigms expose young children to novel social (e.g., unfamiliar peers and adults) and non-social stimuli (e.g., mobiles, tunnel, remote-controlled toys) (e.g., Buss & Goldsmith, 2000; Kagan et al., 1984; Fox et al., 2005; Goldsmith & Rothbart, 1999). Well-established parent-report temperament questionnaires, such as the Toddler Behavior Assessment Questionnaire (Goldsmith, 1996) and the Early Childhood Behavior Questionnaire (Putnam, Garstein, & Rothbart, 2006) include scales of both object fear (e.g., reactions to animals, visiting new places, loud noises) and shyness/social fear (e.g., reactions to new children, parents’ friends, babysitters, strangers). Although both object fear and social fear have been incorporated into composites of fearful temperament/behavioral inhibition, it should be acknowledged that they have also been studied individually and may have different developmental trajectories and relations to risk for maladaptation (Dyson, Klein, Olino, Dougherty, & Durbin, 2011; Kochanska & Radke-Yarrow, 1992; Rubin, Burgess, & Hastings, 2002).

Maternal reports are often the favored measure of children’s temperament because they are easy to complete and capitalize on mothers’ broad range of experiences with their children (Rothbart & Bates, 2006), but they often relate only modestly to laboratory observation (e.g., Olino et al., 2013). Other studies find little convergence between report and observation (e.g., Seifer et al., 1994). Concordance may differ depending on the temperamental construct being measured; for example, Stifter et al. (2008) found moderate concordance between parent report and observation on infant positivity but little to no convergence with respect to infant negativity. This is similar to patterns of convergence found in adult personality, such that greater agreement has been found between self- and informant-ratings for positive emotionality than negative emotionality (Harkness, Tellegen, & Waller, 1995).

It is possible that deficits in concordance result from biases that influence how parents complete questionnaire measures. De Los Reyes and Kazdin’s (2005) Attribution Bias Context Model provides a theoretical framework for understanding discrepancies in reports among different informants. This theory posits that each informant (parent, teacher, child) has a unique perspective and potential biases arising from psychopathology, personality, and memory bias for negative events that motivate questionnaire ratings. In the realm of emotions and temperament, maternal characteristics have indeed been found to impact concordance (Durbin & Wilson, 2012; Leerkes & Crockenberg, 2003). More recent theoretical work on informant discrepancies, such as the Operations Triad Model put forth by De Los Reyes, Thomas, Goodman, and Kundey (2013), further suggests that discrepancies may reflect meaningful variation in behavior. Indeed, laboratory observation of behavior has been used to elucidate the meaning of divergence between mothers and other reporters, such as fathers or teachers, for disruptive behavior in children (De Los Reyes, Henry, Tolan, & Wakschlag, 2009) and anxious behavior in adults (De Los Reyes, Bunnell, & Beidel, 2013). Within this framework, it is suggested that divergence could reflect methodological processes, in addition to other sources. In line with this, we suggest that contextual characteristics of laboratory paradigms may influence concordance between observation and maternal report. Little research has addressed this issue for toddler fear, but enough existing evidence warrants investigation of two such variables: the level of threat and the presence/absence of maternal involvement.

The level of threat of a fear-eliciting task may influence how toddlers behave in the laboratory. The threat-level of a fear-eliciting task has been indexed by the typical level of fear behavior (e.g., fear expressions, avoidance/shyness, low approach) displayed across children, such that high-threat episodes have higher mean scores than low-threat episodes (Buss, 2011). Higher-threat episodes have been characterized by the stimulus being unpredictable, coming close in proximity to the child, and the child having fewer coping resources available, whereas lower-threat episodes use more predictable stimuli, provide space between the child and stimulus, and offer resources like toys with which to play (Buss, 2011; Buss, Davidson, Kalin, & Goldsmith, 2004). The extent to which emotional displays match the eliciting context (e.g., high- versus low-threat) can be understood from broader theory on emotion regulation, which comprises the internal and external processes by which individuals maintain or modulate emotion expressions and experiences in order to interact with their environments (Cole, Martin, & Dennis, 2004). It would be expected that adaptive emotion regulation would be indicated by lower levels of fear when lower levels of threat are present, and deficits in emotion regulation would be indicated by the inability to modulate fear in this manner. Indeed, toddlers who display high levels of fear in low-threat contexts (“dysregulated fear”) have been shown to have more dysregulated physiology and be at higher risk for anxiety problems than children only showing high levels of fear in a more highly threatening context (Buss, 2011; Buss et al., 2004; Buss et al., 2013). Thus, fear behavior exhibited in lower threat contexts may be more reflective of dispositional tendencies, whereas high fear in a more highly threatening context may be a more universal response related to external versus internal causes. This may be particularly important when considering the concordance between maternal report and observations of fear, as mothers’ reports tend to be based on lower threat, everyday activities (e.g., going to the doctor, having their child meet an unfamiliar adult or babysitter) rather than higher threat situations that occur less frequently on a day-to-day basis. Thus, the threat-level of the laboratory episode during which temperament is being assessed may be important to consider when examining agreement between maternal report and observational ratings of temperament.

Further, given that mothers tend to be involved in the situations on which they are reporting, it is possible that laboratory tasks in which mothers are instructed to act naturally, as opposed to remaining uninvolved, would more highly relate to maternal report of fear. Some researchers have suggested that maternal behaviors during observations may moderate children’s behavior, thereby explaining some of the discrepancies between maternal report and observations of temperament (Leerkes & Crockenberg, 2003; Rothbart & Goldsmith, 1985). Indeed, Roque et al. (2013) found that maternal involvement related to an increase in toddlers’ behavioral emotion regulation strategies (e.g., proximity/contact seeking to mother) in fear-eliciting episodes. Additionally, Grolnick, Bridges, and Connell (1996) found that toddlers used different regulatory behaviors when mothers were told to be active versus passive with their toddler. However, whether the mother was involved or uninvolved did not moderate the relation between mother-reported and observed distress to novelty in a sample of 6-month old infants (Leerkes & Crockenberg, 2003). Given the young age of children in this latter study, however, most evidence converges to suggest that maternal involvement could strengthen concordance.

The Current Study

Given, first, theory and empirical evidence for the importance of understanding the nature of concordance versus discrepancy between maternal report and laboratory observation of emotion-based temperament, and, second, a nearly exclusive focus on maternal characteristics that influence concordance in the current literature, the current study sought to examine contextual aspects of laboratory observation that may affect the relation between maternal report and observation of toddlers’ behavioral indicators of fear. In line with the Operations Triad Model (De Los Reyes, Thomas et al., 2013), we suggest that patterns of convergence versus discrepancy between maternal report and observation of fear reflect meaningful variation in behavior, rather than mere measurement error. We hypothesized that the threat-level of the episode would moderate the relation between maternal report and laboratory observation, such that maternal report would relate most strongly to observation for low-threat, compared to high-threat, laboratory episodes. We also hypothesized that maternal involvement would moderate the relation between maternal report and laboratory observation, such that maternal report would most strongly relate to observation for mother-involved, versus mother-uninvolved, episodes.

Method

Participants

The sample comprised 111 mother-toddler dyads who participated in a laboratory assessment when toddlers (49 female) were approximately 24-months old (M = 24.73, SD = 0.70 months). Participants were recruited from locally published birth announcements as well as from the local chapter of the Women, Infants, and Children’s program, which provides resources to low-income mothers with young children. Mothers were primarily European American (89%), although other racial backgrounds were represented (2% African American, 6% Asian American, 1% American Indian, 2% other racial background). One mother reported being Hispanic/Latina. Socioeconomic status (SES) was determined using the Hollingshead Four-Factor Index, which can range from values of 8 to 66, with higher values indicating higher SES. The sample was, on average, middle class (M = 50.03, SD = 11.66, Range = 17 – 66).

Procedure

All procedures were approved by the campus Institutional Review Board. Upon a mother expressing interest in the study, laboratory staff scheduled a laboratory visit for the mother and toddler, and mothers were mailed a consent form and packet of questionnaires to complete prior to the visit. At the laboratory, a primary experimenter (E1) acclimated the mother and toddler to the upcoming procedures, explaining that they would be participating in a variety of activities (“episodes”). E1 then explained the following episodes (which progressed in the order listed for all participants), as well as several others that are not included in the current study. Episodes were modeled after standardized tasks in the Laboratory Temperament Assessment Battery – Toddler Version (Buss & Goldsmith, 2000) and studies elsewhere in the literature (Buss, 2011; Nachmias, Gunnar, Mangelsdorf, Parritz, & Buss, 1996).

For the first two episodes, mothers were instructed to remain relatively uninvolved and refrain from spontaneous interaction with their toddlers. They were told that they could respond to their toddlers if they asked questions or expressed distress. In Stranger Approach, the toddler had several toys with which to play while the mother sat in a chair in the corner of the room. After 30 seconds, an unfamiliar male experimenter wearing a baseball cap came into the room. After 5 seconds, he engaged with the toddler, following a standardized script of questions (e.g., “Are you having fun today?”) that lasted approximately 1 minute. He then excused himself and left the room. For the Robot episode, the toddler started on the mother’s lap, and a remote-controlled robot stood on a low wooden platform in the opposite corner. Controlled from another room behind a one-way mirror, the robot moved and made noises randomly for 1 minute, after which E1 came in and gave the toddler up to three prompts to touch the robot.

For the remaining three episodes, mothers were instructed to be free in their involvement. In the Clown episode, a second experimenter (E2) entered the room, dressed in a clown costume, rainbow wig, and red nose. With a friendly demeanor and according to a standardized script, she introduced herself and invited the child to play three games (blowing bubbles, catch with beach balls, musical instruments) that lasted 1 minute each. After the second game, she took off her wig and nose. She invited the toddler to help her clean up, then said goodbye and left the room. In Puppet Show, E2 controlled two plush animal puppets from behind a wooden stage. The puppets interacted with the child in a friendly manner according to a standardized script and invited the child to play three games (catch, fishing, presentation of a sticker) that each lasted approximately 1 minute. At the end of the puppet show, E2 emerged from behind the stage, took off the puppets, and invited the toddler to play with them as she left. Finally, in Spider, the toddler began seated in the mother’s lap. A large plush spider affixed to a remote-controlled truck sat in the opposite corner. After 10 seconds, the spider, controlled by remote from behind a one-way mirror, approached and withdrew from the mother and toddler twice, with 10 second pauses in between each movement. E1 then entered the room and gave the toddler up to three friendly prompts to touch the spider.

All episodes were recorded for later behavioral coding. Coders received 15–20 hours of training from a master coder and were required to achieve a minimum threshold of reliability (intraclass correlation coefficient [ICC] or kappa ≥ .80) prior to coding independently. The master coder double-scored approximately 20% of cases throughout coding and met with coders to prevent coder drift. Final inter-rater reliability estimates are presented within descriptions of the codes as ranges of kappas or ICCs for categorical versus continuous scales, respectively, determined across the episodes. Of note, reliabilities did not appear to differ systematically across episodes.

Measures

Observed toddler fear behaviors

Toddlers were scored for a variety of behaviors within each episode. Toddlers were scored for proximity to mother (kappas = .71 to .99) as 0 (greater than 2 feet from mother), 1 (within 2 feet of mother), or 2 (touching mother) on a second-by-second basis. Attempt to be held (ICCs = .78 to .98) was scored as 0 (no attempt), 1 (some contact seeking or acquiescence to being picked up), 2 (puts arms out or verbalizes request to be picked up), or 3 (active clamber to be picked up) each 10 second epoch of the episode. Coders also scored distress vocalizations (ICCs = .68 to .97) within 10 second epochs as 0 (no distress), 1 (short duration or mild distress), 2 (low intensity cry), or 3 (full intensity cry/scream). Scores for these first three scores were averaged across epochs of the respective episode for final variables. One score of shyness (ICCs = .69 to .88), or the extent to which the toddler displayed avoidance of, withdrawal from, or discomfort with the stimulus, was given as a holistic assessment of the episode on a 1 (none) to 5 (extreme) scale. A score of boldness/approach (ICCs = .83 to .96), or the extent to which the toddler took initiative to increase proximity to and engage with the stimulus, was provided on the same scale.

For some preliminary analyses, variables remained in raw form. For other preliminary and for primary analyses, the five behaviors were standardized (after reversing boldness/approach) and averaged within each episode. Cronbach’s alphas suggested that these five behaviors were internally consistent for each episode (αs = .80 to .87).

Maternal report of temperamental fear

Mothers completed the Toddler Behavior Assessment Questionnaire (TBAQ; Goldsmith, 1996), a 110-item parent-report measure of several domains of toddler temperament. The TBAQ is a widely used instrument that has shown good internal consistency (αs > .80), moderate inter-parent agreement, and convergent validity with other parent-report temperament measures (Goldsmith, 1996). The current study uses the Social Fear (10 items; e.g., “When your child was approached by a stranger when you and s/he were out [for example, shopping], how often did your child babble or talk?”) and Object Fear (10 items, e.g., ““When a dog or other large animal approached your child, how often did s/he cling to you fearfully?”) scales, given that most of the laboratory episodes had both social and non-social components. The Social Fear scale asks about child clinginess, distress, talking less than usual, and, in reversed items, quickly warming up, enthusiasm, and exploration in response to going to the doctor’s office/clinic, interacting with a stranger or parents’ friends (both at home and when “out”), going to a new childcare setting, and being left at home. The Object Fear scale asks about similar reactions to heights, going through a tunnel or over a bridge, being approached by a large animal, being suddenly in the dark, seeing an insect up close, experiencing a loud storm and other loud noises, seeing blood on the self or others, and being in a crowd or unfamiliar places. Items were averaged within each scale (Social Fear α = .84, Object Fear α = .82), and then the two scales (r[109] = .54, p < .001) were averaged to yield the final variable of maternal report of fear. Of note, results were replicated if using the Social Fear or Object Fear scales individually, more information about which is available from the authors.

Data Analysis Plan

Given that the primary purpose of the current study was to test context-specific relations between observed and mother-reported fear, analyses proceeded in the following manner. We first present descriptive statistics and analyses to determine covariates to include in primary analyses. Then, we present bivariate correlations between maternal report of fear and each of the five episodes. Next, we compared means of observed toddler fear behaviors among the five episodes to validate their threat-level. Previous research has suggested that Stranger Approach, Clown, and Puppet Show be considered low to moderate threat episodes, and that Robot and Spider be considered high threat episodes (Buss, 2011; Kiel & Buss, 2012). To determine empirically whether these designations fit the current data, we examined mean values of the raw variables (prior to standardizing) and performed a repeated measures multivariate analysis of variance (RM-MANOVA), in which the five toddler behaviors were repeated across the within-subject factor of episode.

Finally, we used multilevel modeling to test our primary hypotheses of context-specific concordance between maternal report and observation of fear. Because episodes were nested within participants, multilevel modeling accounts for the nesting of episodes (i.e., observations) within participants while modeling the interaction between either episode threat or mother involvement and mother-reported fear in laboratory observation of fear behavior. Models were two levels, with episodes (Level 1) nested within participants (Level 2). Thus, variables that varied across episodes (i.e., toddler fear composites, threat-level, maternal involvement) resided at Level 1, and variables that were constant across episodes but varied across participants (i.e., demographic variables, maternal report of fear) resided at Level 2. After preliminary models tested the extent to which the nesting accounted for variance in scores and the overall relation between maternal report and observation of fear, the roles of threat-level and maternal involvement were tested in separate models so that Stranger Approach could be included in the analysis of maternal involvement (as we describe below, Stranger Approach did not fit neatly into a threat-level category). In each model, toddler fear composites (a Level 1 variable) served as the dependent variable. SES and gender (dummy coded with females = 1) were included as covariates in all models. Maternal report of fear was centered at its mean prior to creation of interaction terms. To address our first hypothesis, that maternal report would relate to observation of fear more highly in low-threat than high-threat episodes, we added threat-level (dummy coded with low threat = 1) as well as its interaction (cross-product) with maternal report of fear. To address our second hypothesis, that maternal report would relate more highly to observed fear for episodes in which mothers were involved, maternal involvement (dummy coded with involved =1), and its interaction with maternal report were included in a separate model.

Results

Preliminary Analyses

Descriptive statistics of raw toddler fear behaviors, fear behavior composites, and maternal report of fear are presented in Table 1. All values of skew for the fear behavior composites were within reasonable range (< |2.00|). SES related to the Clown fear behavior composite (r[109] = .31, p = .001). Girls had significantly higher fear behavior composites in Stranger Approach (girls: M = 0.24, SD = 0.88; boys: M = −0.20, SD = 0.53; t[74.30] = 3.09, p = .003, Cohen’s d = 0.61), Robot (girls: M = 0.20, SD = 0.79; boys: M = −0.15, SD = 0.81; t[109] = 2.27, p = .025, Cohen’s d = 0.44), Puppet Show (girls: M = 0.15, SD = 0.87; boys: M = −0.13, SD = 0.62; t[83.57] = 1.96, p = .053, Cohen’s d = 0.37), and Spider (girls: M = 0.21, SD = 0.69; boys: M = −0.17, SD = 0.87; t[109] = 2.49, p = .014, Cohen’s d = 0.48). Thus, SES and gender were included as covariates in primary analyses. Notably, patterns of significance did not differ whether or not these variables were included in the models, more information about which is available from the authors.

Table 1.

Descriptive Statistics

Variable	Mean (SD)	Range
Boldness
Puppet Show	3.63 (0.93)a	1.00 – 5.00
Clown	3.31 (1.14)b	1.00 – 5.00
Stranger Approach	2.81 (1.14)c	1.00 – 5.00
Robot	2.55 (1.37)c	1.00 – 5.00
Spider	2.25 (1.28)d	1.00 – 5.00
Shyness
Spider	3.46 (1.31)a	1.00 – 5.00
Stranger Approach	3.05 (1.24)b	1.00 – 5.00
Robot	2.98 (1.24)b	1.00 – 5.00
Clown	2.88 (1.33)b	1.00 – 5.00
Puppet Show	1.98 (1.04)c	1.00 – 5.00
Proximity to Mother
Spider	1.60 (0.45)a	0.17 – 2.00
Robot	1.43 (0.58)b	0.00 – 2.00
Puppet Show	0.78 (0.47)c	0.00 – 2.00
Clown	0.71 (0.64)c,d	0.00 – 2.00
Stranger Approach	0.60 (0.58)d	0.00 – 2.00
Attempt to Be Held
Spider	1.50 (0.58)a	0.08 – 3.00
Robot	1.36 (0.69)b	0.00 – 2.82
Clown	0.43 (0.66)c	0.00 – 2.50
Stranger Approach	0.24 (0.50)d	0.00 – 2.57
Puppet Show	0.24 (0.40)d	0.00 – 2.00
Distress Vocalizations
Spider	0.20 (0.38)a	0.00 – 1.83
Robot	0.13 (0.35)b	0.00 – 2.00
Clown	0.05 (0.25)c	0.00 – 2.50
Stranger Approach	0.03 (0.14)c	0.00 – 1.00
Puppet Show	0.01 (0.03)c	0.00 – 0.28
Stranger Approach Composite	−0.01 (0.73)	−0.90 – 3.23
Robot Composite	0.01 (0.82)	−1.61 – 2.17
Puppet Show Composite	−0.01 (0.72)	−0.98 – 3.17
Clown Composite	−0.01 (0.80)	−0.98 – 3.74
Spider Composite	−0.00 (0.82)	−1.87 – 1.71
Maternal report of fear (TBAQ)	3.03 (0.85)	1.31 – 5.15

Note. TBAQ = Toddler Behavior Assessment Questionnaire. Composites were averages of Z-scores. Scores of raw fear behaviors were compared across episodes within behavior type in a repeated measures ANOVA; superscripts of like-kind reflect non-significant differences.

Prior to creating fear behavior composites, we determined correlations among all raw fear behaviors across episodes (Table 2). Next, we computed bivariate relations between maternal report of fear and the fear composites from each of the five episodes. Maternal report related to the fear composite for Stranger Approach (r = .35, p < .001), Puppet Show (r = .43, p < .001), and Clown (r = .35, p < .001). These effect sizes are in the small to moderate range according to Cohen’s (1988) effect size conventions. The correlation was not significant for Robot (r = .10, p = .283) or Spider (r = −.01, p = .900). We also computed a correlation between a composite of behaviors averaged across all five episodes (a mean of the z-scores of these averages yielded the overall composite) and maternal report of fear. This correlation was in the moderate range (r = .33, p < .001).

Table 2.

Intercorrelations among Fear Behavior Variables

Variable	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25
1. PS boldness	.71**	.36**	.44**	.29**	.50**	.53**	.49**	.40**	.13	.30**	.25**	.25**	.22*	.16	.39**	.25**	.37**	.33**	.20*	.33**	.28**	.32**	.29**	.22*
2. PS shyness	–	.39**	.51**	.39**	.49**	.53**	.43**	.35**	.07	.24*	.18	.29**	.21*	.18	.37**	.22*	.42**	.36**	.22*	.29**	.25**	.36**	.40**	.21*
3. PS proximity to mom		–	.68**	.36**	.28**	.37**	.56**	.50**	.09	.16	.17	.22*	.19*	.13	.31**	.29**	.23*	.27**	.12	.14	.18	.18	.31**	.20*
4. PS attempt to be held			–	.34**	.28**	.42**	.58**	.57**	.18	.13	.16	.30**	.20*	.08	.44**	.35**	.41**	.42**	.19	.30**	.21*	.32**	.38**	.22
5. PS distress vocs				–	.10	.18	.27**	.28**	−.01	.05	.04	.11	.14	.13	.15	.16	.18	.23*	.15	.03	.06	.08	.21*	.06
6. C boldness					–	.80**	.65**	.58**	.28**	.49**	.25**	.36**	.19*	.13	.23*	.11	.24*	.21*	.17	.17	.03	.20*	.22*	.17
7. C shyness						–	.72**	.60**	.23*	.37**	.35**	.32**	.18	.14	.27**	.28**	.25**	.24*	.29**	.23*	.21*	.28**	.29**	.18
8. C proximity to mom							–	.88**	.28**	.37**	.22*	.38**	.29**	.18	.40**	.28**	.36**	.37**	.23*	.24**	.10	.31**	.38**	.23*
9. C attempt to be held								–	.37**	.30**	.17	.36**	.30**	.12	.28**	.18	.27**	.29**	.12	.20*	.05	.23*	.28**	.18
10. C distress vocs									–	.02	−.10	.09	.08	.13	.15	.06	.13	.19*	.23*	.06	−.00	.11	.06	.10
11. SA boldness										–	.54**	.49**	.32**	.26**	.06	−.01	.10	.06	.12	.01	−.05	.05	.01	.11
12. SA shyness											–	.49**	.39**	.24*	.11	.26**	.18	.14	.08	−.06	.14	.11	.05	.03
13. SA proximity to mom												–	.70**	.33**	.29**	.16	.30**	.26**	.11	.19*	.15	.25**	.27**	.18
14. SA attempt to be held													–	.66**	.23*	.15	.27**	.24*	.24*	.19*	.14	.21*	.21*	.13
15. SA distress vocs														–	.12	.11	.13	.14	.30**	.06	−.04	.12	.06	.03
16. R boldness															–	.65**	.85**	.78**	.32**	.61**	.32**	.57**	.60**	.39**
17. R shyness																–	.57**	.58**	.41**	.37**	.51**	.44**	.44**	.44**
18. R proximity to mom																	–	.90**	.32**	.56**	.28**	.59**	.62**	.34**
19. R attempt to be held																		–	.40**	.54**	.28**	.54**	.63**	.39**
20. R distress vocs																			–	.25**	.30**	.22**	.25**	.69**
21. S boldness																				–	.55**	.80**	.77**	.36**
22. S shyness																					–	.54**	.50**	.47**
23. S proximity to mom																						–	.89**	.36**
24. S attempt to be held																							–	.41**
25. S distress vocs																								–

Note. PS = Puppet Show. C = Clown, SA = Stranger Approach. R = Robot. S = Spider.

^*p < .05,

^**p < .01.

Next, we performed a RM-MANOVA to validate the low-threat versus high-threat designation of episodes. See Figure 1 for a graphical representation of scores across episodes. Mauchly’s test of sphericity was significant for all behaviors (χ²s[9] > 30.54, all ps < .001). Univariate tests of episode were significant (ps < .001, partial eta squared ranged from .11 [distress vocalizations] to .64 [attempt to be held]) for all behaviors regardless of correction type (Greenhouse-Geisser, Huynh-Feldt, or lower-bound). Nearly all pairwise comparisons among episodes were significant (partial eta squared ranged from .00 to .79), with exceptions noted in Table 1. Overall, Clown and Puppet Show seemed to represent the lowest threat (i.e., lowest shyness, attempt to be held, proximity to mother, and distress vocalizations; highest boldness), Spider and Robot represented the highest threat (highest shyness, attempt to be held, proximity to mother, and distress vocalizations; lowest boldness), and Stranger Approach was unclear, sometimes showing similar values to Clown and Puppet Show, and, in one case, having a similar value to Robot. Therefore, when testing threat-level as a moderator of concordance, Stranger Approach was excluded from analyses in order to make clear comparisons. Of note, these results remained nearly identical if Stranger Approach was included as a low-threat episode.

Figure 1.

Fear behaviors across episodes.

Concordance Analyses

Multilevel modeling was used to test concordance, but first we ran an empty model with no predictors and observed toddler fear composites as the dependent variable. This provided information for the intraclass correlation coefficient (ICC; proportion of between-subjects variance to total variance in fear behavior composites), which was found to be 0.3747, meaning that 37.47% of the variance in toddler fear behavior was between participants. This value suggests that fear behaviors across episodes tended to be more similar within toddlers than between two random toddlers, warranting this multilevel approach.

An initial model containing covariates and maternal report of fear yielded a significant improvement over the model with no predictors (Table 3). Maternal report of fear related to laboratory fear composites above and beyond the demographic variables.

Table 3.

Multilevel Model of the Main Effect Relation between Maternal Report and Observation of Fear

Fixed Effects	γ	SE	t	p	95% CI
Intercept	−0.14	0.06	−2.24	.027	−0.26, −0.02
Gender	0.30	0.09	3.19	.002	0.11, 0.47
SES	0.01	0.00	2.49	.014	0.002, 0.02
Maternal report of fear	0.21	0.06	3.78	.000	0.10, 0.32
Variance Parameters	Estimate	SE	Wald Z	p	95% CI
Residual (Level 1)	0.37	0.02	14.90	<.001	0.32, 0.42
Intercept (Level 2)	0.17	0.03	5.10	<.001	0.11, 0.24

Note. A deviance change test suggested significant improvement of this model over a model with no predictors (χ²[3] = 94.46, p < .001).

The model assessing the role of episode threat-level represented a significant improvement from the model with no predictors (Table 4). The interaction between maternal report of fear and threat-level was significant. Probing of this interaction (Figure 2) revealed that maternal report of fear did not relate to laboratory fear composites for the high-threat episodes, γ = 0.04, SE = 0.07, t(171.208) = 0.52, p = .602, 95% CI [−0.10, 0.17], but the two assessment methods were related for the low-threat episodes, γ = 0.34, SE = 0.07, t(171.208) = 4.90, p < .001, 95% CI [0.20, 0.48].

Table 4.

Multilevel Models of Moderators of the Relation between Maternal Report and Observation of Fear

Fixed Effects	Moderator = Threat Level χ2[5] =347.67,p < .001					Moderator = Maternal Involvement χ2[5] = 94.71, p < .001
	γ	SE	t	p	95% CI	γ	SE	t	p	95% CI
Intercept	−0.13	0.08	−1.63	.106	−0.27, 0.03	−0.14	0.07	−1.95	.053	−0.27, 0.001
Gender	0.28	0.11	2.61	.010	0.07, 0.49	0.30	0.09	3.19	.002	0.11, 0.49
SES	0.01	0.00	2.34	.021	0.002, 0.02	0.01	0.00	2.76	.014	0.002, 0.02
Maternal report of fear	0.04	0.07	0.52	.602	−0.10, 0.17	0.19	0.07	2.86	.005	0.06, 0.32
Moderator	−0.01	0.05	−0.22	.825	−0.12, 0.09	−0.01	0.05	−0.11	.909	−0.11, 0.10
Maternal report of fear X Moderator	0.31	0.06	4.85	<.001	0.18, 0.43	0.03	0.06	0.48	.629	−0.09, 0.15
Variance Parameters	Estimate	SE	Wald Z	p	95% CI	Estimate	SE	Wald Z	p	95% CI
Residual (Level 1)	0.32	0.02	12.90	<.001	0.27, 0.37	0.37	0.02	14.90	<.001	0.32, 0.42
Intercept (Level 2)	0.23	0.04	5.47	<.001	0.16, 0.33	0.17	0.03	5.10	<.001	0.11, 0.24

Note. Chi-square model tests reflect deviance change tests between the model and a model with no predictors. Dummy codes represented the moderators of threat level (low-threat = 1, high-threat = 0) and maternal involvement (involved = 1, uninvolved = 0), so the simple slope of maternal report of fear can be interpreted for the values of these variables coded 0.

Figure 2.

Interaction between maternal report of toddler fear and threat-level of episode in relation to observed toddler fear. Maternal report of fear related to observed fear for low-threat, but not high-threat episodes. Observed toddler fear represented composites comprised of Z-scores.

***p < .001.

The model assessing the role of maternal involvement was also significant in relation to the empty model (Table 4). However, the interaction between maternal report of fear and maternal involvement was not significant, so this model was not analyzed further.

Discussion

Discrepancies between laboratory observation and parent-reported emotion expressions have both been of interest and caused concern for emotion and temperament researchers. Although maternal characteristics have been studied in relation to concordance, scant attention has been paid to influential aspects of laboratory observation of toddler fear. The current study examined the effect of laboratory episode characteristics on concordance between maternal report and laboratory observation of toddler fear. Specifically, we tested the threat-level of the episode (low versus high) as well as maternal involvement as influencing this relation.

Bivariate relations revealed that the association between maternal report and laboratory fear behavior composites was non-existent for some episodes and small to moderate in effect size for others. This is consistent with previous studies comparing these methods (Durbin & Wilson, 2012; Olino et al., 2013; Seifer et al., 1994; Stifter et al., 2008). It has been suggested that observations are more specific than maternal perceptions and are likely influenced by the eliciting contexts (Seifer et al., 1994). The variation in correlation size across the episodes is consistent with this idea.

Indeed, we found that episode threat-level interacted with maternal report of fear in relation to laboratory observation. Maternal report related to the observed fear composites for low-threat, but not high-threat episodes. Temperament researchers have suggested that concordance may be best achieved when mothers and observers use similar situations to rate child behavior (Rothbart & Bates, 1998). Moreover, theoretical work on informant discrepancies suggests that reporters use contextual information to rate behaviors, and the extent of convergence depends on the extent to which the different reporters or methods rely on similar contexts (De Los Reyes, Thomas, et al., 2013). With this in mind, our results suggest that episodes that tend to elicit lower levels of fear may be more concordant with parent report measures of temperamental fear, which are typically based on daily situations involving being in a new situation, reacting to loud noises, or meeting new people. Granted, the contexts were not identical, as the maternal report measure assessed situations such as going to the doctor’s office, interacting with parents’ friends, going to a new child care setting, and being in a storm. Despite the differences in precise situations, perhaps the low-threat episodes were more similar in the level of fear elicited. Further item analysis may be able to provide more information about the “severity” of laboratory episodes and individual questionnaire items. It may seem counterintuitive that the episodes that most reliably elicit fear expressions (i.e., the high-threat episodes) do not relate as well to toddlers’ broader pattern of expressions, as perceived by mothers. Perhaps these high-threat episodes reflect more normative expressions across development, whereas the low-threat episodes highlight individual differences.

Fear and shyness in less threatening contexts, especially when exhibited by the behaviors we observed in the laboratory (e.g., crying, trying to be held, maintaining proximity) may be especially salient to mothers if they have the opportunity to compare their children to others who do not exhibit fear in these contexts. Given that saliency of behavior has been found to affect concordance in other ways (i.e., between parents and children; Karver, 2006), this may similarly be affecting the results we found. Further, within the framework of the Operations Triad Model, divergence would be expected if high-threat laboratory episodes represent different contexts than those on which mothers are reporting (De Los Reyes, Thomas et al., 2013). Of course, all of the laboratory episodes used in the current study likely diverge from the contexts on which mothers reported toddler fear to some extent, as mothers were not provided the exact same contexts on which to report, and mothers reported on dispositional behavior occurring in the past. Previous research has used an alternative strategy for assessing concordance: asking mothers to rate child behavior from an immediately preceding task (Durbin & Wilson, 2012). In this work, it was shown that discrepancies between maternal ratings and other methods of assessing their children’s fear in laboratory episodes could not be explained by memory biases, but rather may have been due to maternal characteristics (e.g., symptomatology, personality traits, expectations for girls versus boys). In the current study, completing a measure of temperament not only involves assessments of toddler fear and maternal characteristics, but may also place higher demands on memory than rating behavior that just occurred. It is possible that these various influences could affect concordance differently for high- versus low-threat contexts. Therefore, assessing concordance in a manner that eliminates these memory demands could shed further light onto aspects of laboratory observation that affects concordance with maternal perceptions.

Our results may have implications for the study of dysregulated fear, defined by the tendency to express high levels of fear in low-threat contexts (Buss, 2011). The inability to match one’s expression and experience of emotion to the eliciting context in order to interact with environment indicates a problem with emotion regulation (Cole et al., 2004). Difficulties with emotion regulation have been proposed to underlie many aspects of behavior that deviate from typical development (e.g., Cole, Michel, & O’Donnell Teti, 1994). Specific to the emotion of fear, dysregulated fear at 2 years of age, more so than traditional laboratory measures of temperament that may be higher-threat, has been found to predict mother-reported anxiety and social withdrawal across preschool and in the kindergarten year, as well as teacher-reported anxiety in kindergarten (Buss, 2011; Kiel & Buss, 2014). Dysregulated fear in toddlerhood also predicted observed social reticence and social anxiety disorder symptoms in kindergarten (Buss et al., 2013). Thus, when studying how temperamental fear relates to maladaptive outcomes, fear in low-threat contexts has emerged as uniquely salient. Given that mother-reported fear related most strongly to fear in these low-threat contexts, maternal report may be particularly complementary to studying dysregulated fear and subsequent risk processes. In this way, mothers may be identifying difficulties in emotion regulation that put their children at risk. Indeed, previous work has shown that developmental trajectories of maternal report of temperamental fear, but not of fear behaviors aggregated over contexts ranging in threat (not just low-threat), predicted Social Anxiety Disorder diagnosis (Chronis-Tuscano et al., 2009). To best understand how our results contribute to determining the most well-suited measures for studying risk, it would be important to test how maternal report, fear in low-threat laboratory episodes, and fear in high-threat episodes each predicts internalizing problems. Only if maternal report and low threat fear predicted problems with greater strength than high-threat fear would our concordance findings have clear implications for these etiological questions.

Maternal involvement in the laboratory episodes, on the other hand, did not moderate the relation between maternal report and laboratory observation of fear. Given a significant main effect of the relation between maternal report and laboratory assessment, this suggests that fear derived from the two methods were moderately related, regardless of mothers’ involvement. Although previous research has found that children use different regulatory strategies when mothers are involved in the laboratory task (Grolnick et al., 1996), maternal involvement may not be changing toddlers’ behaviors to the extent that they would be more representative of the behaviors mothers observe and report. This is consistent with at least one study investigating concordance for infant temperament, showing that maternal involvement did not affect concordance (Leerkes & Crockenberg, 2003). However, we tested a specific sub-grouping of mother-involved and mother-uninvolved episodes. It is possible that manipulating involvement differently over these, or other, episodes, would yield different results. Certainly, understanding how maternal involvement causes toddlers to act in specific ways related or unrelated to maternal report would require experimental manipulation that was not present in the current study.

One limitation of the current study, and therefore a fruitful area for future studies, is that we did not investigate how characteristics of the episodes and of the mothers may interact to affect concordance. Maternal personality traits and symptoms of psychopathology have previously been found to relate to maternal report and impact concordance with laboratory observation (Bates & Bayles, 1984; Durbin & Wilson, 2012; Leerkes & Crockenberg, 2003). Although the current study provided foundational evidence for the importance of considering contexts of fear expressions in the laboratory when assessing concordance, a next step in this domain would be investigating whether maternal and laboratory episode characteristics make additive and/or interactive contributions to concordance. It could be that episode characteristics make a larger difference in concordance for mothers who show bias in their response styles. Alternatively, this bias may overshadow the effect of laboratory contexts on concordance, making it more salient for mothers with lower levels of certain traits or symptoms of psychopathology.

Another area for future research suggested by our results is to investigate maternal behaviors in low- versus high-threat contexts to determine if these differences in concordance may be related to differences in how mothers interact with their children. Some existing work suggests that mothers display more protective behavior in high-threat than low-threat contexts, but that only protective behavior in low-threat contexts is a relevant mechanism of development from toddlers’ fearful temperament to future anxiety behaviors (Kiel & Buss, 2012). Expanding this area to include investigations of other maternal behaviors (e.g., sensitivity, responsiveness) and real-time responses to displays of toddler fear would further clarify the relation between context-specific concordance and maternal behavior.

To determine the boundaries on the context effects of concordance between laboratory episodes and maternal report, it will be necessary to expand this line of research in several ways. First, it is unclear whether similar effects would emerge in age periods later than toddlerhood. Given that toddlers are more highly reliant on their caregivers than older children and adolescents, it will be important to understand whether these results are replicated with these older age groups. We surmise that effects would be dampened as children begin spending less time with their parents and internalize display rules that influence the minimization of external expressions of negative emotion. Further, these results can speak only to fear, and not to other negative emotions such as sadness or anger, or to positive emotions. Various standardized paradigms for assessing these other emotions exist, so context effects on concordance could be examined. Our findings imply that it is important to understand how different laboratory episodes assessing a particular emotion vary in the typical intensity elicited and then test for differential relations with questionnaire-based parent-report measures. Different levels of concordance may be achieved if some episodes differ more drastically in the intensity of emotion elicited from typical day-to-day activities described in questionnaires. It has been found that maternal characteristics are related to perceptions of child anger, sadness, and positive emotions, (Durbin & Wilson, 2012), but features of the episodes themselves have not been investigated.

An additional limitation of the current study is that we did not consider the role that the social versus non-social nature of the laboratory episodes may have impacted the influence of threat-level on concordance. Although examining the social fear and object fear scales of the TBAQ separately produced the same results, the laboratory episodes were not investigated along these lines. There is evidence of differential associations with maternal report based on the social versus non-social nature of the laboratory episode (Dyson et al., 2011). In our study, it is possible that the social versus non-social nature of the episodes mapped onto the division of episodes by threat-level, with the clown, puppet show, and stranger approach episodes eliciting more social fear, and the spider and robot episodes eliciting more object fear. However, many of our episodes had both social and non-social components. Puppet show, for example, may elicit some object fear, as the puppets may be more object-like. At the end of the spider and robot episodes, the experimenter prompted the child to touch the respective stimulus, adding a social component. More precisely determining the contributions of the threat-level and social/non-social nature of the episodes to concordance will be an important area for future work.

Other limitations of the current study also warrant attention and suggest areas for future research. We focused specifically on maternal report of fear and did not investigate whether similar findings occur for paternal report or reports by other adults (e.g., teachers or other caregivers). In addition, although our sample comprised mothers from a variety of backgrounds, it was relatively homogenous, comprising, for the most part, European American and middle class mothers. Understanding whether laboratory episode characteristics impact concordance similarly for other important adults in children’s lives and for informants from other backgrounds will be an important aspect of replication. We also focused on one particular parent-report measure of temperament for toddlers at one age. Expanding this type of investigation to other measures of temperament and emotion, as well as to other age groups, would indicate the robustness and developmental boundaries of our results. The current study could not fully delineate the psychometric properties of toddler fear across the different contexts. We provided some evidence of reliability, such that coders demonstrated inter-rater reliability and composites had adequate internal consistency. We relied on validity provided by previous studies for various context-based designations of fear (Buss, 2011; Buss et al., 2013) but did not externally validate them beyond analyzing the relation between these observations and maternal report. Future work could expand upon these findings by using techniques such as structural equation modeling and item response theory to validate measurement models and other psychometric properties of toddler fear behaviors. Further, in addition to similarities and differences existing across the laboratory contexts, there were also similarities and differences in types of fear and shyness assessed by the TBAQ (e.g., fear of loud noises, high places, animals; shyness around new adults, new children) that we did not quantify, nor did we compare these to the extent of similarity/difference across the laboratory contexts. Finally, given the sample size of the current study and the difficulty of finding interaction effects in non-experimental research (McClelland & Judd, 1993), it will be important to replicate our findings with independent samples.

In conclusion, the threat-level of, but not necessarily maternal involvement in laboratory episodes designed to elicit fear are important to take into account when assessing their relation to maternal report of fear and fearful temperament. Our results showed that only fear in low-threat laboratory contexts related to maternal report of fear, perhaps because these situations are more similar to situations in which mothers observe their toddlers engage on a day-to-day basis. Threat-level may be an important contextual consideration in interpreting previous reports of concordance, in understanding the utility of maternal report, and in designing laboratory studies intending to maximize concordance with mothers’ perceptions of their children across a wide variety of situations.