Applying Machine Learning to Infant Interaction: The Development is in the Details

Abstract

The face-to-face interactions of infants and their parents are a model system in which critical communicative abilities emerge. We apply machine learning methods to explore the predictability of infant and mother behavior during interaction with an eye to understanding the preconditions of infant intentionality. Overall, developmental changes were most evident when the probability of specific behaviors was examined in specific interactive contexts. Mother’s smiled predictably in response to infant smiles, for example, and infant smile initiations become more predictable over developmental time. Analysis of face-to-face interaction—a tractable model system—promise to pave the way for the construction of virtual and physical agents who are able to interact and develop.

Infant-parent face-to-face interaction is a prototype for social communication throughout the lifecycle. During early interaction, infants gaze at their parent’s face and vocalize and smile. Parents vocalize, smile, hold, and tickle their infants. These expressions influence the parent and parental expressions come to influence infants. During interaction, infants and parents seem to influence and respond to one another as they engage in nonverbal emotional communication. These face-to-face interactions are a tractable model system for understanding communicative development.

A fundamental feature of human development is the emergence of intentional communication. Intentional communication can be defined as engaging in a behavior with the goal of eliciting a response in the partner. Evidence for infant intentional communication arises in the period between 8 and 12 months as infants adopt conventional gestural movements to indicate desired objects and events of interest (Bates, Benigni, Bretherton, Camaioni, & Volterra, 1979; Messinger & Fogel, 1998). A potentially necessary condition of the development of such goal-directed action is repeated experience in which a given action elicits a given response. Here we investigate interactions which might the basis of such repeated experiences for infants between one and six months of age.

The Predictability of Specific Infant and Parent Behaviors

Predictability can be investigated by measuring specific infant and parent behaviors such as facial expressions (Elias & Broerse, 1995; Jaffe, Beebe, Feldstein, Crown, & Jasnow, 2001), gazes and vocalizations (Crown, Feldstein, Jasnow, Beebe, & Jaffe, 2002). One can then examine the likelihood of one partner’s behavior (e.g., a smile or vocalization) predicting the onset of the partner’s behavior (Fogel, 1988; Kaye & Fogel, 1980; Malatesta, Culver, Tesman, & Shepard, 1989; Symons & Moran, 1994; Van Egeren, Barratt, & Roach, 2001).

Rules

Infant and parent smiling patterns may be articulated as a set of dyadic rules’ (Cohn & Tronick, 1987; Kaye & Fogel, 1980; Symons & Moran, 1994). Parents must initiate smiles at their infants. Infants may smile in response to a parent smile but are not obliged to do so. Infants are free to initiate smiling at their parents; this initiation should increase with age such that infant smiling initiations are common between six and nine months of age. Parents must smile in response to an infant smile. Once the infant is smiling, the parent must continue smiling until the infant has stopped smiling. The infant, however, is free to cease smiling in the face of the parent’s smile.

Predictability of specific behaviors

The dyadic rules summarizing interaction patterns indicate that infant smiles tend to elicit parent smiles. In that sense, parent responses to infant smiling appear to be highly predictable. On the other hand, although parent smiles typically precede infant smiles, many parent smiles do not elicit infant smiles (Cohn & Tronick, 1987; Kaye & Fogel, 1980; Symons & Moran, 1994). Thus the predictability of the infant is somewhat unclear. Moreover, relatively little is known about the effects of either infant or parent smile termination on the expressivity of a partner.

The Predictability of Changes in Emotional Engagement

Another research approach that yields information on predictability involves measuring infant and parent behavior with ordinal scales of affective engagement states (Beebe & Gerstman, 1984; Cohn & Tronick, 1987; Weinberg, Tronick, Cohn, & Olson, 1999). These engagement states aggregate expressive behaviors thought to reflect a continuum from negative to neutral to positive (Cohn & Tronick, 1987; Weinberg, et al., 1999). Ordinal scaling approaches are amenable to time-series analyses, which have revealed strong auto-correlation effects. That is, the strongest predictor of infant behavior at a given period in time is the infant’s behavior in the immediately antecedent period. The same auto-correlation effect is seen in mother’s behavior (Beebe, et al., 2007; Cohn & Tronick, 1987).

Predictability predictions

A possible basis of emerging infant intentionality, then, is the stability of dyadic interactions. Two patterns of maternal action might be expected to provide a basis for the development of infant intentionality. Both high overall levels of stability in parental responses to infant actions and levels of stability that increased with infant age might foster the infant’s developing expectations of mother actions. More generally, we might expect infant patterns of responsivity to increasingly resemble those of the mother. That is, interactions might develop such that the infant adopted increasingly mature patterns of initiation and responsivity.

Outstanding questions

Like contingency analyses of discrete behaviors, time-series analyses typically indicate strong infant-to-parent interactive influence. They also reveal a developmental increase in parent-to-infant influence. Between 3 and 9 months, infants become increasingly responsive to their interactive partners (Cohn & Tronick, 1987; Feldman, Greenbaum, & Yirmiya, 1999). During this same time period, however, infants become more likely to initiate positive affect expressions. Are infants becoming more predictable in that they are more responsive, or are they becoming less predictable in that they initiate expressive action without a clear antecedent? To address these questions, we turn to models of interactive behavior inspired by machine-learning.

Machine Learning

Computer models of human behavior have a rich history in the developmental sciences (Bullinaria, 1997; MacWhinney, 1998; Triesch, Teuscher, Deák, & Carlson, 2006). Machine-learning algorithms in particular are being increasingly used to model how human beings interact with and predict their world (Butko, 2008; Maye, Werker, & Gerken, 2002; Schulz, 2004). Using machine learning methods for modeling dyadic behavior has a dual purpose. It can be used to uncover trends in human behavior, but it can also be used as a means for programming artificial agents with interactive and, ultimately, developmental potential.

We use machine learning approaches to model the content and temporal structure of early dyadic interactions. These approaches are supplemented with standard statistical analyses. We address the timing of communicative actions in early interaction from the perspective of the mother and the perspective of the infant. We are especially concerned with the role of temporal expectations, that is, when can I expect my partner to act?

The use of machine learning methods for studying dyadic behavior refocuses our attention around several key questions. How predictable are the mother’s actions from the perspective of the infant? How predictable are the infant’s actions from the perspective of the mother? What are the developmental trends in each partner’s predictability? More concretely, we address a set of questions in order from the more general to the more specific.

1. Does a given infant-mother dyad’s face-to-face interactions become more similar to each other—and so more predictable to each partner—over developmental time?

2. What factors influence the predictability of infant and mother actions within interactions and how does this change with development?

3. What factors influence the predictability of specific infant and mother actions with specific contexts and how does this change with development?

Procedures

Longitudinal data collection

We examine questions of predictability with a rich longitudinal dataset collected to explore developmental process. Thirteen mothers were invited to play with their infants every week between one and six months of age. These play interactions, lasting approximately five minutes each, occurred in a laboratory playroom with the parent holding the infant on their laps. Infants had no apparent risk factors. Parents occasionally skipped weeks resulting in a base dataset of 208 interaction sessions. A subset of analyses required a higher number of transitions per session (e.g., those modeling entropy). Those analyses use 189 of the sessions (a mean of 14.5 sessions per dyad with a range from 9 – 19).

Behaviors

Three behavioral channels—infant smiling, mother smiling, and the direction of infant gazing—were manually coded on a frame-by-frame basis. The direction of infant gazing was coded as at mother’s face and away from mother’s face. The presence of infant and mother smiles was ascertained used the criteria for lip corner puller (AU12) from the Facial Action Coding System (FACS) (Ekman, Friesen, & Hager, 2002; Oster, 2006). These coders were certified in FACS and trained in BabyFACS. All coding was reliable (Messinger, Fogel, & Dickson, 2001).

Dyadic states

The coded infant and mother actions create a matrix of dynamic co-occurrences. A mother and an infant may both be smiling, for example. We refer to this configuration of actions at a give time as a dyadic state. In any given dyadic state, either partner can act to create a new dyadic state by, for example, initiating or terminating a smile. Our focus is the infant’s development of social capacities within this interactive matrix. We regard infant and mother as inextricably bound together in interaction forming a dyadic system, which is the focus of analysis. In some analyses, the system is defined to include infant and mother smile initiations and terminations (smile transitions) as well as shifts in infant gaze direction (gaze transitions). In other analyses, we compare the system defined by infant and mother smile transitions with that defined by infant gaze transitions and mother smile transitions. Within the context of these dyadic systems, we address questions of the predictability of each partner and influence between partners.

Does Dyadic Interaction Become More Predictable With Development?

Increasing specificity

This article follows a path of increasingly specific inquiry into interaction and development. We begin by asking about the similarity of dyadic patterns between interactive sessions over developmental time. As an infant and mother play over successive weekly sessions, do stable patterns of interaction develop? This is essentially the development of dyadic stability, which we regard as a likely component of their predictability. Predictability involves the information one partner has at a given point in time about the other partner’s actions. We follow up our treatment of dyad stability by asking about the overall predictability of infant and mother within a dyadic interaction. This is followed up with analyses of the predictability of specific infant and mother actions given specific actions of the partners.

The development of dyadic stability

A key first step for successful social interaction is establishing predictable protocols that govern social exchanges (Bruner, 1972). The dyads we are investigating provide a bird’s eye view of the emergence of these protocols. They engaged in weekly play interactions during a period in which basic social skills are thought to emerge. Do dyads co-construct repeated patterns that allow infants to expect a certain pattern of social response in the service of eventually acting intentionally to obtain that social response? To answer that question, we asked whether individual sessions of dyadic interaction become more similar to each other over developmental time.

Alternate Models of the Development of Dyadic Stability

We hypothesized that dyadic interaction dynamics would become more stable with development. To test this hypothesis, we considered five models of dyadic interaction dynamics computed over interaction sessions. There are two what models which analyze the sequence of dyadic states without regard to transition between these states. There are three when models which analyze the timing of dyadic states without regard to the specific sequence of states involved. After modeling dyadic patterns of increasing similarity in interaction with age, we examine patterns individual characteristic of the infant and mother.

What Models

A general what model of dyadic state transitions

In this model we describe the interaction dynamics of a particular session by fitting a probabilistic model of dyadic state transitions. We used a maximum likelihood estimation to estimate the probability distribution of transitions between each state p(i_a, m_a, i_a−1, m_a−1) for each session where i indicates an infant action, m a mother action, a a current action state, and a−1 the previous action state. Similarity was defined using the Bhattacharyya coefficient as f:

$$f(p_1,p_2)={\int }_s\sqrt{p_1(s)}\times \sqrt{p_2(s)}ds$$ (1)

Where p₁ is a distribution over state transitions for session 1 and p₂ is a distribution over state transitions for session 2. The integral is computed over each of the 64 possible dyadic state transitions.

A what model of turn-taking

Turn-taking was defined as a mother or infant transition that was immediately preceded by the transition of the other partner. The initiation or termination of an infant smile immediately followed by the initiation or termination of a mother smile, for example, would constitute an instance of turn-taking. If the mother initiated a smile and then terminated it before the infant acted, mother’s smile termination would not be considered turn-taking. The general equation follows:

$$\text{p}(({\text{i}}_{\text{a}}\sim ={\text{i}}_{\text{a}-1}\&{\text{m}}_{\text{a}-1}\sim ={\text{m}}_{\text{a}-2})\mid ({\text{m}}_{\text{a}}\sim ={\text{m}}_{\text{a}-1}\&{\text{i}}_{\text{a}-1}\sim ={\text{i}}_{\text{a}-2}))$$ (2)

Similarity in level of turn-taking was defined as the absolute difference in the proportion of transitions that involved turn-taking between consecutive sessions. This model does not make use of the timing of states, which is the focus of the “when” models we present next.

When Models

There are three when models which analyze the time to transition between dyadic states without regard to the states involved. The timing of states is equivalent to the timing of transitions between the states. These models consider, consecutively, the entire distribution of state transition times, the mean and variance of the transition times; and the mean of the transition times.

A when model of transition time distributions

The session models are constructed by fitting the distribution of dyadic state transition times using a Gaussian kernel density estimator. As with the state transition model, we define the similarity metric using the Bhattacharyya coefficient as:

$$f(p_1,p_2)={\int }_t\sqrt{p_1(t)}\times \sqrt{p_2(t)}dt$$ (3)

Where p₁ is a density over state transition times for session 1 and p₂ is a density over state transition times for session 2. The integral is computed over the domain of p₁ and p₂, which includes all possible state transition times.

A when model of the mean and variance of transition times

We next used a Gaussian distribution defined as a mean and variance to model transition times between dyadic states. The difference score is the negative of the absolute value of the difference between the sums of these variables (normalized to control for scale) in consecutive sessions.

$$f({\mu }_1,{\sigma }_1^2,{\mu }_2,{\sigma }_2^2)=-\frac{1}{\mathit{stddev}(\mu )}\mathit{abs}({\mu }_1-{\mu }_2)-\frac{1}{\mathit{stddev}({\sigma }^2)}\mathit{abs}({\sigma }_1^2-{\sigma }_2^2)$$ (4)

A when model of mean time to transition

This model uses a single parameter, the mean transition time, to characterize interaction sessions. The similarity metric is defined as:

$$f(t_1,t_2)=-\mathit{abs}(t_1-t_2)$$ (5)

Model Results

Our general goal was to determine whether dyadic interaction patterns became more similar with age, an index of developmental stability. In order to investigate the developmental trajectories of the various models, we computed similarity values between consecutive interaction sessions for each dyad. That is, we applied our models to sessions consecutively attended by a given dyad. Correlations between consecutive session similarity and infant age were calculated within dyads. Two-tailed t-tests of these correlations over dyads indicated whether age was a reliable predictor of increasing developmental similarity. All models were implemented on two systems of states: Those defined by a) the co-occurrence of infant and mother smiling and not smiling; and b) the co-occurrence of infant and mother smiling and not smiling, and infant gazing at mother’s face and away from mother’s face.

The what model of overall transitions

Did the overall patterning of transitions between dyadic states become more similar with development? There was no increase in the similarity of consecutive interactions with respect to the overall model of state transitions. This was the case both for the model of infant and mother smiling, mean r = .03, and the model of infant and mother smiling that incorporated infant gaze direction, mean r = −.11, ps > .25.

The what model of turn-taking

There was no increase in the similarity of turn-taking levels in consecutive sessions with age, ps > .05. However, overall levels of turn-taking increased with age in systems involving infant smile transitions. They increased in the system defined by infant and mother smile transitions, mean r = .43, p < .001 and in the system defined by infant and mother smile transitions and infant gaze transitions, mean r = .20, p < .05, but not in the system defined only by mother smile transitions and infant gaze transitions, mean r = .15, p = .10. With increasing age, infants and mothers became more likely to sequence their smiles. Increases in turn-taking also characterized the relationship of mother smile transitions to infant smile transitions and gaze shifts. Overall, mother and infant smile transition turn-taking occurred in somewhat less than half of infant and mother smile transitions (M = .40, SD = .07), about the same proportion of turn-taking that existed when the system was expanded to include infant gaze transition (M = .42, SD = .03).

The when model of the distribution of transition times

The when model incorporating the distribution of transition time exhibited a pattern of increasing similarity between consecutive sessions with increasing age. These developmental associations emerged both for co-occurrences of epochs of infant and mother smiling and not smiling, mean r = .33, and co-occurrences defined by dyadic smiling and the direction of infant gaze, mean r = .39, ps < .05. These models incorporate the entire distribution of transition times. It is conceptually intriguing (and practically useful) to ask whether models incorporating fewer temporal features also exhibit increasing week to week similarity with age.

The when model of mean and variance of transition times

The models of dyadic interaction employing both the mean and variance of the distributions of weekly interactions exhibited increasing similarities between interactive sessions. This was evident when interactions were defined by infant and mother smiling and not smiling, mean r = .27, and when interactions were additionally defined by the direction of infant gaze at and away from mother’s face, mean r = .29, ps < .025. We next examined modeling of the mean of transition times because of their robust performance and simplicity of application and interpretation.

The when model of mean transition times

The when model of mean transition times exhibited significant associations between increasing age and the similarity of consecutive weekly interactive sessions. This was the case both when epochs were defined by infant and mother smiling and not smiling, mean r = .24, and when epochs were also defined by the direction of infant gaze, mean r = .29, ps < .025. These patterns are displayed graphically in Figure 1 in which the similarity in the means of weekly dyadic interaction states is plotted against age. It is of note that while the difference between these mean epochs decreased with age, the mean epochs themselves did not change significantly with age, mean rs = .12 & −.15, ps > 2.

Figure 1.

Figure 1a. Differences in mean epochs formed by the co-occurrences of infant and mother smiles and non-smiles.

Figure 1b. Differences in mean epochs formed by the co-occurrences of infant and mother smiles and non-smiles, and infant gazes at and away from mother’s face.

Note. Absolute differences between successive weekly mean durations of epochs are graphed by age. Each line represents a different dyad. The absolute value of the differences between consecutive weekly sessions decreased with age.

Mean transition for specific infant and mother actions

Are the distributions of specific infant and mother actions becoming more similar over developmental time? If mean dyadic transition times are becoming more similar, are the actions of individual partners also becoming more similar? To address this question, we asked, if there was increasing similarity between consecutive sessions in the distribution of transition times formed only by the distribution of infant smiles, infant non-smiles, mother smiles and non-smiles, and infant gazes at and away from the mother’s face. In addition to considering developmental changes in the similarities between weekly means, we investigated developmental changes in the means themselves.

Developmental changes in the similarity of distributions of specific infant and mother actions (see Table 1)

Table 1.

Developmental emergence of similarities in mean action durations

	Action Type Defining Mean Epochs
	The Negative of Correlations of the Absolute Difference Between Consecutive Weekly Means With Infant Age
	Infant Smiling	Infant Non-Smiling	Mother Smiling	Mother Non-Smiling	Infant Gazing At Parent’s Face	Infant Gazing Away From Parent’s Face
Mean Correlation	−.31	.17	.16	.31	.34	.27
p value	.00	.23	.17	.01	.00	.03
	The Negative of Correlations of the Means Themselves With Infant Age
Mean Correlation	−.41	.24	−.01	.30	.36	.09
p value	.00	.04	.93	.01	.00	.33

Note. The mean correlations were calculated over the 13 infant-mother dyads, and the p-values are based on a two-tailed t-test that the mean correlation is 0. The negative of each correlation is presented so that a positive correlation represents increasing similarity with age.

There were age-related increases in the absolute difference of consecutive weekly means of the duration of infant smiling. That is, differences between consecutive sessions in the mean duration of smiling increased with age. This was reflected in a parallel increase in the overall mean duration of these epochs of infant smiling. There were no corresponding decreases in the similarity of consecutive weekly sessions of infant not smiling epochs. By contrast, there was an age-related decrease in the absolute difference of consecutive weekly means of the duration of epochs of mother not smiling; there was a similar reduction in the overall mean duration of these epochs of mother non-smiling. There were age-related decreases in the absolute difference between the means of infant gazes at and between the means of infant gazes away from mother’s face in consecutive weekly sessions. There were also reductions in the overall mean durations of these epochs of infant gazes at the mother’s face but not in the overall mean duration of infant gazes away from the mother’s face.

Dyadic interaction sessions became more similar with development

Infants and mothers became more likely to alternate their actions—particularly smiling actions—with age. There was also robust evidence that the timing of infant and mother interactions grew more similar over time. Similarities in the shape of temporal distributions within sessions stabilized with age. Associations with age emerged both when interactive sessions were defined as the cross-tabulation of epochs of infant and mother smiling and not smiling, as well as when sessions were defined as the cross-tabulation of epochs of infant and mother smiling and not smiling, and infant gaze direction. This was apparent in dyadic models of the entire distribution of dyadic transition times, the mean and variance of the transitions, and in a parsimonious model of the mean alone. In other words development was characterized by increasingly consistent interaction patterns that governed the social exchanges between mothers and infants.

Increasing similarity in the mean epochs of dyadic states of smiling and not smiling

At an individual level, infants exhibited increasingly dissimilar mean durations of smiling between sessions while mothers exhibited increasingly similar means of epochs of not smiling. There was also increasing week-to-week similarity in dyadic states defined by smiling, not smiling, and infant gaze direction. Individually, there was increasing similarity in epochs of infants’ gazes both at and away from mother; both decreased with age. Overall, then, dyads move toward more fast-paced transitions between states defined by the actions of the partners. This is reflected in increasingly similar levels of infant gaze direction and mother not smiling. This appears to counteract increasingly dissimilar levels and longer durations of infant smiling over development.

What do these patterns mean for infants’ (and mothers’) developing expectations of their partners?

Mothers have reason to expect that their infants’ gaze transitions will become quicker with development, and that their infants’ gazes at their (mothers’) faces will become more consistent between interactive sessions. At the same time, mothers might expect infants to smile for greater periods of time as they get older but that these mean smiling durations will not exhibit greater session-to-session consistency. Infants might expect that epochs of mother not smiling would become briefer and more similar with increasing age. Additionally, infants (and mothers) have reason to expect their partners to increasingly respond to their smiling actions (either an initiation or termination) with a smile transition. These associations, however, exist at the level of an interactive session. We next ask about the predictability of each partner within a given interactive session.

Overall Predictability of Infant and Mother Actions within Interactive Sessions

In this section we shift from analyses of predictability between interactive sessions to modeling predictability within interactive sessions. We focus on when models predicting the time course of each partner’s actions. This allows us to compare different models of infant and mother social action, and ask how they change with infant age. As above, we applied models to two systems of infant and mother actions. This provided an expanded sense of the generalizability of our results. One system was formed by infant and mother smile initiations and terminations. The second system was formed by infant gaze shifts (at and away from mother’s face) together with mother smile initiations and terminations. The systems differ, even when predicting mother smile initiations and terminations, because the prediction of each target partner’s actions is constrained by the actions of the other partner. The figures show distribution curves computed by aggregating across all dyads. To identify developmental trends, infant age was divided into three periods encompassing the age ranges 4 – 10 weeks (1–2.5 months), 11–17 weeks (2.5 – 4 months), and 18 – 24 weeks (4.5 – 6 months).

Modeling the timing of each partner’s actions with respect to dyadic state

Dyadic states are configurations defined, in the first instance, by the actions of each partner, e.g., both infant and mother are smiling. In each dyadic state, the one can model the infant or mother as selecting a time for a next action from a distribution. We model the timing of each partner’s actions by fitting the parameters of gamma distributions independently for each state. If, for example, there are eight possible states, a gamma distribution is fit for each of these eight states independently. We model the timing of the actions of each partner in the dyad sequentially focusing on a given partner, the target partner. It may be, however, that the non-target partner acts before the target partner. The mother, for example, may smile before the infant when the infant is the target. When this occurs, a case provides information about the probability of a target partner’s action only until the temporal point in which the non-target partner acts. This example is akin to an instance of censored information in a survival analysis.

We tested three nested models of the factors that influence the predictability of the timing of infant and mother actions

These models incorporated information concerning the state of the target partner (Self), the state of the other partner (Self plus Partner), and information on which partner acted most recently (Self plus Partner plus History). Several predictions appear reasonable. One might expect, for example, that mother actions would be best predicted by models including the infant’s state and information on which partner acted most frequently, and that the predictability of mother’s actions would remain steady of increase with infant age. One might also expect that, with age, infants’ actions would become more predictable, and would increasingly reflect the influence of mother’s state and information concerning who acted most recently.

The three models:

1. Self model. The state space includes only the current configuration of the partner being predicted, either the mother or the infant. The model for infant smiling actions, for example, would only contain information on whether or not the infant was smiling. The distribution of the next action time is modeled by fitting the parameters of a gamma distribution using standard maximum likelihood techniques.

2. Self plus Partner model. The state space includes the current configuration of the target partner being predicted and the configuration of the other partner. This is a dyadic model. A gamma distribution is fit to predict the action time of the target partner given each of the four possible configurations of the dyad’s state. We compute, for example, the probability distribution of when the infant will smile given that neither infant nor mother is smiling.

3. Self plus Partner plus History. The state space includes the current configuration of the partner being predicted, the configuration of the other partner, and, the historical information as to which partner acted most recently. The addition of information on which partner has acted most recently means that there are eight possible states in this model.

Model fit

Model fit was assessed using a leave-one-event-out cross validation procedure in which each model is evaluated on a previously untrained data point. That is, we iteratively selected one action transition to omit from training, trained a model to predict action times using the remainder of the data, and then evaluates the predictability of the held out transition under the learned model. The timing of individual actions was predicted collapsing over dyad. One-tailed significance values were calculated using paired t-tests over individual events. These test whether there is a statistically significant difference in average predictability for each pair of models.

Factors influencing the timing of infant and mother action over development

Three models were fit for infants and for mothers for two states, each defined by a pair of infant and mother actions (see Figure 2). The three models—Self, Self plus Partner, and Self plus Partner plus History—exhibited similar developmental trajectories in all situations. That is, we did not find clear developmental differences in model fit. The relative predictability afforded by the three models, did not change dramatically with age. Infant actions, in particular, did not become increasingly predicted by mother actions or by historical information with increases in infant age.

Figure 2.

The average predictability of the timing of (a) infant actions and (b) mother’s actions defined with respect to the smiling of infant and mother (top row) and the gaze shifts of the infant and the smiling of the mother (bottom row). The timing of each partner’s actions is modeled using three increasingly complex models: a) Self (target partner current state), b) Self plus Partner (target and non-target current state), c) Self plus Partner plus History (target and non-target current state and who acted most recently). The behavior of each of these models is expressed as a function of infant age.

Differences in the fit of the three models that did not vary with development

In models defined by smile transitions, the Self Plus Partner Plus History models outperformed the Self models for both infant and mother at all ages. That is, the timing of each partner’s actions could be best predicted by knowing the state of the other partner, and which of the two partners had last smiled or stopped smiling. The same held for the timing of mother’s initiations and terminations of smiling in the system created with infant gazing. Infant gazing, however, was better predicted by the Self Plus Partner Plus History model only in the second age interval (from 3–4 months) and not during other intervals. Finally, performance differences between the other two pairs of models—e.g., the self model and the Self Plus Partner model and the Self Plus Partner Plus History models—were not consistent between systems, partners and developmental periods.

Did the predictability of infant and mother actions change with infant age?

All developmental comparisons were made with respect to the Self Plus Partner Plus History model. Overall, the timing of infant initiations and terminations of smiles exhibited high levels of predictability that did not change dramatically with age. By contrast, the timing of shifts in infant gaze direction became substantially more predictable with development. In sum, the predictability of infant gaze transitions rose with age while the predictability of infant smiling did not.

The predictability of the timing of mother initiations and terminations of smiling increased with infant age

This increase in predictability was evident both in states defined with respect to infant smiling and in those defined with respect to infant gaze direction. This increase could provide a potential basis for infants to learn the effects of their actions on the mother. There were no apparent consistent developmental differences between infant and mother predictability. The predictability of infant gaze shifts increased to the level of mother smile transitions with development. However, infant smiling transitions exhibited more consistent predictability than mother smile transitions.

Overall, the timing of infants’ and mothers’ actions did not differ with respect to the influence of the other partner and the immediate history of the interaction

We were surprised by the lack of developmental change in the factors influencing the timing of infant actions. Infants did not, as we expected, show an overall increase in responsivity to mother’s smiling state and to information on which partner had acted most recently. The lack of such associations motivated us to examine the predictability of the timing of each of the infant and mother’s actions in more detail. For example, we separately contrasted the predictability of infant initiations and terminations of smiling.

The Predictability of Specific Actions in Specific Dyadic States

Do specific infant and mother actions differ in their predictability? Does this predictability depend on the partner’s actions and does it change with infant age? To address these questions, we examined the predictability of the timing of infant and mother actions in a fine-grained fashion. These analyses use predictability, the inverse of entropy, as a metric. The analyses are applied to the distributions of time before a given act by a given partner in a given dyadic state. Entropy is a measure of uncertainty in this distribution. Generally, the more kurtotic (peaked) the distribution of these transitions, the more predictable is their timing, and the lower their entropy. The less kurtotic and more spread out the distributions, the less predictable is their timing and the higher their entropy. Figure 3 provides an overall illustration of these distributions.

Figure 3.

The probability density over time of infant and mother (“mom”) actions in various dyadic states in two systems of dyadic interaction. In each system, all eight possible states formed by the actions of each partner and information on which parter acted last (most recently) to bring the dyad into that state are presented. Each configuration defines the possible action of both partners. In the top left hand corner in which neither partner is smiling, for example, the probability densities reflect the probability of each partner smiling.

Negatively skewed distributions

Overall, both infant and mother probability densities were negatively skewed indicating a high probability of action within several seconds of entering a given state (see Figure 3). If either infant or mother exhibited a greater relative likelihood of acting in a given state, this tended to persist for approximately five seconds. That is, if there was a difference in the likelihood of infant or mother acting, this did not appear to change appreciably with time.

Infant versus mother

The probability density functions revealed that infant expressive actions—gazing at mother and smiling—were rare in comparison to mother smile initiations and terminations (see Figure 3). Infants were almost never more likely to gaze at mother than mothers were to initiate or terminate a smile (see top right quadrant of Figure 3). Infants were never more likely to initiate a smile than mother was to initiate or terminate a smile (see top left quadrant of Figure 3). Infants were at least as likely to terminate a smile as mothers were to initiate or terminate a smile (see bottom left quadrant of Figure 3). These analyses underscore the likelihood of mother expressive changes at every juncture of the dyadic interactions.

Variability

Overall, there was considerable variability in whether one partner tended to act before the other, and whether this was the infant or mother. This appeared to vary both with respect to the system under considerations (comparing the left and right halves of the figure), as well as which partner had acted most recently (comparing the left and right columns of each half of the figure), as well as the particular constellation of dyadic states that constrained each partner’s actions (within each column of the figure). To investigate this variability, we quantified the predictability of each partner’s actions in each system of interaction with respect to the partner’s actions, and the partner who had acted most recently. We described each of these distributions as the negative of its entropy. This produced a descriptive mean of the predictability of the distribution and associated confidence intervals (see Figure 4).

Figure 4.

Predictability (reverse signed entropy) of infant and mother actions in multiple contexts. Dyadic states representing infant (on the left) and mother (on the right) smile initiations and terminations are presented in the top set of graphs. Dyadic states representing infant gaze shifting (on the left) and mother smiling (on the right) are presented in the bottom set of graphs. Each panel describes the predictability of a given infant or mother action (e.g., infant smile initiation while mother is not smiling in the hand panel of the top left graph) both when the infant acted most recently (infant last) and when the mother acted most recently (mother last). Predictability is described with respect to infant age categories: 4 – 10 weeks (1–2.5 months), 11–17 weeks (2.5 – 4 months), and 18 – 24 weeks (4.5 – 6 months).

The Infant-Mother Smiling System

Infant smiling

Figure 4 expresses the predictability of the timing of infant and mother’s actions in different circumstances. We begin with infant smiling in the infant and mother smiling system. Infants’ termination of their smiles was more predictable than their initiation of smiles (see top quadrant of Figure 4). These patterns changed with development. With age, infant’s terminations of their smiles became less predictable; their initiation of smiles became more predictable. This suggests that infant smiling became a more stable state developmentally and epochs of not smiling became less stable. Infant smile initiations were more predictable when the infant had acted most recently (by terminating a smiling) than when the mother had acted most recently. From the mother’s perspective, then, the timing of an infant’s smile is more predictable if the mother does not initiate or terminate a smile.

Mother smiling

The predictability of the timing of mother actions was influenced by multiple factors. Mother smile terminations were more predictable when the infant was not smiling than when the infant was smiling. Mother smile initiations were more predictable when the infant was smiling—but only if the infant had acted most recently by smiling—than when the infant was not smiling. To some degree, then, mother’s actions were most predictable when mother was matching the infant’s nonsmiling or smiling state. Mother’s matching of an infant’s smile initiation and mother’s matching of a smile termination exhibited little developmental change. This indicates that mother’s matching (symmetry-creating) actions were highly predictable and relatively stable with development. By contrast, mother actions that broke dyadic symmetry—initiating a smile at a nonsmiling infant or terminating a smile when the infant was smiling—became more predictable with infant age.

The Infant Gaze - Mother Smiling System

Infant gazing

The predictability of infant gazing away from mother’s face increased rather dramatically with age in conditions defined by mother smiling. The predictability of infant gazes at the mother’s face, however, exhibited few clear developmental trends. Overall, the predictability of the timing of gazes at and away from mother’s face was higher when the infant had last switched gaze direction than when the mother had most recently initiated or terminated a smile. That is, mother’s initiation and termination of smiling was associated with decreased predictability of the timing of the infant’s next gaze shift. When mother initiated or terminated a smile she likely created a salient change that reduced the predictability of the timing of the infant’s next gaze shift.

Mother smiling

Generally, mother smiling predictability in states defined by the infant gazing at and away from the mother’s face paralleled that seen in states defined, respectively, by the presense and absence of infant smiling (see Figure 4). The predictability of mother smile initiations, for example, increased consistently with infant age. The predictability of mother smile terminations did not exhibit the same consistent developmental patterning, with one exception. Mother smile terminations exhibited decreasing predictability when the infant had acted last to gaze at her; this paralleled the tendency of mother smile initiations to become increasingly predictable in the same situation. This suggests that infant expressive actions—gazing at the mother’s face—serve to maintain mother expressive actions.

Predicting mother’s actions

For all mother smiling actions in all contexts, mother predictability tended to be lower when it was the infant who had acted most recently by shifting gaze direction. In sum, mother’s smile initiations and terminations were more predictable—and likely more rapid—if the mother had recently initiated or terminated a smile. From the infant’s perspective, this means that mother initiations and terminations of her smiling were more predictable if the infant did not switch gaze direction.

Summary: The Predictability of Specific Actions in Specific Dyadic States (see Figure 4)

The entropy of joy

Infants were generally predictable in terminating smiles and unpredictable in initiating smiles. Developmentally, however, infants became more likely to extend the duration of smiles in an unpredictable fashion, and became less predictable when they terminated smiles. Predictability is the inverse of entropy and smiles are indices of joy. With development, then, smiles became more predicable and stable. The entropy of joy decreased.

Mothers as match-makers

Mothers’ matching actions, which favored the creation of symmetrical states, were relatively predictable, and this changed little with infant age. Mothers were relatively predictable in matching infant actions, particularly the initiations of smiles; they were less predictable in breaking symmetry (transitioning out of states when smiling states were matched).

Predictability in repeated actions

Mother’s smiling actions were typically more predictable if she had acted most recently; an exception was mother’s matching of infant smiles—i.e. infant had acted most recently—which was highly predictable. This means that infants had little influence on the predictability of mother’s actions except in situations in which neither partner was smiling. The predictability of infant gazes away from the mother’s face increased markedly with development, irrespective of wheth mother was or was not smiling. As with mother, infant’s actions were almost uniformly more predictable when it was the infant who had acted last.

Discussion

Face-to-face interactions are a relatively common type of play in middle-class families in the industrialized world (Cohn, Campbell, Matias, & Hopkins, 1990; Fogel, 2001). These interactions involve social exchanges in which early patterns of simultaneous responsivity and turn-taking may develop. We employed modeling approaches derived from machine learning to analyze the development of infant-mother interaction in the first six months of life.

Modeling moved from the general to the specific

Initial models documented increasing similarity in individual sessions of dyadic interaction over developmental time. A second stage of modeling compared overall levels of infant and mother predictability within dyadic interactions over developmental time. Finally, we modeled the predictability of specific infant and mother expressive actions in interactive context to reveal the complexity of developmental change. Below, we discuss these results with respect to the psychological literature on the development of infant-mother interaction. We conclude by suggesting the importance of modeling simulations to understanding early social development.

Developmental Stability: Increasing Similarity Between Dyadic Interactions

An initial set of analyses revealed developmentally emerging regularities in the temporal distibutions and content of dyadic states across interactive sessions. The distributions of dyadic states formed by co-occurring infant and mother actions became more similar with age. Most concretely, the mean of these states became increasingly similar within dyads over infant age. These developmental changes illustrate how dyadic face-to-face interactions become regularized social interchanges. Infants and parents are coming to know both each other, and the likely form of their interchanges. Partners can expect that their interactions will have an increasingly familiar form over successive weeks.

Developmental changes in the structure of interaction

There were also changes in the content of interaction over development. Turn-taking involving smiling increased with age. This means that over successive dyadic exchanges, infants and mothers can be increasingly confident that a smiling transition on the part of one partner will be followed by a transition on the part of the other partner. To our knowledge, this is the first demonstration of this phenomenon. It illustrates the development of a relatively conventional type of turn-taking social skill during interaction. At a session level, infants and mothers are becoming increasingly responsive to one another.

The Overall Predictability of Infant and Mother Actions within Interactive Sessions

Overall predictability

A second next set of analyses explored the basis of developmental regularities over age by modeling overall predictability within interactive sessions (see Figure 2). We were met with surprises. We expected that the timing of mothers’ actions would be more predictable than that of infants. This was not the case. Mother smiling transitions exhibited rising levels of predictability with infant age. Infant smiling transitions, by contrast, exhibited high levels of predictability that did not change appreciably with development. This was the opposite of what we had expected.

Overall responsivity

We hypothesized that, developmentally, the timing of infant actions would be increasingly impacted by the actions of the mother, and by the previous action in the dyad. This was also not the case. Models incorporating information concerning the other partner’s actions and immediate interactive history were typically but not always superior for infant and for mother, but this did not change with infant age.

Preliminary conclusions

The results stemming from this phase of modeling suggest that mother actions vis-à-vis the infant do not provide an overall matrix of predictable responsivity. It also indicates that infants—at least with respect to the timing of their actions—are not becoming more responsive to the other partner and more influenced by past dyadic actions in a relatively simple way. This led us to look at infant action in a more fine-grained, granular fashion.

The Predictability of Specific Actions in Specific Dyadic States

Relative frequency of mother and infant transitions

Probability density functions revealed that infants were never more likely to initiate a smile or gaze at mother than mother was to initiate or terminate a smile. Infant expressive action were rare in comparison to mother smile intiations and terminations (see Figure 3). These analyses underscore the degree to which mother’s initiation and termination of smiles were the norm during dyadic interactions. Concretely, the proportion of mother smile transitions (M = .47, SD = .09) was almost 50% more frequent than infant smile transitions (M = .33, SD = .10) and twice as frequent as infant gaze transitions (M = .22, SD = .06). That is, the proportion of mother smile transitions was greater than the proportions of either infant smile transitions or infant gaze transitions. The proportion of mother smile transitions, in fact, was roughly equivalent to the sum of the proportions of infant smile and gaze transitions. In any given system of dyadic actions, then, mothers accounted for a large share of the changes in dyadic states.

Development in the details

Developmental trends in infant predictability emerged by modeling specific actions in specific interactive contexts (see Figure 4). The dyadic state variable indicating whether infant or mother had acted most recently, for example, was associated with different developmental patterns of the predictability of infant action. The predictability of infant smile initiations after a mother smiling transition rose dramatically with development (see top left quadrant of Figure 4). That is, infant became more predictable in their smiling responses to mother. By contrast, infant smiling that was not in direct response to a change in mother smiling did not exhibit a clear developmentmental trajectory.

Predictability in the details

Different infant actions exhibited different developmental trajectories of predictability. The timing of infant gazes away from mother’s face became increasing predictable with development, but infant gazes at mother did not. All infant actions were much more predictable if the infant had acted last than if the mother had acted last. This suggests that the timing of an infant action is strongly influenced by a previous action on the part of the infant—whatever the content of that action is.

Do patterns of mother predictability provide a basis for the development of infant intentionality?

General increases in mother predictability

There was evidence that the timing of mother’s smiling actions became increasingly predictable with development (see Figure 2). From the infant’s perspective, for example, mother smile initiations tended to gradually become more predictable over developmental time (see Smile Initiations on the top and bottom graphs on the right side of Figure 4). Increases in mother predictability were not, however, contingent on infant actions. Increases in mothers’ predictability were apparent both when the infant had acted most recently as well as when the mother had acted most recently (compare the red dashed lines and blue solid lines in Figure 4). Increases in mother predictability were also not strongly tied to any specific infant action. Infant gazes at the mother’s face were associated with more predictable mother smile initiations than gazes away from the mother. Both types of gaze shifts, however, were developmentally associated with increases in the predictability of mother smile initiations.

Mothers as developmentally stable symmetry-makers

Mothers tended to exhibit relatively high and developmentally stable levels of predictability in acting to create symmetrical states of expressivity with their infants. Mothers tended to predictably terminate their smiles in response to their infants gazing away from them, for example, and in response to their infants terminating their smiles. By the same token, mother’s smiles in response to infant smiles were relatively predictable, and this predictability was stable over developmental time.

From the infant’s perspective

Overall, face-to-face interaction provided infants with relatively predictable consequences to their expressive actions. This predictability could provide the basis for expectations of such consequences and could subserve the emergence of intentional actions to achieve those consequences. The predictability of mother smiling actions, however, was only rarely dependent on the infant acting. As others have noted, infant smiles are sufficient to elicit a mother smile, but are not necessary (Symons & Moran, 1994). It is possible, then, that infants were not differentially reinforced for their own actions.

Increases in infant smiling predictability

Differential predictability to infant actions was observed only in response to infant smile initiations. Relatedly, infant smile initiations became more predictable over developmental time, both when the mother was not smiling and in response to mother smiles. It is possible that increases in the predictability of infant smile initiations coupled with predictable maternal responsive smiling yielded the increases in smile turn-taking over development documented here. If so, the use of temporal distributions to understand the predictability of mother’s response has revealed a pattern of responsivity not evident when considering only the differential probability of a mother response to the infant.

Relative infant and mother responsivity

Overall, the predictability of infant actions was not markedly dependent on mother’s state in any given age period (see Figure 4). Mothers exhibited more differential predictability dependent on the infant’s state. This pattern is becoming increasingly clear in the literature (Beebe, et al., 2007). We recently found, for example, that infant-to-parent interactive influence was structurally more important to modeling face-to face interaction than was parent-to-infant influence (Chow, Haltigan, & Messinger, 2010).

Absence of direct associations between infant and mother predictability

It is tempting but inaccurate to assume that direct associations exist between infant and mother predictability. Note, for example, that the predictability of mother smiling actions was relatively similar in relationship to infant smiling and infant gaze shifts. That is, mothers tended to respond to infant expressivity—gazing at mother and infant smiling—in a similar fashion (see the left half of Figure 4). Infants become increasingly predictable in initiating and prolonging their smiles; however, there was no increase in the predictability of infant gazing at mother.

Stochasticity in Interaction

The machine learning inspired models considered here revealed new characteristics of infant-mother interaction and its development. A focus on predictability suggested the difficulties faced by both infants and mothers in estimating responses to their actions within dyadic interactions. Specifically, partners in dyadic interaction are faced with significant stochasticity in the responses of their interactive partners in response to changes in facial expressions and gaze behavior (Cohn & Tronick, 1988a, 1988b; Fogel, 1988). The development emergence of regularity in the context of this variability is a challenge faced both by infants and those who model their development.

Reducing predictability?

Together, infant and mother behavioral proclivities created a highly dynamic system. Each partner was most predictable when acting in a context (a state) he or she had created, and was less predictable when responding to the other partner’s action. One possibility is that each partner acted, in many respects, to reduce the predictability of the other partner’s actions. When mothers initiate and terminate smiles, for example, they produce a changed visual display that may encourage infants to delay shifting their direction of gaze, and thus decrease the predictability of this action. Likewise, when infants match mothers smiles, they decrease the predictability of mother terminating her smile.

Repeated acting as a goal

An alternate view is that repeated action is a goal of both infants and mothers. This pattern is of reminiscent of autoregression effects in analyses of infant emotional engagement in which past behavior exerts a strong effect on current behavior {cohn; (Chow, et al., 2010; Cohn & Tronick, 1988b). The content of the two types of analyes, however, is different. Typically, auto-regressive effects indicate that current levels of emotional engagement predict future engagement across short time intervals. The current results are event-based; they use time to act as the unit of analysis. In that context, a given expressive action by infant or mother was associated with a relatively predictable opposing action by the same partner.

Limitations

The limitations of the current project are numerous. There is extensive evidence that infant and mother smiling and infant gaze direction are potent avenues of expressive communication (Beebe, et al., 2007; Cohn & Tronick, 1987; Kaye & Fogel, 1980). It may be, however, that the predictability of infant and mother actions were contingent on other actions not considered here such as vocalizations and touch (Crown, et al., 2002; Van Egeren, et al., 2001). More fundamentally, our unit of analysis was the distribution of time until a partner acted. Expressive actions were modeled dichotomously (e.g., smile versus no smile). Ultimately, however, some infant and mother expressive actions may have continuous qualities. Recently, for example, we have begun using computer vision to measure smiling on a multi-step intensity matrix based on the Facial Action Coding System (Ekman, et al., 2002; Messinger, Mahoor, Chow, & Cohn, 2009) (see below for details). Finally, qualitiative approaches to measuring interaction may also be promising. Hsu and Fogel (Hsu & Fogel, 2003) examined transitions between dyadic states such as symmetrical engagement in the current dataset. They found that the effect of dyadic propensities for given elapsed states over developmental time, elapsed time in a given state within the interaction session under examination, and the most recent state occupied within that interaction all exerted influence on the likelihood of a state transition. As in the current project, multiple levels of temporal influence affected the predictability of dyadic interaction.

Conclusion and Future Directions

Model system

Face-to-face interaction may be a model system for understanding early social development (Fogel, 1982). The system is physically constrained but involves multimodal infant-mother expressivity. It is naturalistic but amenable to experimental perturbations such as the still-face procedure in which the parent ceases responding to the infant (Mesman, van Ijzendoorn, & Bakermans-Kranenburg, 2009). It is fertile ground for the application of temporally precise measurement, analysis, and modeling. Face-to-face interactions can also reveal patterns associated with later developmental outcomes.

Predictive validity

In face-to-face interactions, higher levels of infant and parent responsivity appear to foster the development of infant social expectations. Infants whose parents shift affective states to match those of their infants during interactions before six months of age—and infants who are more likely to be affectively synchronous with their parents at nine months—display higher levels of self-control and cognitive functioning at two years (Feldman & Greenbaum, 1997; Feldman, et al., 1999; Feldman, Greenbaum, Yirmiya, & Mayes, 1996). Among typically developing infants, levels of positive affect during face-to-face interaction are associated with smiling referentially at objects and turning that smile toward an examiner at eight months of age (Parlade, et al., 2009). Among infants at risk for autism, the frequency of gaze shifts at and away from the parent’s face may be a rate-limiting factor in the development of later referential gazing between an interesting toy and an examiner (Messinger & Ibanez, 2010).

Precise measurement

A variety of precise measurement and analytic techniques have informed our understanding of the face-to-face system (see measurement.psy.miami.edu for downloadable tools and descriptions). Computer vision face tracking in conjunction with support vector regression has been used to reliably document the interactive flow of increases and decreases in infant and mother smiling intensity over time (Messinger, et al., 2009). Windowed cross-correlations of these intensity measurements have revealed temporally varying levels of influence in positive affect communication.

Temporally continuous measurement

The continuous measurements of non-expert raters have provided a person-on-the-street validation of infant and mother affective valence. These continuous ratings have revealed subtle differences between typically developing infants and infants at-risk for autism in both the level of their affective valence and its auto-regressive patterning (Baker, Haltigan, Brewster, Jaccard, & Messinger, 2010; Chow, et al., 2010). Stochastic regression analyses of these ratings has confirmed that the level of coupling of infant and mother expressivity itself varies over the course of an interactive session (Chow, et al., 2010).

Bootstrapping

Statistical simulations (bootstrapping) have been used to create a portrait of the temporal structure of infant expressive actions in interaction (Yale, Messinger, & Cobo-Lewis, 2003; Yale, Messinger, Cobo-Lewis, Oller, & Eilers, 1999). Infant gazes at their parents’ faces typically precede and set the stage for facial expressions such as smiles. Infants tend to insert vocalizations within the course of a smile, perhaps to call attention to this display (Yale, et al., 1999). More generally, infants’ gaze direction and vocalizations are both coordinated with smiles but not with one another. This type of modeling suggests the centrality of smiling to infant social communication (Yale, et al., 2003).

Inverse Reinforcement Learning

An immediate goal in the modelling of face-to-face interaction is investigating whether mothers and infants schedule their expressive displays in order to accomplish specific goals. For example, if mothers have a tendency to smile more often in situations is likely the infant is likely to respond with a smile, one can surmise that a reasonable representation of a mother’s goal is that she is trying to maximize the time spent in joint smiling with her infant while mimizing unreturned smile bids. Recent developments in a subfield of machine learning—Inverse Reinforcement Learning—provide a principled way of framing this problem (Abbeel & Ng, 2004; Ng & Russell, 2000; Ramachandran & Amir, 2007). This approach assumes that the target interactive partner chooses actions so as to maximize some reward signal (e.g., the other partner smiling), and then determine a reward signal that is most parsimonious given their action choices. This type of analysis provides another tool for understanding dyadic interaction by addressing the question of why specific timing patterns emerge within mother infant interactions.

Artificial agents

The rigorous modeling of infant and mother expressive actions has the potential to inform the development of simulated and actual interactive agents with determinate interactive capacities. An immediate goal of such simulations would be to determine whether given expressive proclivities on the part of each partner would produce interactive processes such as those documented here. An ultimate aim is to determine whether the repeated interactions of artificial agents—simulated or robotic—yield developmental changes that characterize early human development.