Hand-Eye coordination predicts joint attention

Abstract

The present paper shows that infant and dyad differences in hand-eye coordination predict dyad differences in joint attention. In the study reported here, 51 toddlers ranging in age from 11 to 24 months and their parents wore head-mounted eye-trackers as they played with objects together. We found that physically active toddlers aligned their looking behavior with their parent, and achieved a substantial proportion of time spent jointly attending to the same object. However, joint attention did not arise through gaze following but rather through the coordination of gaze with manual actions on objects as both infants and parents attended to their partner’s object manipulations. Moreover, dyad differences in joint attention were associated with dyad differences in hand following.

Momentary looking behavior is tightly tied to one’s internal attentional state (Baron-Cohen, 1997; Brooks & Meltzoff, 2005; Frischen, Bayliss, & Tipper, 2007). It is for this reason that eye-tracking measures are widely used in behavioral research (R. Aslin & McMurray, 2004; M. Hayhoe & Ballard, 2005; S. P. Johnson, Amso, & Slemmer, 2003; Kingstone, Smilek, Ristic, Friesen, & Eastwood, 2003; Knoeferle & Crocker, 2006; Richardson & Dale, 2005; C. Yu & Smith, 2011). It is for this same reason that people attend closely to the eyes of their social partner and use the partner’s gaze direction to establish the common ground necessary for smooth social engagements (Argyle, 2007; Bayliss et al., 2013; Corkum & Moore, 1995; S. Johnson, Slaughter, & Carey, 1998; P. Mundy & Newell, 2007). These smooth social interactions and coordinated visual attention they require are also central to healthy development in many domains. Individual differences in infants’ and children’s ability to coordinate visual attention with a social partner strongly predict individual differences in language, social, and cognitive development (Brooks & Meltzoff, 2005; Peter Mundy & Gomes, 1998).

The traditional laboratory studies in which these predictive individual differences have been documented typically measure infants’ responses to joint attention bids, that is, their ability to follow gaze shifts, head turns, and sometimes manual points so as to jointly attend to the same object with their social partner (Baldwin & Moses, 1996; Brooks & Meltzoff, 2005). In many of these experiments, the signals indicating the direction of attention of the mature partner were designed to be unambiguous (e.g., concurrent gaze and head shifts) and were repeated to ensure that the infant attended to them. Moreover, the spatial tasks in these laboratory settings were purposely simple with an experimenter directly facing the infant and with just two potential targets on opposite sides of midline. This task structure fits the goal of measuring and accessing the infant’s ability to interpret gaze direction as a meaningful social cue, unhindered by potential limitations of the infant’s spatial precision in interpreting gaze direction (Butterworth & Cochran, 1980; Corkum & Moore, 1995; Scaife & Bruner, 1975). Although individual differences in these tasks are related to individual differences in social and language skills (Brooks & Meltzoff, 2005; P. Mundy & Newell, 2007; Wellman, Phillips, Dunphy-Lelii, & LaLonde, 2004), the developmental origins of these individual differences are not known.

Everyday parent-infant interactions such as joint toy play are much messier than the “clean” and diagnostic laboratory tasks described above (also see Kingston, Smilek, & Eastwood, 2010). The spatial context is often crowded with multiple potential targets close to each other (G.O. Deák, Walden, Yale Kaiser, & Lewis, 2008). Therefore, the spatial precision of head and eye direction may not be a sufficient cue to differentiate given multiple spatially close objects (Langton, 2000; Loomis, Kelly, Pusch, Bailenson, & Beall, 2008; Vida & Maurer, 2012). In everyday social interactions, infants do not just respond to joint attention bids but also initiate them (see Mundy & Newell, 2007). Thus, in everyday free-flowing interactions, the two partners may sometimes have competing attentional goals that need to be resolved if they are to share attention to the same object. In brief, everyday social interactions are spatially and dynamically complex and challenging, and thus likely to increase individual differences relative to laboratory tasks. The everyday interactions of parents and infants are also the likely training ground in which infants first learn to read social cues and to coordinate attention with partners (Bakeman & Adamson, 1984; Gredeback, Fikke, & Melinder, 2010; Triesch, Teuscher, Deák, & Carlson, 2006; Ullman, Harari, & Dorfman, 2012; C. Yu & Smith, 2013). Differences in these interactions then may be the source of individual differences in infants’ developing abilities to read and send social cues. This is the overarching hypothesis that motivates the present study.

Within this larger framework, we focus on one key factor: manual actions on objects. Our hypothesis that actions on objects are critical to the establishment of joint attention between parents and infants was suggested by a prior study in which parents and their 12-month-old infants played together with toys (Rader & Zukow-Goldring, 2010; C. Yu & Smith, 2013). In that study, parents and infants both wore head-mounted eye tracking systems that measured the momentary gaze direction of each partner and provided a precise measure of the coordination of visual attention to the same object. Consistent with findings from a growing number of studies (R. N. Aslin, 2009; Gedeon O. Deák, Krasno, Triesch, Lewis, & Sepeta, 2014; Franchak, Kretch, Soska, & Adolph, 2011; Rader & Zukow-Goldring, 2010; Yoshida & Smith, 2008), the gaze data indicated that infants rarely looked to their parent’s face, a fact that precludes gaze following by the infant as a contributing factor to joint attention. The gaze data also indicated that the dynamics of infant visual attention were very different from those of parents, whereas parents rapidly shifted eye gaze among many visual targets generating a series of brief fixations, and infants generated long looks and showed sustained attention to an object, a property of infant attention during toy play that has been noted by other researchers (H.A. Ruff & Lawson, 1990). Nonetheless, parents’ and infants’ visual attention were often coordinated. Indeed, parents and infants not only often fixated the same object at the same time but they often jointly shifted attention from one object to another in near unison, at the time scale of adult-adult interpersonal coordination (Marsh, Richardson, & Schmidt, 2009; Shockley, Santana, & Fowler, 2003).

How did they achieve this smooth coordination despite the different dynamics of parent and infant visual attention and despite the fact that the infants rarely looked to their parent’s face? The eye-tracking data indicated a strong role of hand actions on objects: When infants manually interacted with an object, they looked at the object in contact with their own hands and parents also looked at those infant-handled objects. When parents manually contacted an object, they looked at the object in contact with their own hands and infants also looked at the object being manipulated by the parent. In brief, because one’s own eye gaze and one’s own hand actions are spatially coordinated in goal-driven actions, directing visual attention to the object being manipulated by one’s social partner will results in joint attention between the two partners to the same object without gaze following (Yu & Smith, 2013). Figure 1 shows two paths through which hand-following may yield joint attention between infants and parents: (a) the infant handles an object and parent gaze follows the infant’s hands to the object and (b) the parent handles an object and the infant gaze follows the parent’s hands to the object. Gaze following and hand following may be distinct routes to joint attention that require different sequences of behaviors by the follower. Gaze following requires the follower to look at the initiator’s face and then switch attention to the spatial location to which the initiator’s gaze is directed. In contrast, the hand-following pathway would seem to have just one step: looking at the object in contact with the partner’s hand. Because hands and the handled objects are spatially close to each other, this makes hands a much more salient and robust cue to attention direction. Figure 1(c) also shows the gaze-following pathway to joint attention that has been the sole focus of previous research on joint attention. The present paper focuses on the “hand-following” pathways to joint attention that are potential sources of individual differences in how well parents and infants can coordinate their visual attention to the same object. If hand-following is the principle path to joint attention for parents and toddlers in joint object play, then dyad differences in joint attention should be associated with individual differences in these manual activity and hand-eye coordination components.

Figure 1.

Hand following and Gaze following pathways to joint attention. In (a), the infant holds an object and visually attends to his own hands as they handle the object and the parent attends to the infant’s hand actions, leading to both parent gaze and infant gaze directed to the same object. In (b), the parent holds the object and attends to her own hands as does the infant. The four dashed lined in (a) and (b) show the four hand-eye links that are the focus of the present study– hand-eye coordination within infant, hand-eye coordination within parent, parent eye to infant hand, and infant eye to parent hand. The traditional gaze following pathway to joint attention to an object is shown in (c).

To test this hypothesis, we used a method similar to that in the previous dual eye-tracking study of 12 month olds and their parents (Yu & Smith, 2013). The task context was free-flowing parent-infant play with multiple toys. Head-mounted eye tracking systems were worn by both participants allowing us to record eye-in-head position from both infants and parents during play. Gaze to and hand actions on objects by both parents and infants were recorded and coded. The infants participating in the present study ranged in age from 11 to 24 months in an effort to capture a broader range of individual and dyad differences. We focused on the second year of life because that is when infants become increasingly active and autonomous during this age range (Eckerman & Didow, 1989), because individual differences in both motor behavior and joint attention are noticeable during this period (Landa, Gross, Stuart, & Faherty, 2013), and because individual differences in manual actions on objects have been linked to differences in sustained attention (H. Ruff, 1986; H.A. Ruff, Capozzoli, & Weissberg, 1998), to parent talk about objects (Karasik, Tamis-LeMonda, & Adolph, 2014) and to language development (Iverson, 2010).

Our measure of joint attention was straightforward and transparent based on calculating, frame by frame, the moments that children and parents looked to the same object at the same time. Our measure of hand-eye coordination was one that is taken for granted in adult research: the systematicity with which hands and eyes are directed to the same object. More advanced measures of hand-eye coordination assume the spatial correspondence of hands and eyes and focus on precise timing and velocity profiles (Flanders, Daghestani, & Berthoz, 1999; M. M. Hayhoe, Shrivastava, Mruczek, & Pelz, 2003; Johansson, Westling, Bäckström, & Flanagan, 2001; Sailer, Flanagan, & Johansson, 2005). However, the direction of hands and eyes to the same object is not certain in toddlers (Bertenthal & Von Hofsten, 1998; Bushnell & Boudreau, 1993; Eppler, 1995; Iverson, 2010; Lockman & McHale, 1989; Soska, Adolph, & Johnson, 2010). Accordingly, three sets of analyzed were conducted: the first set focused on gaze patterns and joint attention. In preview, the frequency of these joint attention bouts was only weakly related to infant age and dyad differences were much larger than differences related to infant age. The second set of analyses partitioned the dyads into Low and High Joint Attention (JA) groups based on the frequency of their joint attention bouts and examined between-group differences in the components of the hand-following pathway – manual activity, within individual hand-eye coordination, and between partner hand-eye coordination. The third set of analyses used regression to examine the associations among the components of hand-eye coordination as well as age as predictors of the frequency of joint attention during a dyad’s toy play.

Method

Participants

For the main experiment, the final sample consisted of 51 (24 male toddlers) parent-toddlers dyads with the toddlers ranging in age from 11 to 24 months (mean = 17.92, SD= 4.15); 14 additional dyads began the study but the toddlers refused to wear the measuring equipment throughout the entire procedure. Data were collected between June 2013 and April 2014. Because the eye-tracking equipment on the parent could alter toddler gaze to the parent or the social interaction in some way, we also tested 5 toddlers (2 male, between 16–20 month old) in a version in which only the toddler wore the head-tracking gear to ensure that toddler gaze in the main experiment was not altered by the head gear worn by the parent. The entire sample of toddlers was broadly representative of Monroe County, Indiana (84% European American, 5% African American, 5% Asian American, 2% Latino, 4% Other) consisting of predominantly working- and middle-class families. Toddlers were recruited through birth records and community organizations (e.g., museums, children’s outreach events, boys and girls clubs) that serve a diverse population.

Stimuli

There were 6 unique novel “toys” constructed in the laboratory and pilot-tested to be interesting and engaging to infants. Each novel toy was a complex object made from multiple and often moveable parts, and ranged in size from 5cm to 8cm in length, 6cm to 12cm in width and 4cm to 6cm in depth when measured from their gravitational upright (flat bottom of object placed on a surface). These were organized into two sets of three so that each object in the set had a unique uniform color.

Experimental setup

As shown in Figure 2, parents and toddlers sat across from each other at a small table (61cm × 91cm × 64cm). Parents sat on the floor such that their eyes and heads were at approximately the same distance from the tabletop as those of the toddlers, a posture that parents reported to be natural and comfortable. Both participants wore head-mounted eye trackers (Positive Science LLC, http://www.positivescience.com/; also see Franchak et al., 2011). The Positive Science eye-tracker was designed for use with infants and was designed to be attached to the head so as to be stable on the head (even in self-locomoting infants and toddlers, see Franchak & Adolph, 2010, Frachak, Kretch, Soska, Babcock, & Adolph, 2010). The tracking system has been widely and successfully used in both infant and adult research (Baschnagel, 2013; Evans, Jacobs, Tarduno, & Pelz, 2012; Franchak et al., 2011; Kretch, Franchak, & Adolph, 2014; Macdonald & Tatler, 2013; Maldarelli, Kahrs, Hunt, & Lockman, 2015). Both parent and infant eye-tracking systems include an infrared camera – mounted on the head and pointed to the right eye of the participant – that records eye images, and a scene camera that captures the events from the participant’s perspective. The scene camera’s visual field is 108 degrees, providing a broad view but one less than the full visual field --approximately 170° (L. Smith, Yu, Yoshida, & Fausey, 2014). Each eye tracking system recorded both the egocentric-view video and eye-in-head position (x and y) in the captured scene at a sampling rate of 30 Hz.

Figure 2.

The dual eye tracking experimental paradigm wherein toddlers and parents played with a set of toys on a tabletop in a free-flowing way. Both participants wore a head-mounted eye tracker that recorded their moment-to-moment gaze direction from their egocentric views. Also shown are three of the laboratory-made toys with their multiple moveable parts and uniform colors.

Placing the head gear and eye tracker calibration

Prior to entering the testing room, in the waiting area, the first experimenter desensitized the toddler to touches to the head and hair by lightly touching the hair several times when the attention and interest of the toddler was directed to a toy. Both the parent and the toddler entered the experimental room, and a second experimenter and the parent engaged the toddler with an enticing toy with buttons to push that make animals pop up. The toddler’s head gear was placed while the toddler was engaged with the toy. This was done in one movement and care was taken by the experimenter to ensure that the toddler remained engaged with the toy and that the toddler’s hands didn’t go to the head gear. The first experimenter then adjusted the scene camera to ensure that the button being pushed by the toddler was in the center of the scene camera. We have used this procedure in multiple head-camera and head-mounted eye-tracking experiments (Pereira, Smith, & Yu, 2014; L.B. Smith, Yu, & Pereira, 2011; C. Yu & Smith, 2012; C. Yu & Smith, 2013; C. Yu, Smith, Shen, Pereira, & Smith, 2009) with an overall 70% success rate. Detailed information can be found in Appendix I.

Instructions and procedure

Parents were told that the goal of the experiment was to study how parents and toddlers interacted with objects during play and therefore they were asked to engage their toddlers with the toys and to do so as naturally as possible. Each of the two sets of toys was played with twice for 1.5 min, resulting in 6 minutes of play data from each dyad. Order of sets (ABAB or BABA) was counterbalanced across dyads.

Data processing

During post-processing and before coding, the quality of the eye tracking videos (with eye images superimposed) for each toddler and parent was checked as described above to ensure the quality of calibration at the end as well as the beginning of the session. Re-calibration would be conducted if necessary.

The eye-tracker collected at a rate of 30 frames per second for approximately 360 seconds (four trials with 1.5 minute per trial) of interaction, yielding potentially 10800 data points per measure for each participant. Not all participants provided eye-tracking data for the entire session, the mean number of good eye-tracking frames was 8125 (SD = 984) for toddlers and 8356 (SD= 825) for adults. Roughly 25% of frames from toddlers that were not codable with respect to regions of interest (ROIs, defined in the next paragraph); this was due to 10% eye-tracking failure and the rest due to the toddler’s being off task (looking elsewhere than defined regions of interest). The main data for analyses were gaze data directed to four ROIs (described below). All results are reported in terms of percentage of the total interaction time – as if the number of recorded frames equaled the total number of possible (that is 10800) frames. Therefore, estimates of percentage time on ROIs are under-estimates because they include both off-task time and eye-tracking failures.

ROI coding was done by human coders. These coders were highly trained and code these variables for many different experiments and projects. They were naïve to the specific hypotheses and experimental questions of this study. The four regions-of-interest (ROIs) were defined in the head-camera videos: the three toy objects and the partner’s face. From gaze ROI coding, each dyad provided two gaze data streams containing the four ROIs as shown in Figure 3(a). A second coder independently coded a randomly selected 10% of the frames with the inter-coder reliability ranging from 82% to 95% (Cohen’s kappa = 0.81). Detailed information about coding and reliability is provided in Appendix II.

Figure 3.

An overview of raw and derived data. (a) Visual attention: two gaze data streams from child and parent include four regions of interest – three toy objects and the partner’s face. Sustained visual attention (the third row) is derived based on child’s and parent’s gaze data. (b) manual activities on objects from child and parent. (c) hand-eye coordination: four types of hand-eye coordination within each person and across two partners. At some moments, both child’s and parent’s eyes are “caught” by what the child was manually handling. At other moments, they jointly attended to objects in parent’s hands.

Manual contact with an object (who and which object) was also coded frame-by-frame from the images captured by the overhead camera and the other two third-person cameras (shown in the right column in Figure A1 in Appendix). We developed a custom coding program which allowed coders to access three views simultaneously to determine which object was manually handled frame by frame. In practice, coders most often relied on the view of the overhead camera, but in case of uncertainty, they would consult with the other two views to make a decision. Similar to gaze ROI coding, each video was coded through three rounds wherein one object was focused in each round and the coder made the yes/no decision that a hand –and whose hand –was in contact. This coding scheme increases the total coding time compared with coding all three objects at once, but reduces error. The second coder also independently coded a randomly selected 10% of the frames with the inter-coder reliability ranged from 91% to 100% (Cohen’s kappa = 0.94).

Figure 3(c) shows frame-by-frame measures of four types of hand-eye coordination. For example, at Moment a, both parent’s and toddler’s eyes were on the object manually handled by toddler; at Moment b, both attended to the object manaully handled by parent.

Results

Individual gaze patterns and joint attention

Figure 3(a) shows a representative example of the raw gaze data streams for one dyad. Table 1 provides the summary statistics of several measures of infant and parent looks within ROIs for the entire sample. For each type of looking behavior, we report three measures: 1) percentage of total looking time within ROIs, 2) frequency with which these looks occurred (in rate/min), and 3) mean duration of within-ROI looks (in sec). For all measures, correlations with age of the infant were small and not significant, with one exception (proportion of time infants looked within the ROIs versus “off-task”), suggesting that neither infants’ nor parents’ looking behaviors change systematically as a function of infant age. However, consistent with previous findings (Smith, Yu & Pereira, 2011; Yu & Smith, 2012, 2013), infants and parents differed considerably and reliably on all measures: Infants and parents spent a high proportion of time fixating the ROIs but parents spent more total time overall than infants (M_parent=82.58%, M_infant=75.76%) and exhibited more attentional switches between objects and faces (M_parent=61.29 switches per minute) than infants (M_infant=25.46). Infants, in contrast, had longer unbroken looks within the same ROIs than did the parents (M_parent=806msec, M_infant=1825msec), showing the “stability” often observed in infant and toddler attention during object play (Kannass, Oakes, & Shaddy, 2006; C. Yu & Smith, 2013). The different dynamics of infants’ and parents’ visual attention (e.g. see an example shown in Figure 3(a)) suggest two different attentional systems with two different rhythms which could be challenging for the coordinating of the two partners’ visual attention. Moreover, infants fixated the objects more than their parents (M_parent=64.80%, M_infant=55.33%), whereas parents fixated the faces of their infants much more often than infants looked to their parents’ face (M_parent=34.03%, M_infant=11.61%).

Table 1.

Parent and infant differences in fixations on the defined ROIs: the three objects and each other’s face.

	Infant mean (SD)	correlation with age	Parent mean (SD)	correlation with age	infant-parent comparison
Fixations to ROIs
% of time	75.76 (12.65)	0.256*	82.58 (12.23)	0.016	t(50)=2.77 p<.001, d=0.783
frequency (rate/min)	25.46 (8.33)	0.145	61.29 (13.71)	0.032	t(50) =17.366 p < .001, d = 4.910
duration (msec)	1825 (745)	0.103	806 (282)	0.021	t(50) = 8.79 p < .001, d = 2.486
Looks to faces
% of time	11.61 (7.14)	−0.04	34.03 (13.87)	−0.203	t(50) = 8.038 p<.001, d = 2.273
frequency (rate/min)	4.74 (2.28)	0.099	21.52 (7.22)	−0.155	t(50) = 15.805 p<.001, d=4.265
duration (msec)	1364 (530)	0.085	791 (338)	−0.01	t(50) =6.514 p<.001, d=1.841
Looks to objects
% of time	64.8 (13.71)	−0.127	55.33 (11.63)	−0.235	t(50) = 5.573 p<.001, d=1.576
frequency (rate/min)	20.71 (7.96)	0.123	39.77 (10.80)	0.145	t(50) = 10.13 p<.001, d=2.865
duration (msec)	2093 (885)	0.139	826 (243)	0.049	t(50) =9.86 p<.001, d=2.789

^*p<0.05

These patterns –and perhaps particularly infant looks to parent face -- do not appear to depend on the fact the parent head gear in some way altered infant gaze patterns. The results from the additional 5 infants who interacted with a parent not wearing any head gear were similar to infants in the main study (overall duration: M_infant=1809msec; overall frequency: M_infant=29.67, face look duration: M_infant=1302msec, face look frequency: M_infant=4.65). Within this sample, and consistent with past findings in with both participants wore head gear (Yu & Smith, 2013) and in which only the infant did (Yoshida & Smith, 2008), infants rarely looked to their parents’ face during the play session. This fact precludes infant gaze following as a route to joint attention.

To find joint attention episodes in the gaze stream data, we applied the same method used in the previous dual head-mounted eye-tracking study (Yu & Smith, 2013). We first determined –frame by frame – the frames in which parents and infants looks were within the same ROI (on the same object or on each other’s face). Meaningful shared attention should last some amount of time longer than a frame (33msec) but might also include very brief looks elsewhere. Therefore, a joint attention (JA) bout was defined as a continuous alignment of parent and toddler looks within the same ROI that lasted longer than 500 msec and included segments of these looks that were to the same object but separated by brief looks away by one partner of no longer than 300 msec. We’ve tested slightly different defining windows for JA measures, using 400msec and 600msec for the minimal duration of JA, and 200msec and 300msec for in-between brief looks. Repetition of the analyses with those thresholds yielded the same patterns of results. Examples of the so-defined joint attention bouts from one dyad’s gaze streams are shown in Figure 3(a).

Column 1 of Table 2 summarizes a set of statistics on joint attention measures across the whole sample: the percentage of overall time in joint attention within any ROI, the frequency with which joint attention bouts were formed (in rate/min), and the mean duration of these bouts (in sec). These same statistics are provided for the two subcomponents of overall joint attention – mutual gaze and joint attention to an object. Overall, the results in Table 2 indicate many moments of visual gaze coordination, especially for looks to objects: Parents and toddlers looked at the same object at the same time over 32% of the play session; in contrast, they looked at each other’s faces at the same time (mutual gaze) only 5% of the time. The overall time in coordinated attention to objects consisted of multiple bouts of joint attention, on average, over 8 such bouts per minute. As shown in column 2 of Table 2, these measures were generally not reliably associated with age with the only statistically reliable correlation being between percentage of overall JA time and age. However, there were substantial dyad differences -- joint attention episodes varied from near 14% to over 63% of the play session.

Table 2.

Measures of Joint Attention Low and High Joint Attention (JA) Dyads

	whole sample	correlation with age	Low JA	High JA	low-high comparison
overall
% of time	39.24 (12.07)	0.295*	28.08 (8.06)	48.41 (4.85)	t(49)=10.61 p<.001, d=3.03
frequency (rate/min)	9.44 (2.19)	0.246	7.82 (1.55)	10.76 (1.70)	t(49) =6.45 p < .001, d = 1.84
duration (msec)	2.38 (0.53)	0.199	2.02 (0.434)	2.68 (0.44)	t(49) = 5.49 p < .001, d = 1.56
mutual gaze
% of time	5.74 (4.56)	0.093	4.27 (4.05)	4.92 (6.62)	t(49) = 0.72, n.s.
frequency (rate/min)	1.78 (1.28)	0.06	1.37 (0.96)	1.81 (1.43)	t(49)= 0.93, n.s.
duration (msec)	2.44 (0.68)	−0.02	1.54 (0.69)	1.65 (0.73)	t(49) = 0.78, n.s.
JA to object
% of time	34.72 (4.56)	−0.092	23.82 (7.68)	41.76 (6.90)	t(49) = 8.94 p<.001, d=2.55
frequency (rate/min)	7.66 (1.65)	0.212	6.45 (1.41)	8.65 (1.07)	t(49) = 5.09 p<.001, d=1.45
duration (msec)	1.68 (0.71)	0.42	1.99 (0.46)	2.82 (0.61)	t(49) =5.77 p<.001, d=1.65

^*p<0.05

The main hypothesis motivating this study is that toddlers and parents create joint attention moments by jointly attending to the objects being manually handled and therefore that dyad differences in hand-eye coordination are critical predictors of dyad differences in the frequency of joint attention. Consistent with this hypothesis, across dyads, for 82.34% of joint attention moments on a visual object, the jointly attended object was being manually contacted by at least one partner. For non-joint attention moments, only 43.67% of the time was one of the partners hands in contact with an object, (t(100)=29.40, p<0.001, d=6.07). Infants held the jointly attended object 45.23% of time (SD=5.63%); parents held the jointly attended object 37.72% of time (SD=5.81%), reliably less often than did their infants (t(100)=5.85, p<0.001, d=1.17).

Low and High JA groups

We partitioned dyads into those with high and low incidence of JA bouts using a median split of the overall percentage of joint attention time. Columns 3 and 4 of Table 2 provide the statistics for the two defined groups for the measures of percentage of JA time, frequency of JA bouts and duration of JA bouts. Because the two groups were defined by the overall percentage of time in joint attention, the expectation is that they would differ on all the components contributing to this overall measure. As shown in column 5 of Table 2, this is generally true with the exception of measures of mutual gaze, a low frequency behavior in the present study, and one that at least in the context of active toy play may not be linked to the likelihood of joint attention (see also Yu & Smith, 2013). High and Low JA dyads also did not differ in the frequency with which parents looked to infant faces, M_high=22.77, M_low=20.23, t(49) < 1.00, nor in the frequency with which infants looked to parent faces, M_high=4.56, M_low=4.94, t(49) < 1.00. The High JA infants were younger than the Low JA infants but the difference was not reliable; the mean age of the High JA infants was 18.78 months (SD = 4.24) and the mean of the Low JA infants was 19.8 months (SD=3.91), t(49) = 1.61, p=.113. Thus, neither age of the infant nor visual attention to the partner’s face seems to be a determining factor of dyad differences in joint attention.

For “hand following” to play a role in joint attention, the participants first need to handle the objects, the solid black arrows in the two hand-following pathways in Figure 1(a) and (b). Table 3 shows the summary statistics for the percentage play time that infants and parents were in manual contact with an object across all dyads and also for High and Low JA dyads. Infants only were handling an object more than a quarter of the time, parents only were handling an object also about a quarter of the time, and the two partners were both handling objects 36% of the time. Only the frequency of infant handling differed between High and Low JA groups; infant handling of an object was also reliably (albeit modestly) correlated with age.

Table 3.

Percentage Total Play time in which Hand was in Contact with an Object for Low and High Joint Attention (JA) Dyads, SD in parentheses.

	overall	correlation with age	Low JA	High JA	low-high comparison
infant	29.73 (16.45)	0.31* p=0.02	24.07 (13.72)	35.62 (17.24)	t(49)=2.65 p<.01, d=0.75
parent	26.17 (15.38)	−0.12 p=0.16	29.31 (16.14)	22.91 (14.13)	t(49) =1.50, n.s.
both	35.96 (16.91)	−0.22 p=0.12	38.15 (18.68)	33.7 (14.90)	t(49) = 0.94, n.s.
neither	8.13 (6.56)	0.23 p=0.10	8.48 (7.74)	7.76 (5.19)	t(49) = 0.04, n.s.

^*p<0.05

Recent studies suggest that handling objects matters to joint attention because partners look to their own and to their partner’s hand actions on objects (Rader & Zukow-Goldring, 2010; C. Yu & Smith, 2013; Chen Yu & Smith, 2016). Figure 4 shows the proportion of total time that either the infant’s or parent’s gaze was fixated on a hand-handled object in both high JA and low JA groups for the three kinds of handling moments –infant (only) handling an object, parent (only) handling and object, and both handling an object.

Figure 4.

The proportion of total time child and parent visually fixate the target object, when the child is handling the target and the parent is not manually in contact with an object (A), when the parent is handling the target object the child is not handling any object (B), when the child is handling the target and the parent is handling another object (C); and when the parent is handling the target and the child is handling another object (D).

Figure 4(a) shows the frequency with which gaze was directed to the infant-handled object at the moments that infants (only) were manually in contact an object. As predicted, Low JA infants showed less hand-eye coordination, looking to their own manual actions on objects less frequently than high JA infants. High and Low JA parents also differed: High JA parents attended to the object being handled by their infant more than Low JA parents. These conclusions were confirmed via a 2 (JA group) X 2 (parent gaze vs. infant gaze) analysis of looking behavior for the cases when the (only) infant handled an object. Besides the two main factors, the interaction between the two would indicate that infants across the two groups might look equally long at the objects they handled, but their parents differed in their attention on the infant-handled objects, or that parents across the two groups might look equally long at the objects handled by infants, but infants differed in their attention on self-handled objects. The results revealed only two main effects – Low versus High JA, F(1,98) = 12.11, p<0.001, η_p² = 0.08, and parent gaze vs. infant gaze, F(1,98)=18.37, p<0.001, η_p² = 0.16. Across both groups, when infants were handling an object, the infant was more likely to be looking at that object than the parent, but High JA infants and High JA parents looked more at the object handled by the infant than did Low JA infants and Low JA parents. These findings provide support for the main hypotheses from the infant side of manual actions: Infants who were more likely to achieve joint attention bouts with their parents not only manually act on objects more, but they also looked more to the object when they were handling it. Moreover, they had parents who visually followed their hand actions to objects more frequently than did Low JA infants. Put in other words, when the infant was handling a potential target object for joint attention, the infants in Low JA dyads showed less within-self hand-eye coordination and their parents showed less between-self-and-infant hand-eye coordination.

Figure 4(b) shows the frequency with which gaze was directed to the parent handled object (when infants were not manually in contact with object). Here we see no group differences in gaze directed at the handled object. A 2 (high vs. low JA group) x 2 (parent gaze vs. child gaze) analysis of variance yielded no significant main effects nor interactions (F_group(1,98)=3.56,p=0.06; F_agent(1,98) = 0.17,p=0.67, n.s.; F_interaction(1,98)=0.11,p=0.73, n.s.). The lack of differences in hand-eye coordination across the two groups in these cases suggests that dyad differences may lie primarily in the pathway shown in Figure 1 (a), infant handling an object, than in the pathway shown in Figure 1 (b), parent handling an object.

Figure 4(c) and (d) show the findings from the more complicated cases in which the infant and parent were each holding different objects in which they could attend to either the object handled by the infant or the one handled by the parent. To which object did the partners jointly look? For the objects held by the infant, a 2 (JA group) x 2 (parent gaze vs. infant gaze) ANOVA indicated a main effect of JA group (F(1,98) = 6.87, p<0.01, η_p² = 0.08), and parent gaze vs. infant gaze (F(1,98)=15.74, p<0.001, η_p² = 0.14), but no interaction, F(1,98) = 0.09, p=0.75, n.s. High JA parents and infants paid more attention to the objects being handled by the infant than did Low JA parents and infants. The same analyses with respect to the object handled by the parent revealed only a significant effect of JA group (F(1,98) = 10.48, p<0.005,η_p² = 0.11) with parents and infants in the High JA dyads attending more to the objects handled by the parent than did Low JA infants and parents. No other effects approached significance. In a context in which there were two potential targets for shared attention, one in the parent’s hands and one in the infant’s hands, High JA infants and parents managed to find a joint solution more frequently than did Low JA infants and parents.

These results provide clear support for the hand-following pathway in parent-infant joint attention and suggest that the origins of individual differences may be located in infant manual activity: High and Low JA dyads are distinguished by the frequency of infant manual activity on objects, by infant attention to their own object manipulations, by parent attention to infant object manipulations, and by the joint resolution of competing targets when the two partners are holding different objects.

Correlational analyses

Table 4 shows the bivariate correlations among joint attention (percentage time) and the three components of the hand-following path: 1) manual activity, 2) within-partner hand-eye coordination and (3) between-partner hand-eye coordination. All measures except parent handling of objects correlated reliably with joint attention to an object. These correlations are consistent with the categorical group analyses and provide support that parents and infants use both of the hand-following pathways in Figure 1 (a) and (b). The new information concerns the dependencies among components of hand-following pathways. Infant object manipulation correlated strongly and negatively with parent object manipulation. Given that parents and infants often acted simultaneously on separate objects, this is not a necessary dependency but suggests that parents are more active when their infants less active (or vice versa). There was a strong association (0.56) between the likelihood that the infant looked to their own object manipulations and the likelihood that the parent looked to infant object manipulations, the two hand-eye coordination components critical to the infant-handling object pathway shown in Figure 1(a). If infants do not systematically look to their own hand actions on objects, parents may be less likely to follow those hand actions with their own gaze because hand actions are unreliable cues as to the direction of their infant’s visual attention. There was not a strong correlation between parent hand-eye coordination when the parent was handling the object and infant gaze to the parent-handled object, the two hand-eye coordination components of the parent-handling object pathway in Figure 1(b). Apparently, parent hand-eye coordination is not a factor in infants’ visually following of the parent’s hand movements. The correlation between the parent’s own hand-eye coordination and parent looks to infant object handling was reliable; the more parents paid attention to the self-handled object, the more likely they were to pay attention to the object handled by the infant.

Table 4.

Bivariate correlations among the hand, eye, object links in Figure 1a and b.

	Manual activity		within hand-eye		between hand-eye		JA
	Infant actions	Parent actions	infant gaze when infant holding	parent gaze when parent holding	infant gaze when parent holding	parent gaze when infant holding
manual activity
infant actions		0.63***	0.23	−0.03	0.03	0.24	0.33*
parent actions			0.01	0	0.06	−0.16	−0.11
within hand-eye
infant gaze when infant holding				0.23	0.04	0.56***	0.45***
parent gaze when parent holding					0.21	0.37**	0.41**
between hand-eye
infant gaze when parent holding						−0.02	0.34*
parent gaze when infant holding							0.44**

^*p<0.05,

^**p<0.01,

^***p<0.005

The overall pattern of correlations suggests the hypothesis shown in Figure 5: The causal pathways through which within- and between-partner hand-eye coordination contributes to joint attention during active play with toys may be primarily from infant’s own hand-eye coordination to parent attention to infant hands. A confirmatory path analysis was conducted as a hierarchical sequential analysis as recommended by (Pedhazur, 1997). Parents’ gaze to infant hand and joint attention were the endogenous variables since their variance is hypothesized to be explained by other variables in the mode. A multiple regression was conducted for each endogenous variable in which all variables hypothesized to have direct effects on the endogenous variable were included. The beta weights for these multiple regressions are the path weights in the model. In the confirmatory model, the two between-partner components of hand-eye coordination were treated as independent contributors to parents’ attention to objects handled by their infant. Because infant attention to the object that is the target of parent actions and parent attention to the object that is the target of infant actions were uncorrelated in their bivariate correlations, they were treated as independent contributors to joint attention. The beta weights for the paths in this model are given in Figure 5, and indicate that the strongest predictive path to joint attention was from infant hand-eye coordination through parent visual attention to the targets of infant manual actions.

Figure 5.

Beta weights and component t values (df = 50) for a confirmatory path analysis of the proposed relations among the four measures of hand-eye coordination (within each partner and between the partners) and percentage time of the dyad in joint attention.

Discussion

The traditional pathway to joint attention to an object is through gaze following as shown in Figure 1(c). However, gaze is a spatially imprecise and difficult cue for infants, children and even adults to read in contexts in which there are multiple, spatially-near and moving visual targets. Because people coordinate their attention to objects in these more complex contexts, there must be other routes than gaze following. The present results provide evidence for a hand following path to coordinated attention to an object as in Figure 1(a) and (b). By hypothesis, these hand-following routes characterize parent-infant everyday interactions are thus possible sources of individual differences in the development of socially-coordinated attention. Consistent with this larger idea, the present results show that dyad differences in joint attention resided principally in components of the hand-following pathway shown in Figure 1(a), the path in which infants look at their own object manipulations and parents also look at the object handled by their infant. In the following discussion, we consider the implications of hand-following for individual differences in the development of socially coordinated visual attention and how the three pathways in Figure 1 may be developmentally related to each other.

The hand-following paths to joint attention

Our overarching hypothesis is that the sensorimotor coordination of parents and infants as they jointly interact with objects teaches infants how to rapidly read and respond appropriately to social signals, and how to use their own behavior to send signals to their parent. Because hand actions on objects provide precise and readily perceived cues as to the target of interest, hand actions – and attentional responses to the objects on which hands act– may play a critical role in training more precise gaze following (Ullman et al., 2012). By hypothesis, parents who effectively scaffold joint attention with their infants during object play provide the kind of coherent context in which the relevant signals and behavioral responses to those signals are discovered. Thus, parent-infant dyads who for whatever reason have difficulty coordinating attention in object play may put the infant at risk for poorer developmental outcomes. If, as the present results imply, weaker hand-eye coordination on the part of the infant, limits the parent’s ability to effectively scaffold joint attention, then hand-eye dis-coordination could cascade into longer term consequences in social development and language learning. These proposals highlight the importance of infant object manipulation to the development of joint attention and add to the now growing list of domains in which object manipulation appears to be an important component of the developmental pathway (Iverson, 2010; Libertus & Needham, 2011), a list that includes visual object learning (Needham, 2000; Needham, Barrett, & Peterman, 2002; Soska et al., 2010) and understanding others’ intentions (Woodward, 2009).

The present proposal about the role of object manipulation and hand-eye coordination in joint attention is also relevant to the well-documented but not well-understood link between atypical sensorimotor development and atypical social and language development. More specifically, infants at risk for significant delays in social and language development have been reported to show atypical patterns of early sensorimotor development, including delayed and unusual manual interactions with objects (Baranek, 1999; Koterba, Leezenbaum, & Iverson, 2014; Provost, Lopez, & Heimerl, 2007), limited fine motor skills (e.g., Libertus et al,, 2014), discoordination of hands and eyes in prospective reaching (Ekberg, Falck-Ytter, Bölte, Gredebäck, & the EASE Team, 2015), and, perhaps related to the present findings, the exploration of objects with one modality at a time (Kawa & Pisula, 2010). Because social behavior depends on the signals we send through bodily actions (Wolpert, Doya, & Kawato, 2003), atypical sensorimotor behaviors may cause a problematic developmental cascade for optimal social development (Ekberg et al., 2015; Thelen, 2004).

The present results implicate the systematicity with which infants look at their own hand actions on objects as a limiting factor in establishing joint attention. Why do some infants show less coordination between hands and eyes in this context than others at the same age? Motor development is known to show wide variation in the timing of specific achievements (Adolph & Berger, 2007) and thus the observed differences in the typically developing children in present sample could reflect differences in motor development. If this is correct, hand-eye coordination in social tasks should be related to hand-eye coordination in nonsocial tasks, for example, to performance in insertion tasks which are also known to develop markedly during this period (L. B. Smith, 2009). Alternatively, or in addition, the observed individual differences in joint attention and hand-eye coordination may be linked to the development of sustained attention in nonsocial settings (Richards, 1989; Richards & Casey, 1992), which is also related to the infant’s handling of objects (Pereira et al., 2014; Holly A Ruff & Capozzoli, 2003; C. Yu & Smith, 2012). A child who is less distractible and plays longer and more coherently with objects, may provide better cues to their social partner. This suggests a possible developmental relation between sustained attention and socially coordinated attention. Moreover, an additional possibility is suggested by two components of the present findings on the parent side: First, parents’ own hand-eye coordination predicts the proportion of time in joint attention with their infant; and second, high JA dyads were better at resolving the competition between the two hand-following pathways in the case when each partner held an object as they jointly attended more to the objects either held by the parent or by the child. Less coordinated parents (through either genetic or experiential history) are likely to have less coordinated infants and together such parents and infants may have difficulty in providing the sensorimotor cues needed to resolve the competition among the targets. In sum, the present findings by locating one source of dyad differences in joint attention — in infant object manipulation – offer new and testable hypotheses about how the development of socially coordinated attention is supported by – as well as supports – other developmental achievements.

Multiple interacting pathways to joint attention

Newborns have been shown to shift their own gaze to match the direction of an eye movement in the context of a still frontal face (Farroni, Massaccesi, Pividori, & Johnson, 2004). In laboratory experiments with well separated targets and clear social signals, infants as young as 8-month old follow another person’s gaze to an object (Brooks & Meltzoff, 2005). The ability of toddlers to follow the gaze of a partner in laboratory tests strongly predicts developmental outcomes in language learning (Brooks & Meltzoff, 2005; Markus, Mundy, Morales, Delgado, & Yale, 2000; P. Mundy, Sigman, & Kasari, 1990). Results such as these implicate gaze following as a core ability in the social coordination of attention. However, gaze following may be hard to use in spatially complex contexts in which toddlers rarely look at parent faces and but rather look to their hands (Gedeon O. Deák et al., 2014; Yoshida & Smith, 2008; C. Yu & Smith, 2013). In the present study, both high and low JA infants showed this pattern. What then are the relations between hand following and gaze following?

One possibility is that the social coordination of visual attention begins with gaze following in simple contexts, and meanwhile, in more complex situations, hand following is used. This indirect path to shared gaze may scaffold and train joint attention enabling infants over time to better follow both hands and eyes in spatially complex spatial situations. By this line of reasoning, the dyad differences observed in the present study should predict infants’ developing abilities in laboratory tasks to respond appropriately to an experimenter’s signals. Another possibility is that the three pathways shown in Figure 1 (along with potentially other pathways not shown) are not really separable but form a complex system of social coordination in which all the elements co-develop. Face-to-face play in early infancy may set the stage for later hand-following (Libertus & Needham, 2011). Hand following may help tune inferred gaze direction (Frischen et al., 2007) through hand-action cues to turn-taking and gaze shifting (Nyström, Ljunghammar, Rosander, & von Hofsten, 2011; Pereira, Smith, & Yu, 2008). These ideas suggest perhaps systematic but interdependent developmental changes in the prevalent pathways to coordinated social interactions. This idea of multiple but inter-related paths may help explain the not-well-understood shift from so-called dyadic to triadic (or object-centered) social interactions that occurs between 9 and 12 months (Adamson & Bakeman, 1991; De Barbaro, Johnson, & Deák, 2013), a shift that has been linked to the initial decoupling of infants’ own hands and eyes with respect to objects (De Barbaro et al, 2013). Perhaps the low JA infants in the present study – those who also showed lower hand-eye coordination when engaging objects – are showing the developmentally earlier pattern. That is, the initial decoupling of hands and eyes in those infants is a transition point, enabling the infant to shift attention between objects and social partner, but then followed by hand-eye re-coordination in support of organized object play and shared attention to objects within a context of joint action on objects.

In conclusion, using head-mounted eye tracking to record and analyze high-density gaze data during parent-child toy play, we found that joint attention to an object emerged through the coordination of gaze with manual actions on an object. Hand movements to an object if coordinated with eye movements, provide redundant and easy to read information about the object of interest. Dyad differences in joint attention are associated with dyad differences in hand following, with parents’ and infants’ manual activities on objects and with within- and between-partner coordination of hands and eyes. Infants who systematically coordinate their own gaze and hand actions on objects are likely to experience more bouts of joint of attention with their parents, a potentially consequential fact if these infant-parent interactions are the training ground for learning the cues that support smooth social interactions.