Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Jul 1.
Published in final edited form as: Neuropsychology. 2017 May 25;31(5):525–534. doi: 10.1037/neu0000382

The Feasibility of Automated Eye Tracking With the Early Childhood Vigilance Test of Attention in Younger HIV-Exposed Ugandan Children

Michael J Boivin 1,*, Jonathan Weiss 2,#, Ronak Chhaya 3, Victoria Seffren 4, Jorem Awadu 5, Alla Sikorskii 6, Bruno Giordani 7
PMCID: PMC5501490  NIHMSID: NIHMS872597  PMID: 28541084

Abstract

Objective

Tobii eye tracking was compared to webcam-based observer scoring on an animation viewing measure of attention (ECVT) to evaluate the feasibility of automating measurement and scoring. Outcomes from both scoring approaches were compared to the Mullen Scales of Early Learning (MSEL), Color-Object Association Test (COAT), and Behavior Rating Inventory of Executive Function for preschool children (BRIEF-P).

Method

44 children 44 to 65 months of age were evaluated with the ECVT, COAT, MSEL, and BRIEF-P. Tobii X2-30 portable infrared cameras were programmed to monitor pupil direction during the ECVT 6-minute animation, and compared to observer-based PROCODER webcam scoring.

Results

Children watched 78% of the cartoon (Tobii) compared to 67% (webcam scoring), although the two measures were highly correlated (r=0.90,P=0.001). It is possible for two such measures to be highly correlated even if one is consistently higher than the other (Bergemann et al., 2012). Both ECVT Tobii and webcam ECVT measures significantly correlated with COAT immediate recall (r=0.37,P=0.02 versus r=0.38,P=0.01 respectively) and total recall (r=0.33,P=0.06 versus r=0.42,P=0.005) measures. However, neither the Tobii eye tracking nor PROCODER webcam ECVT measures of attention correlated with MSEL composite cognitive performance or BRIEF-P global executive composite.

Conclusion

ECVT scoring using Tobii eye tracking is feasible with at-risk very young African children and consistent with webcam-based scoring approaches in their correspondence to one another and other neurocognitive performance-based measures. By automating measurement and scoring, eye tracking technologies can improve the efficiency and help better standardize ECVT testing of attention in younger children. This holds promise for other neurodevelopmental tests where eye movements, tracking, and gaze length can provide important behavioral markers of neuropsychological and neurodevelopmental processes associated with such tests.

Keywords: HIV, attention, memory, child development, Africa, eye tracking

In evaluating the neurodevelopmental and neuropsychological effects of infectious diseases such as malaria and HIV, it is important to have valid, sensitive, and reliable measures of attention in children who are too young to complete the Tests of Variables of Attention (TOVA) (children less than 5 yrs of age) (Boivin & Giordani, 2009). The Early Childhood Vigilance Test (ECVT) is an experimental measure of vigilance initially validated in children 12 to 44 months of age in order to evaluate sustained attention (Goldman, Shapiro, & Nelson, 2004; Zelinsky, Hughes, Rumsey, Jordan, & Shapiro, 1996). It is modeled after Ruff’s puppet vigilance paradigm (Ruff, Capozzoli, Dubiner, & Parrinello, 1990; Ruff & Capozzoli, 2003) only using lively and colorful animal-like figures which periodically appear in an animation cartoon to actively greet the child viewing the program, providing a means of evaluating attention in a very young child.

The ECVT has proven sensitive to the effects of severe malaria anemia and cerebral malaria in Ugandan children 2 through 5 years of age (Bangirana et al., 2014). It has also been used to evaluate the benefits of early anti-retroviral treatment in Ugandan children with HIV 2 to 5 years of age (Boivin, Bangirana, Nakasujja, et al., 2013), whose caregivers participated in a year-long biweekly training program to improve their daily interactions with their children so as to enhance their child’s development. In this study, the ECVT proved sensitive to the benefits of an early childhood development (ECD) caregiver training program to enhance early cognitive stimulation of children. We witnessed significant improvements in ECVT performance in children in the first 6 months of caregiver training, when compared to a non-ECD intervention arm (Busman, Page, Oka, Giordani, & Boivin, 2013).

In a subsequent study evaluating an ECD intervention in the present study setting (Boivin, Bangirana, Nakasuja, et al., 2013) we administered a caregiver training program based on the Mediational Intervention for Sensitizing Caregivers (MISC) approach (Klein, 1996, 2001; Klein & Boivin, 1997; Klein & Rye, 2004). The ECVT improved significantly after 6 months of biweekly training for the preschool-age children (2 to 5 yrs) of caregivers receiving MISC intervention, as compared to a non-ECD training intervention focusing on children’s health and nutrition. The ECVT was also less varied for study children who were virally suppressed and clinically stable on anti-retroviral treatment, compared to those children not on treatment (Boivin, Bangirana, Nakasuja, et al., 2013) (unpublished data).

The present study is an attempt to evaluate the feasibility of using eye tracking instrumentation with the ECVT. This is in order to enhance the validity and reliability of this test by automating the measurement and scoring of the principal behavioral outcome for this test – which is the proportion of time gazing at the animated video presentation. This ECVT instrumentation substudy is embedded within a randomized controlled trial (RCT) of MISC with caregivers and children exposed to HIV (but not infected) at a study site of caregiver training to enhance early childhood development (ECD) in eastern Uganda (Bass et al., 2016; Murray et al., 2016).

The eye tracking instrumentation, Tobii X2-30, used for the ECVT results, has proven to be used effectively to provide important outcomes for child development brain/behavior indicators in infancy and early childhood (Melinder, Konijnenberg, & Sarfi, 2013; Richmond & Nelson, 2009). However, we know of only one published report where Tobii eye tracking instrumentation has been used to evaluate neurocognitive response with children in the African setting. Forssman and colleagues (2017) presented a visual tracking and visual switching task to a sample of both Finnish (N=39) and Malawian (N=40) nine-month old infants in order to establish the feasibility of this sort of eye-tracking measure in a cross-cultural study (Forssman et al., 2017). Both groups of infants had a very high completion rate for the visual testing procedure while providing valid eye tracking data (> 90%), and parents in both cultures had a high acceptance rate of this kind of evaluation for their children. However, compared with Finnish infants, Malawian infants had a lower rate of successful visual search responses, longer attention visual tracking shift times (indicative of processing speed), and a lower rate of gaze disengagements from the presentation of human faces on the computer monitor.

Forssman and colleagues concluded that in addition to risk factors for brain/behavior development, a number of other reasons could explain the difference, such as a heightened sensitivity of the Malawian infants to stressful environments that could disrupt their social responsiveness, or responsiveness to computer monitor displays. Irrespective, Forssman and colleagues were able to demonstrate that the use of eye tracking technologies in infant cognitive testing could yield a consistent pattern of responses for infants cross-culturally, and that such technology was feasible and acceptable in the African context (i.e., proof of concept). The present study also seeks to establish the feasibility of eye tracking as part of a neurocognitive assessment in African context. Furthermore, this is the first time that this kind of eye tracking instrumentation and technology is being used in a LMIC to automate the administration and measurement a neurocognitive behavioral outcome in preschool-age children perinatally exposed to HIV.

Method

Participants

IRB approval for this study was obtained from the Makerere University School of Medicine Ethics in Research IRB and the Michigan State University Biomedical Research IRB. Written informed consent was provided by the mothers of the study children after the consent form was explained to the mothers in their local language. Research permission was issued by the Ugandan National Council for Science and Technology. Participants were 44 children (24 boys, 20 girls) 44 to 65 months of age (M=4.36, SD=0.98 yrs) perinatally exposed to HIV but uninfected (ELISA test) and cared for by the biological mother, staying within a 20 km catchment area of Tororo town. These dyads were referred to the parent MISC RCT study by the Infectious Disease Research Collaboration (IDRC) at Tororo District Hospital, which primarily supports NIH-sponsored research study collaborations between Makerere University and the University of California – San Francisco (Kamya et al., 2014).

The present study children were from the final 44 dyads (out of 200) scheduled for their 2 year post-enrollment assessment as part of the MISC caregiver training RCT parent study described in more detail elsewhere (Bass et al., 2016). These 44 children completed the assessment protocol as approved in the RCT (clinicaltrials.gov identifier: NCT01640561). The two trial arms in this RCT compared biweekly year-long MISC versus a nutrition/health/hygiene curriculum promoted in Uganda for impoverished rural families. Children were cared for by birth mothers with HIV at the time of the child’s birth, but the child was confirmed uninfected at the time of enrollment with ELISA testing. Because mild to moderate risk factors such as birth complications, malnutrition, other known mild head injury or concussion requiring hospitalization would also have an impact on attention, we excluded any children with any history of hospitalization. The 44 children’s final assessment was conducted as specified within this RCT in obtaining the final developmental endpoint for study children, only with the exception of having the Tobii eye tracking instrumentation included as part of the child’s ECVT test. This instrumentation is described further below.

A child was excluded from the RCT MISC caregiver training study and from the present substudy if he/she had a medical history of serious birth complications, severe malnutrition, bacterial meningitis, encephalitis, cerebral malaria, or other known brain injury or disorder requiring hospitalization which could overshadow the developmental benefits of MISC caregiver training. A clinical medical officer completed a health history questionnaire with the mother for information on any previous hospitalizations, along with a review of the medical health records available for each child referred from the IDRC at Tororo District Hospital. The medical officer then performed a physical examination along with the Ten Question Questionnaire (TQQ) with the principal caregiver to screen for neurodisability (Durkin, Hasan, & Hasan, 1995). The height (range 79 to 115 cm; M=101, SD=8.56) and weight (range 8 to 20 kg; M=15.34, SD=2.80) were well within the normal range reported for the total sample of HIV exposed children in the RCT study (Judith K. Bass et al., 2016), as was the quality of home environment based on the modified Caldwell Home Observation for Measurement of the Environment (HOME) scale (Caldwell & Bradley, 1979) (range 12 to 28; M=22.4, SD=3.63) used previously with HIV-exposed children in this setting (Boivin, Bangirana, Nakasuja, et al., 2013). Likewise, maternal depression as measured with the Hopkins Symptoms Checklist (Boivin, Bangirana, Nakasujja, et al., 2013; Familiar, Murray, et al., 2016a; Familiar, Nakasujja, et al., 2016) for the present sample of dyads was not exceptional (item average for the 15 depression items; M=0.79, SD=0.53).

Neurodevelopmental Measures

After a brief medical exam verified that the child was in good health and could proceed with the assessment, we administered the tests in the order described below, starting with the ECVT. While the child was being tested with the ECVT, MSEL, and COAT, the mother completed the BRIEF-P for the study child, with the questions being read to her in her local language. All of the assessments described below had already been adapted to the rural Ugandan context with children infected with HIV (Boivin, Bangirana, Nakasujja, et al., 2013a) as well as those perinatally exposed to HIV (not infected) and evaluated at the present study site (Boivin, Bangirana, Nakasuja, et al., 2013b).

Early Childhood Vigilance Test (ECVT)

The ECVT was our principal measure of childhood attention. Initially validated for American children 12- to 46-months of age, this measure evaluates how vigilance is used in very young children to evaluate sustained attention (Goldman et al., 2004; Zelinsky et al., 1996). The age range for the initial validation of the ECVT does not preclude its effective use with Ugandan severe malaria survivors up through five years of age, or with HIV-exposed children up to 65 months of age in the present study (Bangirana et al., 2014; Bass et al., 2016). Children are required to monitor a colorful computer screen for the sudden appearance of active and engaging cartoon animal characters that appear unpredictably at 5-to-15 second intervals. The ECVT involves a six minute and forty-four second cartoon in which a child watches an animated video, with animal characters moving across the screen for about 6 minutes of the total animation time. The more vigilant the children are in monitoring the screen, the more likely they are to be rewarded with cartoons engaging in lively movements when they appear for about 10 seconds.

The child was videotaped with a webcam video placed above the computer monitor (see Figure 1). The videos were coded and scored by a trained researcher using a software program called PROCODER. Coding videos involved tracking the amount of time in seconds the child was definitely looking at the screen and turning that amount into a percentage representing the amount of time spent attending to the screen out of the total time of the animation. The same research assistant completed the PROCODER webcam ECVT scoring for the study children, and this research assistant responsible for this scoring had extensive experience, completing hundreds of previous ECVT tests subject to inter-rater reliability checks at the study site with 10 percent of completed ECVT tests.

Figure 1.

Figure 1

Open in a new tab

These photographs show the screen viewing and seating arrangement for children in the present study viewing the Early Childhood Vigilance Test (ECVT), a six minute and 44 second cartoon animation test in which colorful and active animals would appear and would intermittently great the child (see pink bunny above) to gain the child’s attention to the screen. The web cam is mounted at the top of the screen and the Tobii X2-300 infrared camera strip to record pupillary gaze for eye tracking is situated in a strip below the screen.

PROCODER webcam inter-observer reliability ratings for the ECVT are not only time consuming, but can have disappointing outcomes even in highly experienced observers. For our present study sample, we did have two of our principal Ugandan ECVT testing research assistants at the present study site code the same 10 videos. Time and cost precluded doing all 44 children. Both of these evaluators had coded scores of ECVT videos previously in the same population of HIV-exposed children. We had them code each ECVT videos using PROCODER three times: proportion of time definitely not looking at the screen, proportion of time definitely looking, and proportion of time where the gaze direction was ambiguous. The resulting reliability between raters was very low for all three measures (Definitely not looking r= 0.05; Definitely looking r=0.10; Ambiguous r= 0.28). Therefore, for the present study PROCODER data was only coded once instead of having multiple coders do the same video. The consistency of having the same coder evaluate all of the ECVT webcam videos removes variability that might result from multiple coders doing the scoring, perhaps resulting in a more optimal comparison between PROCODER webcam-based ECVT measures and the corresponding Tobii eye tracking measures.

Besides correlational reliability analyses, another useful approach to agreement when there are only two raters is to calculate the differences between each pair of the raters’ observations. The mean of the difference can be interpreted as the bias and the inter-confidence (IC) 95% interval is termed “limits of agreement.” If raters tend to agree, the difference between observations will be close to zero. Results from this analysis showed that there are systematic measurement differences in both looking and not looking times (both average differences are different than zero). In 95% of paired observations, the difference was within 150 secs for both “definitely looking” and “definitely not looking” time measures. For looking time, testers tended to disagree more as looking time increases; while the opposite is observed in “definitely not looking” measures. Therefore, while the absence of multiple PROCODER raters in our present study might be viewed as a study limitation, the poor inter-observer reliability ratings that we obtained with some of the children in our present sample further supports the advantage of Tobii eye tracking in measuring the proportion of screen viewing time for the ECVT.

Tobii Professional Studio Eye Tracking Programming

The other ECVT performance measures were obtained automatically using a Tobii X2-30 portable infrared camera calibrated to track eye gaze by means of pupillary direction. Studio Professional software was programmed with the Tobii X2-30 portable infrared camera to monitor the child’s pupil direction during the cartoon to automatically calculate the proportion of time the child gazed at the monitor screen during the entire animation presentation. The Tobii system was also programmed to measure the proportion of time watching the screen only when colorful animals were moving across it.

After successful calibration of pupillary gaze direction at the start of each testing session (using the Tobii Studio Professional calibration instrumentation verification program), the animation cartoon was started and total time spent looking at the cartoon animation was automatically measured using the following program features. An area of interest (AOI) is a pre-determined area, or zone, of the screen that is of interest. In this study, for example, an AOI would be the animated animals as they moved across the screen as well as the entire screen in general during the video cartoon. AOIs are set in order to determine total gaze length on that area throughout the test.

A fixation is defined as the time that both eyes are statically locked (i.e. not perceptibly moving) on an area of the screen. Fixation duration measures are the sum of fixation time while looking within the AOI. This measure does not include the time it takes for the eyes to move between points A and B on the computer screen for the animation presentation.

First-visit duration was defined as the interval of time between the first fixation within the AOI and the next fixation outside of the AOI (not gazing at the animation screen). Total visit duration measures the duration of each individual visit to the total viewing screen for the animation within an AOI. This measure includes the time it takes for the eyes to move between point A and point B as the animal character moved across the screen left to right or right to left.

Our principal measures in the present study were ECVT PROCODER scored webcam video measures of total time watching the screen during the animation, and ECVT Tobii eye tracking automated measures of proportion of time watching the screen during the animation. For the ECVT Tobii eye tracking, we also used a 2nd outcome measure, that of total proportion of time viewing the animation only when animals were moving across the animation screen. This particular measure was not feasible with the PROCODER webcam scoring, but only with Tobii eye tracking. We included this measure as well because of the possibility that it might potentially be a worthwhile additional measure of attention with the ECVT. This is because this measure would gauge the ability of the child to monitor and track the moving animal on a more continual basis, without the more salient greeting features of the pink rabbit which are interspersed during the animation presentation (Figure 1).

Mullen Scales of Early Learning

This assessment is intended for use in assessing children from birth to 68 months and provides assessment scores for the developmental domains of Gross Motor (up to 36 months), Visual Reception, Fine Motor, Receptive Language, and Expressive Language (Mullen, 1995). The Early Learning Composite provides a measure of g, the general measure of fluid intelligence thought to underlie cognitive ability in general (up to 60 months). It is derived from the standardized T scores of the four Mullen cognitive scales, all of which have memory and learning items as part of the assessment (Visual Reception, Fine Motor, Receptive Language, & Expressive Language). In validation studies, the Early Learning Composite has a correlation coefficient of 0.70 with the Bayley Mental Development Index measure (Bradley-Johnson, 2001).

Color Object Association Test (COAT)

The COAT was our principal assessment of childhood memory (Jordan, Johnson, Hughes, & Shapiro, 2008). The COAT is one of the few existing methods to evaluate declarative (explicit) memory in children during the toddler and preschool years (age 2 to 5 years). Using 4-inch square wooden boxes upon which a colored felt patch can be attached with Velcro, novel “space creature” pictures are presented and given a nonsensical name. The creature pictures are attached to the outside of the wooden boxes, and the child is trained to “select” that picture/box by placing small, colorful familiar toys with handles (e.g., ball, doll, book) inside the box. The child is then asked to pick that picture from a group of similar pictures when given its name, by placing the toy inside the box to which that picture is attached. As such, this is a more active memory task that provides for more than just a “pointing” response. Boxes are rotated after each trial to discourage “position-based” responding, and neither the color of the boxes or the names of the toys are ever presented to the child so as not to encourage or foster verbal language mediators as part of the memory process.

The number of pictures increases at a rate of one per trial with a new picture presented at the start of each trial. This is considered a long-term recall (learning) measure because the child’s recall on all the items is scored, including the ones presented on previous trials. The principal outcome measures from the test are the immediate memory score of number of recalled items following each trial, and an overall total recall or learning score of correctly placed items presented on previous trials.

Behavior Rating Inventory of Executive Function–Preschool (BRIEF-P)

The BRIEF-P inventory (Gioia, Espy, & Isquith, 2003) is useful for evaluating and planning treatment strategies for a wide spectrum of developmental and acquired neurological conditions, including learning disabilities, low birth weight, ADHD, Tourette’s disorder, traumatic brain injury, and autism. The BRIEF-P scales include the behavior/cognitive functions of Inhibit, Shift, Emotional Control, Initiate, Working Memory, Plan/Organize, Organization of Materials, and Monitor. A global executive composite (GEC) score was obtained from this questionnaire to the caregiver, with higher scores indicating more symptoms of executive function problems. The GEC was our principal outcome measures from this assessment. BRIEF-P items were read to the caregiver in her local language.

Statistical analysis

The distributions of and correlations among the principal measures of the developmental test (MSEL, COAT, ECVT Tobii eye tracking, ECVT PROCODER, BRIEF-P) were evaluated. The associations between the ECVT Tobii eye tracking, ECVT PROCODER were also depicted using scatterplots. Analyses were completed using SAS 9.4.

Results

In terms of the normative mean for the Mullen Scales of Early Learning (M=100, SD=15), the present study sample of children exposed to HIV were mostly in the lower quartile for the Mullen Scales of Early Learning composite cognitive standardized score (range 49 to 90, M=68.39, SD=9.87). In terms of behavior symptoms for executive function difficulties, BRIEF-P scores for standard scores (normative M=50; SD=10) were mostly in the middle quartiles (range 37 to 80; M=53.0, SD=12.06).

Children watched 78% of the cartoon as automatically measured using Tobii eye tracking software and instrumentation, while PROCODER webcam scoring resulted in an average of 67% (r=0.90, P<0.001) (see Tables 1 and 2). Age of children was significantly correlated with both Tobii and PROCODER measures of ECVT attention (Table 2). ECVT Tobii eye tracking scores for the animation portion with active moving animals across the screen also strongly correlated with webcam PROCODER-based scoring by observers (r(41)= 0.84, p<0.001) (Table 2).

Table 1.

Descriptive statistics of the Early Childhood Vigilance Test (ECVT) proportion of time viewing the screen with the animation (Tobii automated scoring and PROCODER Webcam observer scoring); and the proportion of time viewing the animation when an animal character was moving across the screen (Tobii automated scoring only. N denotes the number of children for which this ECVT measure was within a valid range. When missing for the Tobii, this was usually because of inability of the infrared camera to track pupillary gaze throughout the test.

Early Childhood Vigilance Test (ECVT) Performance Measures N Mean Std Dev Minimum Score Maximum Score
Tobii Eye Tracking Proportion of Total Time Viewing Screen During Animation 42# 0.78 0.14 0.29 0.99
Tobii Eye Tracking Proportion of Total Time Viewing Screen During Appearance of Moving Animals in Animation 41# 0.60 1.71 0.00 0.90
PROCODER webcam scoring of Proportion of Total Time Viewing Screen During Animation 44 0.67 0.18 0.09 0.97
Mullen Scales of Early Learning (MSEL) Composite Standard Score of Cognitive Ability 44 68.39 9.87 49 90
Behavior Rating Inventory of Executive Function Preschool (BRIEF-P); Global Executive Composite Standard Score 42 53.00 12.06 37 80
Open in a new tab

Table 2.

This table contains the Pearson product-moment correlation coefficients and below that the corresponding probability (P) value between the Early Childhood Vigilance Test (ECVT) Tobii eye tracking measures (total proportion viewing animation screen and proportion of time viewing moving animals) and the ECVT observer PROCODER scored webcam video of that session (total proportion time viewing animation screen), with the child’s age, the Mullen Scales of Early Learning (MSEL) standardized composite score for cognitive performance, the Color-Object Association Test (COAT) total recall (learning) and immediate recall (memory) measures, and the Behavior Rating Inventory of Executive Function (Preschool) rating of symptoms by the principal caregiver.

Early Childhood Vigilance Test (ECVT) Performance Measures Age of Child ECVT PROCODER Proportion MSEL Composite Total COAT Total Recall (Learning) COAT Immediate Memory BRIEF-P Global Index
Tobii Eye Tracking Proportion Screen Viewing of Animation r = 0.41
P=0.008 		r = 0.90
P<.0001 		r = −0.047
P = 0.77 		r=0.33
P=0.06 		r = 0.37
P = 0.02 		r = −0.18
P = 0.27
Tobii Eye Tracking Proportion of Screen Viewing of Moving Animals r = 0.25
P = 0.12 		r = 0.84
P < .0001 		r = 0.24
P = 0.13 		r=0.28
P=0.07 		r = 0.33
P = 0.03 		r = 0.02
P = 0.88
PROCODER webcam scoring of Proportion Viewing of Animation r = 0.43
P = 0.003 		----- r = 0.03
P = 0.83 		r = 0.42
P = 0.005 		r = 0.38
P = 0.01 		r = −0.23
P = 0.14
Open in a new tab

Tobii eye tracking significantly correlated with COAT immediate recall for color-object placement associations (tracking moving animals (r(41) =0.33, P=0.03; total percent time screen gaze r(42)=0.37, P=0.02) (Table 2). ECVT scored by webcam PROCODER % was also significantly correlated with COAT immediate memory (r(44)=0.38, P=0.01) and learning (r(44)=0.42, P=0.005) (Table 2). However, the Tobii ECVT measures did not quite correlate significantly with COAT learning.

None of the ECVT measures were significantly correlated with MSEL composite total score, or with the BRIEF-P global index (Table 2). Among the MSEL scale measures, the ECVT when scored by Tobii eye tracking (total proportion time looking at cartoon) was significantly correlated with MSEL Fine Motor performance (visual-spatial learning with motor response; r(42)=0.33, P=0.037).

Discussion

The purpose of the present project is to evaluate the feasibility of automated eye tracking technology with the ECVT as a measure of attention with younger Ugandan children. Based on our present findings when compared to the ECVT PROCODER video scoring measures, we conclude that Tobii eye tracking technology is feasible and provides for outcome measures that are consistent with the use of observer-based scoring of webcam videos for this attention test. However, the present study was embedded within a larger clinical study of caregiver training with HIV-exposed (non-infected) children, so this evaluation of eye tracking feasibility was with a special population of children. Irrespective, the use of Tobii eye-tracking technology has the additional advantage of providing for an automated means of administration and measurement for the ECVT. Tobii eye tracking with its automated measures can also provide immediate results for the ECVT test, as opposed to the need for PROCODER webcam scoring at a later date by well-trained observers, assuming that the webcam-based measures have good inter-observer reliability as a result of good observer training and supervision.

In our present study, overall PROCODER scores for proportion of time viewing the ECVT cartoon were lower than for the Tobii measures. It is possible that our single PROCODER evaluator took a more conservative approach as to what constituted a child “definitely” looking at the monitor screen during the scoring of the webcam. This would result in a consistently lower percent of time that children were watching the ECVT test when compared with the Tobii data, even though the two measures were still highly correlated. It would be worthwhile to recode the webcam videos with an alternative coder to see if similar results are obtained. Children sometimes had difficulty sitting still in the chair, which resulted in improper calibrations between the subjects and the Tobii device. Additionally, multiple power outages in Tororo resulted in data loss. For several of the study children, some of the Tobii eye tracking data was not readable due to the loss of pupil tracking calibration by the Tobii camera during the session. None of the webcam videos were lost or unreadable in the present sample.

As did Forssman et al (2017) with Malawian infants using Tobii eye tacking instrumentation and a cognitive visual tracking task, we also established the feasibility of eye tracking measures with children in a medical clinic serving a disadvantaged rural Ugandan sample of at-risk children. We also established ways in which of ECVT eye tracking measures correlated to other cognitive and neurodevelopmental assessments in our study children. In a recent review of a variety of pediatric neurodevelopmental and neuropsychological assessments used in low and middle income countries (LMICs), the authors noted that many of the assessments adapted for use have not been well validated in these contexts (Semrud-Clikeman et al., 2016). Furthermore, in order to administer these assessments in a valid and reliable matter, significant training and monitoring of the assessment personnel is necessary. This is especially the case for test measures which depend on observer evaluation of a child’s behavior, even if preserved within a video record. Because of this, observer variability in evaluating video-based scoring of behavioral outcomes has proven to be a serious limitation for such measures (Dickerson, Gerhardstein, Zack, & Barr, 2013).

To illustrate, the Fagan Test of Infant Intelligence measures gaze length of infants for photographic presentations of familiar and unfamiliar human faces. This is in order to assess working memory as an indicator of neurocognitive development (Fagan & Detterman, 1992). Since the pioneering work for Fagan and colleagues with the FTII in the 1980s and 1990s in high-income countries, the sensitivity of this test to neurodevelopmental risk and disorders in infants and very young children has been further established with outcomes such as premature birth and low birth weight infants (Guzzetta et al., 2006), malnutrition and developmental delay (Nelson, Goldenberg, Hoffman, & Cliver, 1997), polyunsaturated fat oils and other micronutrient interventions for under-nutrition (vitamin A, iron) (O’Connor et al., 2001), environmental toxic exposures (e.g., methyl-mercury, lead, PCBs) (Davidson et al., 1999; Emory, Ansari, Pattillo, Archibold, & Chevalier, 2003; Jedrychowski et al., 2008; Myers et al., 1995), congenital and early childhood diseases (e.g., Rett’s Syndrome) (von Tetzchner et al., 1996), and gestational exposure to cocaine and other maternal drug and alcohol use (Chiriboga, Kuhn, & Wasserman, 2007; Jacobson, Chiodo, Sokol, & Jacobson, 2002).

However, the FTII was not sensitive to the neurodevelopmental effects of perinatal infection of Ugandan infants with HIV in comparison to the Bayley Scales of Infant Development (Drotar et al., 1997). This is perhaps because of the challenges in reliably measuring gaze length in infants by human observers using stop watches while peering through observation slits from behind the photographs of the human faces presented to the infants in a hectic clinic setting. Therefore, more consideration should be given to automating such measures whenever possible, particularly in ways which will help standardize the administration and scoring of such tests across cultural contexts.

Due to the current HIV epidemic in sub-Saharan Africa, multiple children are at risk of compromised caregiving. The ECVT test of attention is one of the neurocognitive assessments that is used to measure the efficacy of the interventions implemented to improve caregiving. Eye tracking technology is used in multiple types of research. This is the first time that we know of that this technology has been used to evaluate children exposed to HIV in sub-Saharan Africa. Our principal accomplishment for this study was establishing the feasibility of using eye tracking technology in a resource-constrained clinic setting in rural Africa as part of a developmental assessment in children. Eye tracking technology can help standardize the ECVT scoring process and eliminate issues surrounding inter-rater reliability. The Tobii eye tracking system also allows the clinician or researcher to look at specific areas of interest (AOI) in order to dynamically track eye movement across the screen. The use of areas AOIs can provide us with a more precise measurement of childhood attention. Such measures are not available with webcam-based scoring for gaze fixation and direction because of the difficulty of making such determinations in a precise manner with human observers.

Study Limitations

Using only the same evaluator for the PROCODER scoring rather than a composite of multiple coders could be seen as a study limitation. However, as noted in our methodology for the PROCODER evaluation, poor inter-coder reliability in a pilot study subsample of 10 children was the principal reason we went with only one coder (our most experienced one). This was in addition to the time and resources needed for multiple ECVT coders for the entire sample, given the resources available for the present “proof of concept” eye tracking study.

Another study limitation was that it was completed with a special population of children—those perinatally exposed to HIV. Also, while these subjects are all primary aged children, 44 to 65 months represents a period of very dynamic developmental growth and the allocation of attentional resources changes substantially with age. Many health, nutritional, socio-economic, and caregiving risk factors among these children can contribute to variation in measures of eye tracking and scanning used to evaluate attention processes. Further evaluation of the feasibility of an eye-tracking technology in neurocognitive tests with younger African children should include children both at-risk and not at-risk from exposure to infections such as HIV. They should also include a sufficiently large sample to allow for an adequate evaluation of the validity of the eye-tracking methodology measures for attention measures for children through a broader age span in our study population with ECD intervention (1.5 to 5.5 years). For example, Forrsman et al. (2016) infant eye tracking study with cognitive search tasks, the Malawian infants had lower completion rates and slower processing speeds than did the Finnish children, presumably because of a greater prevalence of developmental risk factors such as poor nutrition and anemia. However, their study was underpowered to directly evaluate the relationship of such factors with their eye tracking performance measures in the Malawian children (Forssman et al., 2017).

Attention in children is associated with basic demographic factors such as the complex associations between poverty, living status, education, race/ethnicity and health status/outcomes (Boivin & Giordani, 2009). Furthermore, socioeconomically disadvantaged individuals tend to experience less linguistic, social and cognitive stimulation and these can also affect brain/behavior development as it pertains to attention. Individuals from lower SES homes report more stressful events during their lifetime and their biological response to stressors has been hypothesized as one of the underlying mechanisms for health and cognitive disparities in relation to SES (Noble et al., 2015; Noble, Houston, Kan, & Sowell, 2012). In turn, these suboptimal sociocultural or psychosocial experiences likely have specific downstream effects on particular brain structures (Boivin & Giordani, 2009). Likewise, risk factors prevalent for mothers with HIV such as vulnerability to malaria and other infections during pregnancy, exposures to potentially toxic effects from ante- and post-natal exposure to combination anti-retroviral medications, birth complications or prematurity for the child, and early cessation of breast feeding with subsequent risk for poor nutrition and diarrheal disease, can all affect early brain development as it pertains to foundational neurocognitive processes such as attention (Brahmbhatt et al., 2014; Brahmbhatt et al., 2006; Brahmbhatt et al., 2003; Brahmbhatt et al., 2008).

Given the above important considerations in the neuropsychological development of attention processes in resource-constrained cross-cultural settings, another limitation of our study is that it was not designed to identify the etiology of the cognitive impairment found in this population. It is likely that maternal/caregiver poor physical health from HIV infection along with emotional distress (e.g., depression), lower caregiver education attainment and the stress of impoverishment can all affect the quality of caregiving (Familiar, Murray, et al., 2016a). These factors can modify how attention processes should best be measured in HEU children such as our present sample (Bass et al., 2016; Familiar, Murray, et al., 2016; Familiar, Nakasujja, et al., 2016). Thus, another limitation of this study is that it could not adequately gauge the degree of attention deficits as we how it pertained to our other cognitive ability and development indicators (e.g., COAT memory and learning, MSEL cognitive development) as they related to the Tobii and webcam ECVT measures.

Although we did not adjust for the above potential modifying factors in our correlational analyses for the Tobii eye tracking and PROCODER webcam measures (Table 2), our present study sample was fairly homogenous on these demographic factors. Therefore, our present study findings are reasonable given our principal study aims in this feasibility evaluation of Tobii eye tracking measures for the ECVT in our study sample of Ugandan HEU children.

Conclusions

We established that eye tracking technology is feasible in automating the measurement of attention in a public hospital clinical setting in sub-Sahara Africa. In doing so, in can provide for behavioral measures dependent on eye gaze length and direction that can be better standardized and obtained in a more efficient manner. Measures of attention correspond well to measures of working memory based on object placement (color-object association test), but not as well on more general tests of children development (Mullen Scales of Early Learning).

Ours is the first study to establish the feasibility of the use of eye tracking technology to measure attention in HIV-exposed very young children in sub-Sahara Africa. We have shown that neurocognitive assessments are feasible. However, presently such eye tracking technologies are not accessible in most resource-constrained settings in low and middle income countries (LMICs) without a high level of supportive expertise. This is because there are a number of methodological challenges in terms of the setup, calibration, stable electricity, and the ability to minimize fluctuation in lighting, camera positioning, and movement. Equipment cost and technical training for the operation and programming of eye tracking devices is also a major constraint in making this technology readily available in LMICs. Finally, in a few cases, Tobii ECVT measures were not available due to a loss of pupillary calibration during the cartoon presentation and viewing (Table 1). However, valid Tobii ECVT measures was obtained for “total proportion of time viewing the screen” in 42 of 44 cases, denoting a 95% success rate (Table 1). This failure rate is comparable to what we have obtained in webcam-based ECVT recordings in other studies in this setting (Boivin et al., 2013; unpublished data).

The present feasibility pilot study is a first step towards the goal that one day eye tracking technology can be used to enhance other behavioral measures of early child development pertaining to cognition. It is presently difficult to see utility of this “proof of concept” feasibility study, given the limited accessibility of the computer, eye tracking instrumentation, and technical expertise needed to implement such measures in LMICs. However, mobile network coverage is available presently for all of Uganda, making internet-based mHealth technologies far more accessible than previously thought possible, even for tele-ophthalmological evaluation (Betjeman, Soghoian, & Foran, 2013; Perez, Swart, Munyenyembe, & Saranchuk, 2014). To illustrate, eye tracking technologies such as infrared reflectance oculography have been used to evaluate the long-term visuomotor development effects in Malawian children surviving cerebral malaria.

Likewise, eye tracking has been used to help map cognitive ability performance in very young Malawian children (Forssman et al., 2017), and we have done the same in a rural Ugandan hospital using a modified Fagan test (Boivin, unpublished data). In time, such it is possible to see eye tracking instrumentation interfaced to a mobile network platform in mHealth applications that makes such assessments readily available for community health workers to evaluate, for example, multi-step attention vigilance and monitoring tasks, visual-spatial working memory, maze learning and navigation tasks, reasoning by analogy with progressive matrices tests, and the processing of visual information as a part of literacy training in children. In conclusion, eye tracking technology can revolutionize mHealth in the global context, providing a wealth of behavioral data to help clinicians and researchers better interpret the neuropsychological performance measures of such task in children in response to developmental factors of risk and resilience.

The accessibility of such data through new technologies (e.g., eye tracking, neurocognitive assessment) will allow neuropsychologists to more readily incorporate LMIC pediatric evaluative data into child development brain/behavior science. The infusion of such diverse and massive sources of population-based surveillance and performance-based data, made possible by linking sophisticated instrumentation measurement to mobile/internet platforms will revolutionize the existing paradigm of child development science. This is because such mHealth innovations have the potential to drive the future of evidence-based public health child development programs on a global scale never before seen in human history (Boivin, Kakooza, Warf, Davidson, & Grigorenko, 2015).

Figure 2.

Figure 2

Open in a new tab

This scatterplot depicts the relationship with linear least-squares regression line (formula in the middle of the graph) between the Tobii Eye Tracking proportion of time gazing at the screen during the entire animation with the observer PROCODER scoring of the webcam videos (horizontal axes). The R squared value denotes the proportion of variance in the Tobii eye tracking measure explained by the webcam PROCODER measure.

PUBLIC SIGNIFICANCE STATEMENT.

This is the first study to establish the feasibility of the use of eye tracking technology to measure attention in HIV-exposed very young children in a sub-Sahara Africa. These pioneering efforts show how ever-more accessible eye tracking technologies can give rise to more efficient ways to measure behavior concomitants of neuropsychological integrity in younger children. This can improve the standardization of such measures in gauging the neurodevelopmental impact of infectious disease impacting on brain development, and the benefits of prevention and/or early intervention for such diseases.

Acknowledgments

Dr. Itziar Familiar-Lopez provided scientific oversight for the study site and support for the student research internship program; Julius Caesar Ojuka (on-site study coordinator) led the field team responsible for the intervention training of the testers, caregiver and child assessments, and translation into the local languages, along with providing ongoing clinical care. George Okwakol, Claire Apayi, Moses Ejula, Sylvia Adongo, and Anthony Ekisa served as the research assistants for this study, which would not have been possible without their efforts. Their efforts are greatly appreciated. The authors also acknowledgement the extraordinary efforts of the anonymous reviewers to make this a much better manuscript.

Funding: This work was supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development at the National Institutes of Health (Grant number RO1 HD070723) to A.S., M.J.B., and B.G.. The Michigan State University College of Human Medicine supported J.W. and R.C. with summer global health research stipends, and the University of Michigan School of Public Health supported V.S.. The study sponsors had no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.

Footnotes

Conflict of Interest: None of the authors have any conflicts of interest or financial disclosures pertaining to this study.

Contributor Information

Michael J. Boivin, Michigan State University Department of Psychiatry and of Neurology & Ophthalmology, University of Michigan Department of Psychiatry.

Jonathan Weiss, Michigan State University College of Human Medicine.

Ronak Chhaya, Michigan State University College of Human Medicine

Victoria Seffren, University of Michigan School of Public Health.

Jorem Awadu, Michigan State University College of Education.

Alla Sikorskii, Michigan State University Department of Statistics and Probability.

Bruno Giordani, University of Michigan Department of Psychiatry, Psychology, and Nursing.

Reference List

  1. Bangirana P, Opoka RO, Boivin MJ, Idro R, Hodges JS, Romero RA, John CC. Severe Malarial Anemia is Associated With Long-term Neurocognitive Impairment. Clin Infect Dis. 2014;59(3):336–344. doi: 10.1093/cid/ciu293. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bass JK, Nakasujja N, Familiar-Lopez I, Sikorskii A, Murray SM, Opoka R, Boivin MJ. Association of caregiver quality of care with neurocognitive outcomes in HIV-affected children aged 2–5 years in Uganda. AIDS Care. 2016;28(Suppl 1):76–83. doi: 10.1080/09540121.2016.1146215. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bass JK, Nakasujja N, Familiar-Lopez I, Sikorskii A, Murray SM, Opoka R, Boivin MJ. Association of caregiver quality of care with neurocognitive outcomes in HIV-affected children aged 2–5 years in Uganda. AIDS Care. 2016:1–8. doi: 10.1080/09540121.2016.1146215. http://dx.doi.org/10.1080/09540121.2016.1146215. [DOI] [PMC free article] [PubMed]
  4. Bergemann TL, Bangirana P, Boivin MJ, Connett JE, Giordani BJ, John CC. Statistical approaches to assess the effects of disease on neurocognitive function over time. J Biomet Biostat. 2012:S7:016. doi: 10.4172/2155-6180.S7-016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Betjeman TJ, Soghoian SE, Foran MP. mHealth in Sub-Saharan Africa. Int J Telemed Appl. 2013;2013:482324. doi: 10.1155/2013/482324. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Boivin MJ, Bangirana P, Nakasuja N, Page CF, Shohet C, Givon D, Klein PS. A year-long caregiver training program to improve neurocognition in preschool Ugandan HIV-exposed children. J Dev Behav Pediatr. 2013b;34(2):269–278. doi: 10.1097/DBP.0b013e318285fba9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Boivin MJ, Bangirana P, Nakasujja N, Page CF, Shohet C, Givon D, Klein PS. A year-long caregiver training program improves cognition in preschool Ugandan children with human immunodeficiency virus. J Pediatr. 2013a;163:1409–1416. doi: 10.1016/j.jpeds.2013.06.055. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Boivin MJ, Giordani B. Neuropsychological assessment of African children: evidence for a universal basis to cognitive ability. In: Chiao JY, editor. Cultural Neuroscience: Cultural Influences on Brain Function. Vol. 178. New York, NY: Elsevier Publications; 2009. pp. 113–135. [DOI] [PubMed] [Google Scholar]
  9. Boivin MJ, Kakooza AM, Warf BC, Davidson LL, Grigorenko EL. Reducing neurodevelopmental disorders and disability through research and interventions. Nature. 2015;527(7578):S155–160. doi: 10.1038/nature16029. [DOI] [PubMed] [Google Scholar]
  10. Bradley-Johnson S. Cognitive assessment for the youngest children: A critical review of tests. Journal of Psychoeducational Assessment. 2001;19(1):19–44. [Google Scholar]
  11. Brahmbhatt H, Boivin M, Ssempijja V, Kigozi G, Kagaayi J, Serwadda D, Gray RH. Neurodevelopmental benefits of antiretroviral therapy in Ugandan children aged 0–6 years with HIV. J Acquir Immune Defic Syndr. 2014;67(3):316–322. doi: 10.1097/QAI.0000000000000295. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Brahmbhatt H, Kigozi G, Wabwire-Mangen F, Serwadda D, Lutalo T, Nalugoda F, Gray R. Mortality in HIV-infected and uninfected children of HIV-infected and uninfected mothers in rural Uganda. J Acquir Immune Defic Syndr. 2006;41(4):504–508. doi: 10.1097/01.qai.0000188122.15493.0a. [DOI] [PubMed] [Google Scholar]
  13. Brahmbhatt H, Kigozi G, Wabwire-Mangen F, Serwadda D, Sewankambo N, Lutalo T, Gray R. The effects of placental malaria on mother-to-child HIV transmission in Rakai, Uganda. Aids. 2003;17(17):2539–2541. doi: 10.1097/00002030-200311210-00020. [DOI] [PubMed] [Google Scholar]
  14. Brahmbhatt H, Sullivan D, Kigozi G, Askin F, Wabwire-Mangenm F, Serwadda D, Gray R. Association of HIV and malaria with mother-to-child transmission, birth outcomes, and child mortality. J Acquir Immune Defic Syndr. 2008;47(4):472–476. doi: 10.1097/QAI.0b013e318162afe0. [DOI] [PubMed] [Google Scholar]
  15. Busman RA, Page C, Oka E, Giordani B, Boivin MJ. Factors contributing to the psychosocial adjustment of Ugandan preschool children with HIV/AIDS. In: Boivin MJ, Giordani B, editors. Neuropsychology of Children in Africa: Perspectives on Risk & Resilience. New York, NY: Springer Media & Business Publishing; 2013. pp. 95–115. [Google Scholar]
  16. Caldwell BM, Bradley RH. Home Observation for Measurement of the Environment. Little Rock, AR: University of Arkansas Press; 1979. [Google Scholar]
  17. Chiriboga CA, Kuhn L, Wasserman GA. Prenatal cocaine exposures and dose-related cocaine effects on infant tone and behavior. Neurotoxicol Teratol. 2007;29(3):323–330. doi: 10.1016/j.ntt.2006.12.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Davidson PW, Myer GJ, Shamlaye C, Cox C, Gao P, Axtell C, Clarkson TW. Association between prenatal exposure to methylmercury and developmental outcomes in Seychellois children: effect modification by social and environmental factors. Neurotoxicology. 1999;20(5):833–841. [PubMed] [Google Scholar]
  19. Dickerson K, Gerhardstein P, Zack E, Barr R. Age-related changes in learning across early childhood: a new imitation task. Dev Psychobiol. 2013;55(7):719–732. doi: 10.1002/dev.21068. [DOI] [PubMed] [Google Scholar]
  20. Drotar D, Olness K, Wiznitzer M, Guay L, Marum L, Svilar G, Kiziri-Mayengo R. Neurodevelopmental outcomes of Ugandan infants with human immunodeficiency virus type 1 infection. Pediatrics. 1997;100(1):E5. doi: 10.1542/peds.100.1.e5. [DOI] [PubMed] [Google Scholar]
  21. Durkin MS, Hasan ZM, Hasan KZ. The ten questions screen for childhood disabilities: its uses and limitations in Pakistan. J Epidemiol Community Health. 1995;49(4):431–436. doi: 10.1136/jech.49.4.431. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Emory E, Ansari Z, Pattillo R, Archibold E, Chevalier J. Maternal blood lead effects on infant intelligence at age 7 months. Am J Obstet Gynecol. 2003;188(4):S26–32. doi: 10.1067/mob.2003.244. [DOI] [PubMed] [Google Scholar]
  23. Fagan JF, Detterman DK. The fagan Test of Infant Intelligence: A Technical Summary. Journal of Applied Developmental Psychology. 1992;13:173–193. [Google Scholar]
  24. Familiar I, Murray S, Ruisenor-Escudero H, Sikorskii A, Nakasujja N, Boivin MJ, Bass JK. Socio-demographic correlates of depression and anxiety among female caregivers living with HIV in rural Uganda. AIDS Care. 2016;28(12):1541–1545. doi: 10.1080/09540121.2016.1191609. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Familiar I, Nakasujja N, Bass J, Sikorskii A, Murray S, Ruisenor-Escudero H, Boivin MJ. Caregivers’ depressive symptoms and parent-report of child executive function among young children in Uganda. Learn Individ Differ. 2016;46:17–24. doi: 10.1016/j.lindif.2015.01.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Forssman L, Ashorn P, Ashorn U, Maleta K, Matchado A, Kortekangas E, Leppanen JM. Eye-tracking-based assessment of cognitive function in low-resource settings. Arch Dis Child. 2017;102(4):301–302. doi: 10.1136/archdischild-2016-310525. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Gioia GA, Espy KA, Isquith PK. BRIEF-P Behavior Rating Inventory of Executive Function – Preschool Version: Professional Manual. Lutz, FL: Psychological Assessment Resources (PAR); 2003. [Google Scholar]
  28. Goldman DZ, Shapiro EG, Nelson CA. Measurement of vigilance in 2-year-old children. Developmental Neuropsychology. 2004;25(3):227–250. doi: 10.1207/s15326942dn2503_1. [DOI] [PubMed] [Google Scholar]
  29. Guzzetta A, Mazzotti S, Tinelli F, Bancale A, Ferretti G, Battini R, Cioni G. Early assessment of visual information processing and neurological outcome in preterm infants. Neuropediatrics. 2006;37(5):278–285. doi: 10.1055/s-2006-955929. [DOI] [PubMed] [Google Scholar]
  30. Jacobson SW, Chiodo LM, Sokol RJ, Jacobson JL. Validity of maternal report of prenatal alcohol, cocaine, and smoking in relation to neurobehavioral outcome. Pediatrics. 2002;109(5):815–825. doi: 10.1542/peds.109.5.815. [DOI] [PubMed] [Google Scholar]
  31. Jedrychowski W, Perera F, Jankowski J, Rauh V, Flak E, Caldwell KL, Lisowska-Miszczyk I. Prenatal low-level lead exposure and developmental delay of infants at age 6 months (Krakow inner city study) Int J Hyg Environ Health. 2008;211(3–4):345–351. doi: 10.1016/j.ijheh.2007.07.023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Jordan CM, Johnson AL, Hughes SJ, Shapiro EG. The Color Object Association Test (COAT): the development of a new measure of declarative memory for 18- to 36-month-old toddlers. Child Neuropsychology. 2008;14(1):21–41. doi: 10.1080/09297040601100430. [DOI] [PubMed] [Google Scholar]
  33. Kamya MR, Kapisi J, Bigira V, Clark TD, Kinara S, Mwangwa F, Dorsey G. Efficacy and safety of three regimens for the prevention of malaria in young HIV-exposed Ugandan children: a randomized controlled trial. Aids. 2014;28(18):2701–2709. doi: 10.1097/QAD.0000000000000497. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Klein P. Early Intervention: Cross-cultural experiences with a mediational approach. New York, NY: Garland Press; 1996. [Google Scholar]
  35. Klein P. Seeds of hope: twelve years of early intervention in Africa. Oslo, Norway: unipub forlag; 2001. [Google Scholar]
  36. Klein P, Boivin MJ. Final report on the advanced research training seminar (ARTS) on Mediational Intervention for Sensitizing Caregivers (MISC) Journal of Psychology. 1997;32:270–272. [Google Scholar]
  37. Klein P, Rye H. Interaction-Oriented Early Intervention in Ethiopia: the MISC Approach. Infants and Young Children. 2004;17:340–354. [Google Scholar]
  38. Melinder A, Konijnenberg C, Sarfi M. Deviant smooth pursuit in preschool children exposed prenatally to methadone or buprenorphine and tobacco affects integrative visuomotor capabilities. Addiction. 2013;108(12):2175–2182. doi: 10.1111/add.12267. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Mullen EM. Mullen Scales of Early Learning:AGS Edition. Minneapolis, MN: American Guidance Services; 1995. [Google Scholar]
  40. Murray SM, Familiar I, Nakasujja N, Winch PJ, Gallo JJ, Opoka R, Bass JK. Caregiver mental health and HIV-infected child wellness: perspectives from Ugandan caregivers. AIDS Care. 2016:1–7. doi: 10.1080/09540121.2016.1263722. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Myers GJ, Marsh DO, Davidson PW, Cox C, Shamlaye CF, Tanner M, Clarkson TW. Main neurodevelopmental study of Seychellois children following in utero exposure to methylmercury from a maternal fish diet: outcome at six months. Neurotoxicology. 1995;16(4):653–664. [PubMed] [Google Scholar]
  42. Nelson KG, Goldenberg RL, Hoffman HJ, Cliver SP. Growth and development during the first year in a cohort of low income term-born American children. Acta Obstet Gynecol Scand Suppl. 1997;165:87–92. [PubMed] [Google Scholar]
  43. Noble KG, Houston SM, Brito NH, Bartsch H, Kan E, Kuperman JM, Sowell ER. Family income, parental education and brain structure in children and adolescents. Nat Neurosci. 2015;18(5):773–778. doi: 10.1038/nn.3983. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Noble KG, Houston SM, Kan E, Sowell ER. Neural correlates of socioeconomic status in the developing human brain. Dev Sci. 2012;15(4):516–527. doi: 10.1111/j.1467-7687.2012.01147.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. O’Connor DL, Hall R, Adamkin D, Auestad N, Castillo M, Connor WE, Ross Preterm Lipid S. Growth and development in preterm infants fed long-chain polyunsaturated fatty acids: a prospective, randomized controlled trial. Pediatrics. 2001;108(2):359–371. doi: 10.1542/peds.108.2.359. [DOI] [PubMed] [Google Scholar]
  46. Perez GM, Swart W, Munyenyembe JK, Saranchuk P. Barriers to pilot mobile teleophthalmology in a rural hospital in Southern Malawi. Pan Afr Med J. 2014;19:136. doi: 10.11604/pamj.2014.19.136.5196. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Richmond J, Nelson CA. Relational memory during infancy: evidence from eye tracking. Dev Sci. 2009;12(4):549–556. doi: 10.1111/j.1467-7687.2009.00795.x. [DOI] [PubMed] [Google Scholar]
  48. Ruff HA, Capozzoli M, Dubiner K, Parrinello RA. A measure of vigilance in infancy. Infant Behavior and Development. 1990;13:1–20. [Google Scholar]
  49. Ruff HA, Capozzoli MC. Development of attention and distractibility in the first 4 years of life. Dev Psychol. 2003;39(5):877–890. doi: 10.1037/0012-1649.39.5.877. [DOI] [PubMed] [Google Scholar]
  50. Semrud-Clikeman M, Romero RA, Prado EL, Shapiro EG, Bangirana P, John CC. Selecting measures for the neurodevelopmental assessment of children in low- and middle-income countries. Child Neuropsychol. 2016:1–42. doi: 10.1080/09297049.2016.1216536. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. von Tetzchner S, Jacobsen KH, Smith L, Skjeldal OH, Heiberg A, Fagan JF. Vision, cognition and developmental characteristics of girls and women with Rett syndrome. Dev Med Child Neurol. 1996;38(3):212–225. doi: 10.1111/j.1469-8749.1996.tb15083.x. [DOI] [PubMed] [Google Scholar]
  52. Zelinsky D, Hughes S, Rumsey RI, Jordan C, Shapiro EG. The Early Childhood Vigilance Task: A new technique for the measurement of sustained attention in very young children. Journal of the International Neuropsychological Society. 1996;2(1):23. [Google Scholar]

RESOURCES