Validation and adaptation of Rapid Neurodevelopmental Assessment Instrument for Infants in Guatemala

Abstract

Background

Timely detection of neurodevelopmental impairments in children can prompt referral for critical services that may prevent permanent disability. However, screening of impairments is a significant challenge in low resource countries. We adapted and validated the Rapid Neurodevelopmental Assessment (RNDA) instrument developed in Bangladesh to assess impairment in nine domains: primitive reflexes, gross and fine motor development, vision, hearing, speech, cognition, behavior and seizures.

Methods

We conducted a cross-sectional study of 77 infants (0–12 months) in rural Guatemala in July 2012 and July 2013. We assessed inter-rater reliability and predictive validity between the 27-item RNDA and the 325-item Bayley Scales of Infant and Toddler Development, Third Edition (BSID-III) and concurrent validity based on chronic malnutrition, a condition associated with neurodevelopmental impairments. For both RNDA and BSID-III, standardized scores below 80 were defined as borderline impairment.

Results

Children came from rural households (92%), were born to indigenous women of Mayan descent (73%), and had moderate or severe growth stunting (43%). Inter-rater reliability for eight RNDA domains was of moderate to high reliability (weighted κ coefficients, 0.49–0.99). Children screened positive for impairment in fine motor (17%) and gross motor (14%) domains using the RNDA. The RNDA had good concurrent ability; infants who were growth stunted had higher mean levels of impairment in gross motor, speech and cognition domains (all p<0.001). The RNDA took 20–30 minutes to complete, compared to 45–60 minutes for BSID-III.

Conclusions

Wide-scale implementation of a simple, valid and reliable screening tool like the RNDA by community health workers would facilitate early screening and referral of infants at-risk for neurodevelopmental impairment.

Introduction

Detection and early treatment of child neurodevelopmental impairments (NDIs) is a challenge in low resource countries, where the majority of children reside (Walker et al. 2007). Chronic malnutrition, specifically protein-energy malnutrition and micronutrient deficiencies, during early childhood may impact neurological development (Scrimshaw 2003, Grantham-McGregor and Fernald 1997). Guatemala is one of the top five countries in the world burdened with chronic malnutrition (Reurings et al. 2013). Almost 55% of children <5 years of age are moderately growth stunted, defined as ≤ 2 standard deviations below median z-score for height-for-age based on the World Health Organization’s children growth standards (de Onis and Blossner 2003). Malnutrition, assessed by failure to achieve optimal linear growth, has been associated with subsequent neurodevelopmental delays (Pfister and Ramel 2014, Prendergast and Humphrey 2014, Stephens and Vohr 2014), and these delays are even more pronounced in low birth weight (Makrides et al. 2013) and preterm infants (Ramel et al. 2012). The prevalence of NDIs among Guatemalan infants is unknown. Children with impaired neurodevelopment are at high risk for permanent functional limitations, reducing their educational and economic opportunities later in life. Early identification of children with neurological deficits is critical to preventing long-term disability.

Valid and reliable neurodevelopmental assessment tools such as the 115-item Network Neurobehavioral Scale (Lester and Tronick 2004, Lester et al. 2004), the 53-item Neonatal Behavior Assessment (Brazelton 1978), and the 325-item Bayley Scales of Infant and Toddler Development, Third Edition (BSID-III) (Bayley 2006) have been developed to discriminate neurologic and behavioral deviations from normal infant behavior. These scales have been used extensively in clinical research (Smith et al. 2008, Law et al. 2003), but all require intensive training in administration and interpretation of results (Johnson et al. 2014) and few have been validated in non-English speaking populations in low-resource countries. Minimally skilled health workers may find it difficult to adhere to standardized testing procedures, and children from low-resource settings may not be familiar with test items (Cromwell et al. 2014). Furthermore, these scales are normed to a reference population that may not be representative of populations in resource-constrained international settings. BSID-III scores are based on a normative sample of 1,700 English-speaking children in the United States (Bayley 2006).

The Ten Questions instrument, developed by researchers in Bangladesh, is one of the most common neurodevelopmental screening tools used; however, its validity is lower for milder forms of disabilities and its applicability is limited to children over 2 years of age (Zaman et al. 1990, Maulik and Darmstadt 2007). The Developmental Screening Questionnaire was recently validated for use with children less than 2 years of age. This questionnaire has 97% specificity and sensitivity ranging from 100% for vision and hearing impairment to 45% for cognitive impairment (Khan et al. 2013). Many screening instruments lack standardized definitions and measurement methods. The WHO developed the International Classification of Functioning, Disability and Health (ICF), a tool to facilitate standardized, universal assessment of a spectrum of health and disability, including NDIs across the lifespan (World Health Organization October, 2013). A standardized, validated tool that is suitable to local contexts, feasible for use by CHWs in low and middle-income countries, and accurately screens young infants is needed.

The Rapid Neurodevelopmental Assessment (RNDA) was developed in Bangladesh to assess neurodevelopmental status of children < 2 years of age with limited access to health screening (Khan et al. 2010). The RNDA is meant to provide a comprehensive neurodevelopmental evaluation equivalent to more comprehensive scales like the BSID-III, against which the tool has been validated. The 27-item assessment tool has been shown to discriminate between children with and without NDIs in both hospital and community settings in Bangladesh (Khan et al. 2010). RNDA measures not only developmental milestones, but also measures vision, hearing, behavior, seizures and sleep patterns which may be more predictive of long term developmental problems than developmental milestones alone. The tool was intended to be used in low-resource settings where communities rely on semi-skilled health workers. Here we describe the validation and adaptation of the RNDA instrument for use with rural Guatemalan infants. The aims of this study were to: 1) establish face validity of the RNDA; 2) determine inter-rater reliability between health professionals and community field workers in performing the RNDA; 3) assess the concurrent validity of the RNDA against the BSID-III, considered the gold standard in infant developmental assessment; and 4) evaluate the concurrent validity of the RNDA in Guatemalan infants who are chronically malnourished.

Participants and Methods

A convenience sample of Spanish- and Mam-speaking (indigenous) mothers and their infants were recruited from Western Highland communities in rural Guatemala. Seventy-seven infants (18 infants at < 1 month; 16 at 3 months; 17 at 6 months; 13 at 9 months; and 13 at 12 months of age) participated in this cross-sectional study, which was conducted in July 2012 and July 2013. Infants with a history of known pre-term birth (e.g. mother reported that child was born too early and was subsequently hospitalized for respiratory or feeding conditions related to prematurity), chronic medical conditions (e.g. cardiac disease), or obvious NDIs (e.g. deafness, facial features suggestive of trisomy 21) were excluded. The study took place in a rural region where we have conducted research for the past decade (Smith et al. 2011, Thompson et al. 2013, Thompson et al. 2011).

The primary investigator (LMT) was trained on the RNDA by the instrument developers in Bangladesh and was the gold standard for assessing inter-rater reliability in Guatemala. Licensed psychologists and a neonatologist at UCSF trained the primary investigator to administer the BSID III. We subsequently trained two local Guatemalan community health workers over a 2-week period in July 2012, and during a 1-week recalibration period in July 2013, to perform the RNDA. We assessed face validity and inter-rater reliability during the pilot period and again during the data collection period. During the pilot period, we focused on modifications to testing procedures (e.g. quiet setting and evaluator’s location to minimize child distraction, using heater to warm room, ordering of items for assessment). We assessed face validity of the instrument during the pilot period by observing whether the 12 mothers were comfortable and engaged during the infant assessment, two methods for assessing acceptability. During the pilot period, we assessed health workers’ ease and adherence to testing procedures. To do this, we videotaped all pilot and study assessments and reviewed and discussed videos immediately after each assessment to improve ease of performing procedures and to calibrate raters. Videos were reviewed by the developers in Bangladesh (NZK and HM) and corrective actions were taken to standardize future infant evaluations after the pilot and recalibration periods.

We collected demographic, pregnancy, and birth history information from the mothers, adapting the questionnaires from the RNDA manual (Kahn and Muslima 2012) and questionnaires used in our former studies on birth outcomes (Thompson et al. 2011, Thompson et al. 2013). We performed one RNDA assessment per child in a quiet room with the infant sitting on the mother’s lap. A health worker (GD) and the primary investigator stood behind the child and each scored the items on the RNDA independently, without knowledge of the other’s score. At the end of the assessment, if evaluators were not able to score an item (e.g., could not see infant looking at an object), videos were reviewed to complete independent scoring. After completion of the RNDA, the primary investigator conducted the BSID-III under the same testing conditions.

We tested the concurrent validity of the RNDA on stunting. Stunting has been shown to be an objective health measure predictive of future neurodevelopmental delay (Grantham-McGregor et al. 2007, Prendergast and Humphrey 2014). We conducted anthropometric measurements on all children after the RNDA and BSID III assessments. Infants wore standardized, weighed clothing and weight was measured to the nearest 10 grams using a Seca 334 digital infant scale. Head circumference was measured to the nearest 0.5 cm using a Gulick II 150 cm anthropometric tape. Length was measured to the nearest 0.5 cm using a Seca 210 measuring mat. Scales were calibrated with 5 pound standard weights before and after each testing day. Evaluators were trained using standardized protocols based on protocols developed by NHANES (National Health and Nutrition Examination Survey (NHANES) 2011), with over 10 years’ experience conducting anthropometry on children under the age of 5 years during a longitudinal birth cohort study (Smith et al. 2011, Bruce et al. 2007, Thompson et al. 2011) and follow-up studies.

Instruments

Rapid Neurodevelopmental Assessment (RNDA)

The RNDA assesses nine domains: primitive reflexes, gross and fine motor development, vision, hearing, speech, cognition, behavior and seizures in children <24 months of age. The evaluation takes between 30–45 minutes to complete, depending on the age and alertness of the child. Scoring for infants < 1 month is graded as low, moderate, or high risk for impairment. Scoring for infants > 1 month is normal, mild, moderate, or severe. The RNDA has been found to be reliable (inter-rater reliability κ scores ranged from 0.76 to 1.0 across the nine domains) and valid (concurrent validity between the RNDA domains and the BSID, Second Edition; discriminant validity between rural and urban children) for assessing NDIs in Bangladeshi children (Khan et al. 2010).

Spanish translation of the RNDA

For the present study, we developed Spanish-language questionnaires and laminated cards with pictograms to organize and describe the testing procedure for the following age groups: 0 to < 1 month, 1 to < 3 months, 3 to < 6 months, 6 to < 9 months, and 9 to <12 months of age. All forms were back-translated into English to check for consistency and pilot tested during the initial training session and modified to improve understanding. The RNDA was scored using the instruction manual (Kahn and Muslima 2012). Face validity was assessed by how well the RNDA was accepted by mothers of infants who participated in the study and how well the health workers understood and adhered to the test procedures during each evaluation.

Bayley Scales of Infant and Toddler Development, Third Edition (BSID-III)

The BSID-III assesses the development of children between the ages of 16 days and 42 months. Depending on the age of the child, test administration is estimated to take between 45 and 90 minutes. The child’s chronological age determines the starting point. The first three items in a series must be performed correctly. If a child is unable to correctly perform the first three items at the starting point, then the examiner must go to previous starting points and continue forward until the child completes three consecutive items before moving forward. Then, the examiner tests forward until 5 consecutive items failures occurs (the ceiling point). For the present study, we assessed five BSID-III domains: cognition, receptive and expressive communication, and fine and gross motor development. An independent researcher at UCSF calculated the BSID-III scores using the Scoring Assistant Software (available from http://www.pearsonclinical.co.uk) (Bayley 2006).

Statistical Analysis

We assessed inter-rater reliability of the RNDA using ordinal scores for each of the domains, using a weighted kappa (κ) statistic that assigned less weight to scores that were further apart. This strategy was selected so that extreme differences would be made more apparent (Cohen 1968). Commonly accepted levels of agreement are: moderate agreement (0.41–0.60); substantial agreement (0.61–0.80) and almost perfect agreement (0.81–0.99)(Viera and Garrett 2005).

To determine NDI in this Guatemalan sample, we used age-specific means and standard deviations for each domain because each domain used a different number of tests to evaluate impairment for each age group. Raw scores for each domain and age group were converted into z-scores and then standardized to a mean of 100 and a standard deviation (SD) of 15 to homogenize the scales using procedures described in the BSID-III reference manual (Bayley 2006). For both RNDA and BSID-III, standardized scores below 80 were borderline impairment. We assessed concurrent validity of four domains (fine motor, gross motor, speech, and cognition) on the BSID-III and the RNDA. We assessed sensitivity, specificity, positive and negative predictive validity to demonstrate equivalence between the two instruments.

We calculated height-for-age, weight-for-age, and head circumference-for-age using the 2006 World Health Organization (WHO) Multicentre Growth Reference Standard. Values more than 2 standard deviations below the median international reference population are “moderate” deficits and values more than 3 standard deviations below the median are “severe” deficits (de Onis and Blossner 2003).

We received ethical approval from the Committee for Human Research at the University of California, San Francisco, and from the Universidad del Valle in Guatemala. Written informed consent was obtained in Spanish or Mam.

Results

Subjects

Participating children came from rural households (92%) and 74% were born to Mam-speaking indigenous women (Table 1). Infant complications at birth were reported by 19% of Spanish-speaking mothers and 37% of Mam-speaking mothers. The two most common reported complications were “didn’t cry immediately after birth” and “small in size at birth”. Only four (19%) of the infants born to Spanish-speaking mothers were moderately stunted (height-for-age >2 SD below median z-score), while 30 (41%) of the infants born to Mam-speaking mothers were moderately (n=17, 29%) or severely (height-for-age >3 SD below median z-score) (n=13, 22%) stunted. Nine of the 10 infants with head circumferences more than 2 SD below median z-score were born to Mam-speaking mothers.

Table 1.

Demographic characteristics of study sample (n=77)

	N (%)
Maternal Characteristics
Ethnicity
Mam-speaking indigenous Mayan	56 (73%)
Spanish-speaking mestizo	21 (27%)
Maternal education
None	6 (8%)
Elementary	36 (47%)
Middle/Secondary/Technical school	35 (45%)
Infant Characteristics
Infant sex
Female	38 (49%)
Feeding
Breast	53 (71%)
Bottle/breast	22 (30%)
Birth place
Home	23 (30%)
Health Center/National Hospital	49 (64%)
Private Hospital	5 (6%)
Delivery type
Vaginal	58 (75%)
Cesarean	19 (25%)
Neonatal complications at birth	24 (31%)
Height for age z-score
Moderate (% >2 SD below median z-score)	20 (26%)
Severe (% >3 SD below median z-score)	13 (17%)
Weight for age z-score
Moderate (% >2 SD below median z-score)	5 (7%)
Severe (% > 3 SD below median z-score)	4 (5%)
Head circumference for age z-score
Moderate (% >2 SD below median z-score)	10 (13%)
Severe (% >3 SD below median z-score)	0 (0%)
Household Characteristics
Residence
Urban	6 (8%)
Rural	71 (92%)
Primary household occupation
Cultivates other’s land	26 (34%)
Cultivates own land	19 (25%)
Employee (public or private); artisan; commerce	30 (39%)
Number of household members, mean (SD)	7.6 (3%)
Number of people providing economic support, mean (SD)	1.9 (1%)
Ratio household members to economic support	4.7 (2%)
Stove type
Wood-fired chimney stove	58 (75%)
Wood-fired open fire	13 (17%)
Propane gas stove	6 (8%)
Lighting
Electricity	70 (91%)
Candles/kerosene lamp	7(9%)

Reliability

None of the mothers reported a history of current seizures in their children, thus this domain was not evaluated. Inter-rater reliability was substantial to almost perfect across 7 domains. Weighted κ coefficients based on ordinal 3-point (for infants ≤ 1 month) and a 4-point (for infants ≥1 months of age) scales: primitive reflexes in infants < 1 month (κ=0.63); cognition (κ=0.99); gross motor (κ=0.90); fine motor (κ=0.87); speech (κ=0.86); behavior (κ=0.81); vision (κ=0.78). Hearing (κ=0.49) had the lowest agreement between the two observers, which was due to difficulty seeing infants’ eyes and head turns when assessing infants from behind, even after viewing videos.

Validity

The RNDA had adequate face validity. The assessment was well accepted by the mothers and health workers were able to adhere to the procedures. We explained to the mothers the purpose of testing primitive reflexes. The Moro, or startle reflex, and the stepping and placing reflex tests were “unusual” to mothers, but well tolerated by the infants. Other tests, such as ringing a bell to test hearing or suspending a yarn ball within the child’s line of vision and watching eye movement were well received by the mothers. Health workers adhered to the procedures through the use of laminated sheets that graphically displayed and organized the order of procedures for the health worker.

Overall, specificity between the RNDA and the BSID-III was high; 85% or greater for infants who were assessed to be negative for NDIs across the 4 domains. However sensitivity was low. The highest sensitivity (43%) was in the gross motor domain. Of the seven children who were found to have borderline gross motor impairment on the BSID-III, only 3 were found to have a gross motor impairment on the RNDA. Of the four children who were found to have borderline speech impairment on the BSID-III, none were found to have a speech impairment using the RNDA. Correspondingly, negative predictive validity was high and positive predictive validity was low (Table 2).

Table 2.

Agreement between Rapid Neurodevelopmental Assessment (RNDA) and Bayley Scales of Infant and Toddler Development, Third Edition (BSID III) for four common developmental domains

RNDA Borderline Impairmenta
BSID III	Sensitivity % (n) (95% CI)	Specificity % (n) (95% CI)	Positive Predictive Validity % (95% CI)	Negative Predictive Validity % (95% CI)
Gross motor
Borderlinea	43% (3/7) (10–82%)	90% (62/69) (80–96%)	30% (7–65%)	94% (85–98%)
Fine motor
Borderlinea	40% (2/5) (5–85%)	85% (61/72) (74–92%)	15% (2–45%)	95% (87–99%)
Speech
Borderlinea	0% (0/4) (0–60%)	94% (67/71) (86–98%)	0% (0–60%)	94% (86–98%)
Cognition
Borderlinea	29% (2/7) (4–71%)	97% (68/70) (90–99%)	50% (7–93%)	93% (85–98%)

Concurrent validity of the RNDA was assessed by examining levels of NDIs between 33 moderate/severe stunted infants and 44 infants above the cut-off for moderate stunting (Figure 1). Infants who were moderately/severely stunted had higher mean levels of impairment than infants who were not, and there were statistically significant mean differences in RNDA scores for gross motor (moderate/severe stunting, mean=0.83 [SD 0.71)]; mild/no stunting, mean=0.42 [SD 0.42], p< 0.01); speech (moderate/severe stunting, mean=1.12 [SD 0.85]; mild/no stunting, mean=0.59 [SD 0.81], p<0.01) and cognition (moderate/severe stunting, mean=0.76 [SD 0.73]; mild/no stunting, mean=0.22 [SD 0.45], p<0.001).

Figure 1.

Administration and Neurodevelopmental Impairment

Depending on the age of the child, the RNDA exam took 20–30 minutes and the BSID-III took 45–60 minutes to complete. Moderate delay on the RNDA was found in the following domains: gross motor (n=11; 14%); fine motor (n=13; 17%); speech (n=4; 5%); cognition (n=4; 5%); vision (n=6; 8%); hearing (n=7; 9%) and behavior (n=10; 13%) for all age groups and in primitive reflexes (n=1; 6%) for infants < 1 month of age.

Discussion

We successfully trained community health workers to implement the RNDA, achieving moderate to high inter-rater reliability. Mothers who participated in the study were comfortable with the RNDA procedures and viewed it as an acceptable method for screening for neurodevelopment of their infants. The advantage to using the RNDA is its ability to be performed by community health workers after a short training period with intermittent recalibration of assessments. The RNDA also assesses vision and hearing impairment, which are not directly evaluated by the BSID-III.

There is a trade-off between sensitivity and specificity when developing a rapid screening instrument. The RNDA and BSID-III perform comparably well in identifying children without NDIs and demonstrated moderate to high agreement. The negative predictive validity was high, which demonstrates that both tests agreed when there were no signs of NDIs. However, the positive predictive validity, or the proportion of infants who screened positive for NDIs, was low. Thus, the RNDA is appropriate as a rapid assessment tool, but it may not accurately detect the full spectrum of NDIs in these four domains. Misclassification of NDI can occur for at least two reasons. First, the range of RNDA items used to assess NDI at each age interval varies from 27 items at less than 1 month to 16 items at 12 months. In contrast, the BSID-III incorporates more testing items and the number of items tested is dependent on infant’s success or failure at each level. The RNDA evaluates expressive speech using one item in each of the age groups, while the BSID-III offers more opportunities to evaluate both expressive and receptive speech. Thus, four children were found to have borderline speech impairments with the BSID-III, but because these same children passed the single RNDA item, they were not found to have speech impairments. Conversely, the four children who failed the single item on the RNDA were able to successfully pass BSID-III speech evaluation. A potential solution to this would be to add items to the RNDA that would allow more opportunities to pass or fail in each domain. Second, the BSID-III allows for children who fail the test at their chronological age to move back to a younger age and then move forward until they are unable to perform five consecutive tests. The width of the age bands for the BSID-III is 10 days for infants less than 5 ½ months of age, and 1 month for children up to one year. The age band width for the RNDA is 30 days for all age groups. Child development falls on a spectrum and using wide age bands means that some children will be misclassified as having, or not having, NDIs. This can lower sensitivity of tests to 70% or less (Veldhuizen et al. 2014), which may explain the sensitivity levels in our study.

For the RNDA scores, we determined NDI norms based on this Guatemalan sample, converting raw scores for each domain and age group into z-scores and then standardizing scores to a mean of 100 and a standard deviation of 15. This procedure could be used with future studies that wish to screen for NDI’s using the RNDA, thus allowing for comparability across different populations. The BSID-III is normed to a reference population of US children, and may lead to distorted findings in other populations. To our knowledge, the BSID-III has not been validated for use in Guatemalan infants.

Although the participants in this study are socio-economically similar to the sample used to develop the RNDA in Bangladesh, more infants screened positive for neurodevelopmental impairment among our sample (Khan et al. 2010). We found good concurrent ability to detect NDIs in the children who were chronically malnourished (43% of our sample were moderately or severely stunted). Given the sample size of this study, and the young age of the infants, larger studies would need to be conducted to assess whether these impairments are present at later ages, or if they resolved over time. Malnutrition, as measured by household food insecurity, during early infancy was associated with language delay at 18 months of age among a large cohort of Bangladeshi infants (Saha et al. 2010).

Additional factors, besides chronic malnutrition, may have contributed to our findings of high NDI and need to be considered in future research. Lack of familiarity with testing activities (i.e. not used to playing with blocks) and stranger anxiety/shyness may have biased results. For instance, the BSID III used books with pictures of children engaging in activities that were unfamiliar to these children, like riding a shiny bike, or other illustrations of affluence. Even when tests were administered by someone from their own ethnic group and in their native language, some children were reluctant to engage in activities. We noticed that mothers were generally reserved even with their own children; speech may come later for these children, perhaps due to parental modeling. Gross motor may also develop later among rural, indigenous Guatemalan infants. Infants are either worn on mothers’ backs or bound in restrictive clothing, which would limit skills like rolling over or crawling. Opportunities to explore environments that are socially mediated may appear to contribute to NDIs in these infants as has been seen in children in other indigenous communities (Cappiello and Gahagan 2009). Instruments that fail to integrate sociocultural factors that contribute to children’s development may lead to incorrect estimates of delay.

Conclusion

Early identification of children at-risk for NDIs in low-resource settings remains a challenge. In many countries such as Guatemala, infants born at home and delivered by traditional birth attendants are not seen at health centers until after the neonatal period, at which time impairments may be more difficult to correct (Replogle 2005). Implementation of a simple screening tool like the RNDA by community health workers may facilitate early detection and referral of infants at risk for neurodevelopmental delays. A caveat to early detection is the urgent need for infrastructure in low-income countries that can support young children with NDIs to achieve their fullest potential.

Key messages.

The RNDA:

• Is a simple instrument used to screen for infant neurodevelopmental impairment

• Can be used by community health workers in low resource settings

• Can be adapted and used with community workers in other low income settings, such as Guatemala

• Has high specificity and is thus able to detect those without neurodevelopmental impairment

• Has concurrent validity for detecting cognitive and gross motor delays among infants with stunting, a measure of chronic malnutrition

Abbreviations

BSID III

Bayley Scales of Infant and Toddler Development, Third Edition

NDIs

neurodevelopmental impairments

RNDA

Rapid Neurodevelopmental Assessment