Using parent questionnaires to assess neurodevelopment in former preterm infants: a validation study

Abstract

Background

Former preterm and very low birth weight (VLBW) infants require close neurodevelopmental surveillance after hospital discharge, but in-person professional testing is resource-intensive and inconvenient for families. A standardised developmental questionnaire completed by parents offers an alternative to in-person testing, but few such questionnaires have been validated. Our aim was to validate the Motor and Social Development (MSD) scale in a sample of former preterm infants.

Methods

We studied 321 visits to a neonatal follow-up clinic. Parents completed the MSD, which measures cognitive, motor, and social abilities. Psychologists and physical therapists administered the Bayley Scales of Infant Development, 3^rd edition (Bayley-III) cognitive and motor scales.

Results

The median (range) gestational age was 28 (23, 34) weeks and birthweight 980 (400, 2700) g. Corrected age at study participation ranged 5–35 months. The mean (standard deviation) Bayley-III motor score was 94 (16); cognitive 98 (16); and MSD 91 (18). Internal consistency of the MSD was moderate to high (Cronbach alpha of 0.65 to 0.88). The MSD was moderately correlated with the Bayley-III motor (Pearson r=0.49, P<0.001) and cognitive (r=0.45, P<0.001) scales. The area under the receiver operating characteristic curve was 0.88 (95% confidence interval [CI] 0.81, 0.95) for the MSD to detect a low Bayley-III motor score (<70); and 0.88 (95% CI 0.82, 0.95) for a low cognitive score, indicating good discrimination.

Conclusions

The MSD has good internal and concurrent validity, and may be useful for neurodevelopmental assessment of former preterm and VLBW infants in clinical and research settings.

Introduction

Improvements in obstetrical and neonatal intensive care have led to a marked increase in survival after preterm birth, but also to an increase in the number of infants at risk for neurodevelopmental impairments, particularly preterm infants with very low birth weight (VLBW, <1500g).^{1, 2}

Neurodevelopmental assessments for VLBW infants are recommended at least twice in the first 3 years of life; when not possible, multidimensional neurodevelopmental screening is an alternative.³ Such assessments may serve 3 purposes: (i) identification of individual children for prompt referral to early intervention services, with the potential for improving outcomes^{4, 5}; (ii) surveillance, to inform researchers and policy makers on outcomes and identify potentially ameliorative strategies; and (iii) facilitation of research on the long-term impact of specific neonatal practices, for example in randomised clinical trials.

Formal, multi-disciplinary assessments by trained professionals have important advantages, such as the ability to tailor therapies and provide specific referrals, but can be expensive, time-consuming, and inconvenient for families. Clinical resources may be insufficient to provide testing to all VLBW infants in many areas, and insurance may not pay for testing. In the research setting, neurodevelopmental follow-up requires substantial funding. Identifying less costly and burdensome methods to obtain information about neurodevelopment could benefit clinicians, researchers, and families.

Standardised developmental questionnaires that are completed by parents offer an alternative to formal in-person testing, but few^6–9 have been validated in populations of preterm or VLBW infants. The Motor and Social Development (MSD)^{10, 11} scale is a brief parent questionnaire that was developed for children aged 4 months to 4 years. Many MSD items correspond closely to items on the Bayley Scales of Infant Development, a gold standard for in-person neurodevelopmental assessment of VLBW infants. The MSD has normative scoring based on a large, representative US sample, and a Spanish version.

Our aims were to: (i) validate the MSD scale, as compared with formal neurodevelopmental testing with the Bayley Scales of Infant and Toddler Development, 3^rd edition (Bayley-III) and; (ii) examine the reliability of parent report for specific developmental items, as compared with professional assessors.

Methods

Study design and participants

We recruited participants from the Infant Follow-up Program, a tertiary children’s hospital-based multidisciplinary clinic that provides medical and neurodevelopmental follow-up for infants discharged from 3 Boston Neonatal Intensive Care Units (NICU’s). Eligibility criteria for attending the clinic are: gestational age <32 completed weeks; birth weight <1500 grams; and/or social risk factors (e.g., teenage mother, maternal substance abuse). Typically, children are first evaluated six months after hospital discharge and then every 6 months until 3 years of age.

All patients attending the clinic were eligible for this study. Parents of scheduled patients were invited by mail to complete the study questionnaire 2 weeks before their appointment; parents who did not bring a completed questionnaire to the appointment could complete it on the day of their appointment. Consent was obtained when the parent agreed to complete the questionnaire. The Children’s Hospital Boston human subjects committee approved the study protocol.

Participant flow is shown in Figure 1. Of the 495 clinic visits from July, 2009 to December, 2011, parents completed study questionnaires for 452 (91%). Of those, we excluded 121 visits because parents responded “I don’t know” or skipped one or more MSD scale items. Of the remaining 331 visits, we included 321 for which the child had also completed either the Bayley-III motor or cognitive scales. The 321 visits represented 187 participants, of whom 86 participated once, 69 twice, 31 three times, and 1 four times.

Figure 1.

Participant flow. MSD is motor and social development scale and CP is cerebral palsy.

Developmental questionnaires

The MSD scale^{11, 12} was developed by the National Center for Health Statistics to measure motor, social, and cognitive development of young children. It comprises 48 items derived from standard measures of child development, including the Bayley Scales of Infant Development, 2^nd edition, Gesell Scale, and Denver Developmental Screening Test¹⁰. The MSD was used first in the Child Health Supplement to the 1981 National Health Interview Survey, and subsequently in the National Longitudinal Survey of Youth and the third National Health and Nutrition Examination Survey. Of note, the MSD can be given to children 4 months to 4 years, but appears to be relatively insensitive to developmental differences in children older than 3 years¹².

For each MSD item, parents responded “yes,” “no,” or “I don’t know” to whether the child was able to perform a particular activity. We administered all items, and instructed parents to stop answering questions once they had responded “no” to 6 or more questions in a row. Using fifteen age-appropriate items for each age group (Supplemental Figure 1), we summed the number of “yes” responses, and calculated scaled scores based on national norms¹² for the child’s age corrected for preterm birth.

In-person neurodevelopmental assessments

The Bayley-III ¹³ was administered as part of routine clinical care by doctoral level psychologists (cognitive scale) and physical therapists (motor scale, with fine and gross motor subtests). We did not routinely administer the Language, Social-emotional, or Adaptive Behavior scales. The Bayley-III is scaled to a mean score of 100 and SD of 15. We calculated scores based on participant age corrected for preterm birth.

Child, maternal, and family characteristics

From the child’s NICU discharge summary and Infant Follow-up Program medical record, we collected data regarding maternal and neonatal characteristics. Parents completed a short questionnaire about sociodemographic characteristics including race, ethnicity, mother’s primary language and education level, and annual household income.

Analysis

To assess internal consistency of the MSD, we calculated the raw Cronbach alpha coefficient¹⁴ for each age group. To assess concurrent validity with the Bayley-III cognitive and motor scales, we calculated Pearson correlation coefficients and P-values. We calculated correlations for the entire cohort, and stratified by participant age and maternal education level.

We examined test characteristics (sensitivity, specificity, positive predictive value, and negative predictive value) with confidence interval [CI]’s of the MSD score cutoff with maximum sensitivity to detect a low Bayley-III score (<70), while also maximizing specificity, and also examined a low Bayley-III score <85 because the Bayley-III appears to underestimate the degree of developmental delay in preterm infants¹⁵. To calculate these 95% CIs, we used standard errors from intercept-only linear models using a binary error distribution and log link function, fit with generalised estimating equations to account for repeated measurements within subjects¹⁶. To assess discrimination at varying MSD cutoffs, we created receiver operating characteristic (ROC) curves by plotting the sensitivity on the y-axis and 1-specificity on the x-axis, and calculated the area under the ROC curve with 95% confidence intervals (CI’s).

For the 24 MSD items that were identical or nearly identical to Bayley-III items, we examined agreement between parents and professionals using the Kappa statistic¹⁷. We used SAS version 9.2 (SAS Institute Inc., Cary, NC) and SPSS Statistics 19 (SPSS Inc., Chicago IL) for analyses.

Results

Participant characteristics are shown in Table 1 and neurodevelopmental test scores in Table 2. Internal consistency of the MSD was moderate to high for all age groups (Table 3; Cronbach alpha coefficient range, 0.65 to 0.88).

Table 1.

Description of 187 participants

Characteristics	Median (range)
Birth weight (grams)	980 (400, 2700)
Gestational age (weeks)	28 (23, 34)
Maternal age (years)	32 (19, 45)
Apgar score at 5 minutes	8 (1, 9)
Chronologic age (months)	17.6 (7.2, 38.1)
Corrected age (months)	15.7 (4.9, 34.9)
	Number (percent)
Male	98 (52%)
Gestation
Singleton	98 (52%)
Twin	70 (38%)
Triplet	19 (10%)
Child’s race/ethnicity
White, non-Hispanic	118 (63%)
Non-white and/or Hispanic or missing	69 (37%)
Mother’s primary language is English	158 (85%)
Mother’s attained education
Less than a college degree	65 (35%)
At least a college degree	122 (65%)
Annual household income
<$60,000 or missing	66 (35%)
≥$60,000	121 (65%)
Parent completing questionnaire
Mother	158 (84%)
Father or other	29 (16%)
NICU diagnoses
Surfactant deficiency	171 (91%)
Chronic lung disease	90 (48%)
Patent ductus arteriosus	96 (51%)
Retinopathy of prematurity (any)	94 (50%)
Necrotizing enterocolitis	16 (9%)
Congenital anomaly	14 (7%)
Intraventricular hemorrhage, grade 3 or 4	18 (10%)
Periventricular leukomalacia	14 (7%)
Post-discharge diagnoses
Cerebral palsy	10 (5%)
Hearing impaired/deaf	3 (2%)
Visually impaired/blind	12 (6%)
Global developmental delay	3 (2%)
Autism/pervasive developmental disorder	2 (1%)
None of these	162 (87%)
Post-discharge services/interventions
Early intervention	162 (87%)
Home oxygen	6 (3%)
Gastrostomy tube	18 (8%)
Tracheostomy	4 (2%)
Adaptive stroller / wheelchair	3 (2%)

Table 2.

Neurodevelopmental test scores for 321 clinic visits

Test	N	Mean (SD)
Provider administered test
Bayley-III
Motor	299	94 (16)
Cognitive	207	98 (16)
Bayley-III score <85		Percent
Motor	58	21%
Cognitive	30	14%
Bayley-III score <70
Motor	15	5%
Cognitive	9	4%
Parent questionnaire		Mean (SD)
MSD scale	321	91 (18)
Low MSD
≤83	108	34%
≤95	198	62%
≤104	262	82%

Bayley-III is the Bayley Scales of Infant and Toddler Development, 3^rd edition. MSD is motor and social development.

Table 3.

Internal consistency of the Motor and Social Development scale

Age category (months)*	n	Raw Cronbach alpha coefficient
7–9	31	0.70
10–12	35	0.75
13–15	31	0.65
16–18	39	0.77
19–21	40	0.70
22–24	36	0.74
25–27	21	0.75
28–30	25	0.88
31–34	27	0.81

^*age is corrected for prematurity

Table 4 shows moderate correlations of the MSD with the Bayley-III motor and cognitive scales. In younger children (<18 months), correlations generally appeared stronger for motor vs. cognitive scores. Additionally, correlations were weaker for children of mothers with higher vs. lower education, particularly for the cognitive scale.

Table 4.

Correlations of Motor & Social Development scale with Bayley-III scores

	Bayley-III motor scale	Bayley-III cognitive scale
	Pearson correlation coefficient (p-value), number
All visits	0.49 (p<0.0001) N=279	0.45 (p<0.0001) N=207
First visit only	0.45 (p<0.0001) N=166	0.36 (p<0.0001) N=109
Age categories (all visits)
<9 months	0.36 (p=0.36) N=58	0.25 (p=0.25) N=20
9 to <12 months	0.54 (p=0.07) N=12	0.32 (==0.34) N=11
12 to <15 months	0.48 (p=0.0009) N=44	0.28 (P=0.14) N=29
15 to <18 months	0.59 (P=0.0006) N=30	0.33 (P=0.12) N=23
18 to <24 months	0.47 (P<0.0001) N=68	0.52 (P<0.0001) N=49
≥24 months	0.54 (P<0.0001) N=67	0.50 P<0.0001) N=67
Maternal education
Low	0.45 (p<0.0001) N=89	0.32 (p=0.008) N=67
High	0.54 (p<0.0001) N=190	0.54 (p<0.0001) N=140

The areas under the ROC curves demonstrated moderately high discrimination of the MSD for detecting low Bayley-III scores (Table 5). For detecting a Bayley-III motor score <70, we chose an MSD cutoff of ≤95 points to maximize sensitivity (100%) with maximum specificity. For detecting a Bayley-III cognitive score <70, an MSD cutoff of ≤83 optimized sensitivity with maximum specificity. ROC curves are shown in Supplemental Figure 2.

Table 5.

Motor and Social Development scale screening for low Bayley-III scores: discrimination and test characteristics

Outcome	AUC (95% CI)	Cutoff (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Bayley motor <70	0.88 (0.81, 0.95)	≤95	1.00 (0.78, 1.00)	0.42 (0.37, 0.48)	0.09 (0.05, 0.14)	1.00 (0.97, 1.00)
Bayley motor <85	0.83 (0.77, 0.88)	≤95	0.97 (0.88, 1.00)	0.48 (0.43, 0.54)	0.32 (0.26, 0.40)	0.95 (0.91, 0.99)
Bayley cognitive <70	0.88 (0.82, 0.95)	≤83	1.00 (0.66, 1.00)	0.69 (0.63, 0.76)	0.11 (0.06, 0.21)	1.00 (0.97, 1.00)
Bayley cognitive <85	0.78 (0.67, 0.87)	≤104	0.97 (0.83, 1.00)	0.20 (0.16, 0.26)	0.16 (0.12, 0.23)	0.93 (0.85, 1.00)

AUC is area under the curve. PPV is positive predictive value and NPV is negative predictive value.

To facilitate comparison with another study⁶, we also considered the MSD cutoff (<94) that would yield a sensitivity of 90%. In our sample, 55.5% of participants had an MSD score <94. That cutoff yielded a specificity of 56% to detect a Bayley-III motor score <85; the positive predictive value was 0.34 and the false negative rate was 0.05.

Table 6 shows items on the Bayley-III cognitive and motor (gross and fine) scales, with the corresponding MSD item. Agreement for many gross motor items was good (κ=0.61 to 0.80)¹⁸, for example for rolling from back to front (κ=0.62), crawling (κ=0.65), and rising to stand (κ=0.78). In contrast, agreement for fine motor and cognitive items was substantially lower.

Table 6.

Agreement between provider and parent responses to similar Bayley-III and Motor and Social Development Scale items

Bayley-III item obtained by provider	Motor and Social Development scale item from parent questionnaire	n	Kappa
Gross Motor
Elevates truck while prone series: Extended arms	When lying on his/her stomach, has your child ever raised his/her head AND chest from the surface while resting his/her weight on his/her lower arms or hands?	60	0.22
Sits without support series: 5 seconds	Has your child ever sat alone with no help except for leaning forward on his/her hands or with just a little help from someone else?	77	0.33
Pulls up to sit	While lying on his/her back and being pulled up to a sitting position, did your child ever hold his/her head stiffly so that it DID NOT hang back as he/she was pulled up?	72	0.12
Rolls from back to stomach	Has your child ever rolled over on his/her own on purpose	75	0.62
Sits without support series: 30 seconds	Did your child ever sit for 10 minutes without any support at all?	79	0.61
Supports weight	Has your child ever been pulled from a sitting to a standing position and supported his/her own weight with legs stretched out?	45	0.39
Crawls Series: Crawl movement (forward progress of at least 5 feet)	Has your child ever crawled when left lying on his/her stomach?	46	0.65
Raises self to standing position	Has your child ever pulled him/her self to a standing position without help from another person?	65	0.78
Walks series: With support	Has your child ever walked at least 2 steps with one hand held or holding something?	73	0.61
Stands alone (at least 3 seconds)	Has your child ever stood alone on his/her feet for 10 seconds or more without holding on to anything or another person?	80	0.72
Walks series: Alone (at least 3 steps without support)	Has your child ever walked at least 2 steps without holding on to anything or another person?	111	0.69
Walks up stairs series: Both feet on each step, with support	Did your child ever walk up at least 2 stairs with one hand held or holding the railing?	130	0.31
Runs with coordination	Has your child ever run?	140	0.30
Walks up stairs series: Both feet on each step, alone	Has your child ever walked upstairs by him/her self w/out holding on to a rail?	132	0.43
Walks up stairs series: Alternating feet, alone	Has your child ever walked up stairs by him/herself with no help, stepping on each step with only one foot?	55	0.19
Fine Motor
Block series: whole hand grasp	Has your child ever held a moderate sized object, such as a block or rattle, in one hand?	54	−0.03
Food pellet series: thumb-fingertip grasp	Has your child ever picked up small objects, such as raisins or cookie crumbs, using only his/her thumb and first finger?	105	0.40
Imitates stoke series: horizontal	Has your child ever made a line with a crayon or pencil?	147	0.04
Imitates stroke series: vertical	Has your child ever made a line with a crayon or pencil?	142	0.08
Cognitive
Mirror image series: responds positively	Has your child ever seemed to enjoy looking in the mirror at him/her self?	27	−0.13
Searches for fallen object	Has your child ever looked around with his/her eyes for a toy which was lost and not nearby?	33	0.34
Child demonstrates relational play	Has your child ever played with several children at the same time?	90	0.06
Counts (one-to-one correspondence)	Has your child ever counted 3 objects correctly?	39	0.02
Counts (cardinality)	Has your child ever counted out loud up to 10?	11	−0.00

Discussion

We investigated the validity of the MSD scale for use in former preterm infants attending a high risk infant follow-up program. Our results support both internal validity and modest concurrent validity with the Bayley-III, a professionally administered test used commonly for preterm and VLBW infants in both clinical and research settings. Additionally, in our study the MSD was useful in detecting children with low Bayley-III scores, although the false positive rate was relatively high.

In previous population based surveys, the MSD has reflected developmental differences related to lower birth weight and gestational age^{19, 20}, higher maternal age²¹, and social factors such as parental marital status, positive parent-child interactions, and maternity leave duration^{19, 22}. Our study extends those findings by demonstrating concurrent validity of the MSD with formal neurodevelopmental testing in preterm infants, although the degree of correlation with Bayley-III scores was only modest.

While current clinical guidelines³ for VLBW children recommend at least 2 formal in-person neurodevelopmental assessments in the first 3 years of life, a substantial proportion of children who undergo testing will not have low scores. Additionally, many follow-up programs conduct in-person testing more frequently than the recommended minimum. It is possible that routine use of parent questionnaires such as the MSD combined with surveillance by pediatricians could improve the efficiency with which follow-up programs target their limited resources for in-person testing. In addition to screening tests such as the MSD, clinicians should also consider markers of biological and social risk, as well as the level of parent concern, to determine the frequency with which preterm VLBW children undergo formal testing.

We chose our MSD cutoff with the intent of detecting virtually all children with low Bayley-III scores because we did not want to “miss” any child who could benefit from services. At a sensitivity level of 100% for detecting a low Bayley-III motor score (<70), maximum specificity was 41%. Although this specificity level is rather low, based on our data, if we used the MSD to screen a hypothetical cohort of 1000 VLBW children, we would administer the Bayley-III to 610 children with MSD scores ≤95, thus saving the cost and inconvenience of in-person testing for the remaining 390. Of the 610 children referred for Bayley-III testing, 50 would have motor scores <70, whereas the remaining 560 would not (false positive MSD screens). By design, all children in our sample with a Bayley-III motor score <70 would be picked up by MSD screening using this cutoff. We could reduce the proportion of false positive MSD screens by lowering the cutoff, but doing so would also increase the number of children missed by screening (false negatives). Individual programs might choose different cutoffs depending on their goals and available resources for follow up testing.

Prior studies have considered parent questionnaires as screening tests for preterm infants. Skellern et al⁶. validated the Ages and Stages Questionnaire (ASQ) in children <31 weeks gestation, and reported that at a sensitivity level of 90%, the false positive rate was 60% for detecting low Griffiths scores (1 SD below mean); in younger children, the false positive rate was higher. Test characteristics of the MSD are comparable to the ASQ, but the MSD requires only one questionnaire as compared with the multiple age-specific forms of the ASQ. Other studies have reported poorer performance of parent questionnaires^{8, 23}.

Particularly relevant for surveillance and research is the extent to which parent questionnaire and “gold standard” neurodevelopmental measures are correlated. We found only modest correlations between the MSD and Bayley-III, with correlations somewhat stronger with the Bayley-III motor vs. cognitive scale. Only a few other developmental screening questionnaires have been validated in preterm or VLBW infants. Compared with the MSD, the Child Development Inventory (CDI) is more highly correlated with the Bayley-II (r=0.86 with the Mental Development Index, MDI) ⁹, but the CDI includes 300 items and takes 30–50 minutes to complete. In certain settings, minimizing participant burden with a short questionnaire such as the MSD (15 items) might outweigh the importance of maximizing the correlation with the gold standard. The Revised Prescreening Developmental Questionnaire is shorter that the CDI (30 items, 10–15 minutes) and a German study found good correlation (r=0.67) with the Griffiths Developmental Scale²⁴, but it lacks normative scoring based on U.S. population data.

The strength of the correlation between the MSD and Bayley-III reflects the agreement between parents and professionals regarding whether the child has achieved certain developmental milestones. In our study, agreement on individual test items was substantially better for gross motor items such as rolling and walking, as compared with fine motor items such as grasping, or cognitive items such as counting. In interpreting this finding, it is important to note that parents observe their children frequently, for long periods of time, and in a variety of settings, whereas professional assessment takes place in a medical setting over a short time window. It is possible that lack of agreement between parents and professionals is due to the fact that parents observe and report abilities that children do not demonstrate in the clinical setting. However, some parents may lack the ability to recognize more subtle developmental milestones, supported by our finding that for mothers with lower attained education, the correlation of the MSD with the Bayley-III cognitive scale was weaker than for the Bayley-III motor scale; there was minimal discrepancy for mothers of higher attained education. Educating parents to recognize developmental milestones and/or improving question design (e.g. with photographs or videos) might improve parents’ ability to report their children’s development more accurately. Based on the discrepancy we observed between more and less educated mothers, clinicians should use caution in interpreting parent questionnaires for children from less advantaged backgrounds. Similarly, caution should be used in interpreting questionnaires of younger children, as we also noted weaker correlations for children <18 months as compared with ≥18 months of age, particularly on cognitive items.

A strength of our study is that parents completed the MSD at 91% of clinic visits, however some parents indicated “I don’t know” or skipped questions; even one missing item precludes scoring the MSD. Although we did not have the resources in this study, other investigators should consider computer assisted administration of the MSD to improve completeness of responses¹². At some visits, Bayley-III testing was not performed as part of routine clinical care, so we could not include those visits in our analysis. At a small number of visits (5 for cognitive and 12 for motor), Bayley-III testing was not performed due to known impairments in the child. Thus, the prevalence of impairments in our study cohort is likely to be lower than in our overall clinic population, and in similar populations of former preterm or VLBW infants. Additionally, the mean Bayley-III cognitive score of 98 was higher than might be expected for a high risk group of children, although this finding is consistent with another report¹⁵ that the Bayley-III may underestimate impairment in preterm children. Finally, this was a cross sectional study. We did not examine the predictive validity of the MSD to identify children destined to have longer-term neurodevelopmental deficits. Regardless of MSD screening, surveillance by pediatricians and parents should be ongoing to detect deficits that many not emerge until preschool or later. Future studies might consider the extent to which repeated developmental screening measures in infancy predict later outcomes.

Conclusion

Our findings suggest that the MSD is a valid measure of neurodevelopment in preterm and VLBW infants, and may be useful in clinical, surveillance, and research contexts.

Supplementary Material

Supp FigureS1. Supplemental figure 1.

Motor and Social Development Scale items¹¹ by age group (modified from Hediger et al.²⁰)

Supp FigureS2. Supplemental figure 2.

Receiver operating characteristic curves for Motor and Social Development scale scores detecting low scores on the Bayley-III scales: (A) motor score <70; (B) motor score <85; (C) cognitive score <70; and (D) cognitive score <85