Predicting developmental delay in a longitudinal cohort of preschool children with single-suture craniosynostosis: is neurobehavioral assessment important?

Abstract

AIM

The aim of this study was to determine whether neurobehavioral assessment before and after cranial vault surgery can improve prediction of developmental delay in children with single-suture craniosynostosis (SSC), after accounting for clinical (SSC diagnosis and surgery age) and demographic ‘baseline’ variables.

METHOD

Children with SSC were referred by the treating surgeon or pediatrician before surgery. Neurobehavioral assessments were performed at ages of approximately 6, 18, and 36 months. Iterative models were developed to predict delay, as determined by one or more tests of cognitive, motor, and language skills at 36 months. We selected from groups of variables entered in order of timing (before or after corrective surgery), and source of information (parent questionnaire or psychometric testing).

RESULTS

Good predictive accuracy (as determined by area under the receiver operating characteristic curve [AUC]) was obtained with the baseline model (AUC=0.66), which incorporated age at surgery, sex, and socio-economic status. However, predictive accuracy was improved by including pre- and post-surgery neurobehavioral assessments. Models incorporating post-surgery neurobehavioral testing (AUC=0.79), pre-surgery testing (AUC=0.74), or both pre- and post-surgery testing (AUC=0.79) performed similarly. However, the specifity of all models was considered to be moderate (≤0.62).

INTERPRETATION

Prediction of delay was enhanced by assessment of neurobehavioral status. Findings provide tentative support for guidelines of care that call for routine testing of children with SSC.

Single-suture craniosynostosis (SSC) is a relatively common congenital condition (presenting in approximately 1 in every 2000 live births) in which one suture of an infant's skull is prematurely fused.¹ Children with SSC typically undergo cranial vault surgery within the first year of life because of concerns about increased intracranial pressure, and skull and brain morphology.¹ Despite the near universality of surgical correction in the USA, neurodevelopmental delays or impairments are consistently observed in children with SSC, and are likely to persist into later life with widely varying outcomes.¹ As a result, multidisciplinary care guidelines for SSC include neurodevelopmental screening beginning before age 4 months, with continued monitoring until age 8 years.² Identifying individuals with SSC at high risk of neurodevelopmental delay could lead to earlier intervention, which may mitigate or eliminate later problems. However, there has been little research on the effectiveness of such screening. Algorithms for identifying children with SSC at risk of developmental delay are lacking, and the optimal timing of developmental assessment is unclear (e.g. before or after corrective surgery).

Potential predictors of neurodevelopment in children with SSC include demographic variables commonly associated with early development: sex of child,³ family socioeconomic status (SES),⁴ and race.⁵ Also potentially important are clinical features of SSC, such as the affected suture and age at surgery. Recent evidence suggests that the most common suture fusion (sagittal) may be associated with the lowest risk of developmental delay.^1,6 Earlier surgery may reduce risk by limiting exposure to intracranial pressure;⁷ yet well-controlled studies provide inconsistent evidence for this association.⁸ Finally, later development may be best predicted by using standardized testing methods (e.g. the Bayley Scales of Infant Development [BSID])⁹ supplemented by parent measures of a child's behavioral adjustment.¹⁰ However, testing of mental or motor development early in life has had mixed predictive ability when used in other high-risk populations,¹¹ and the utility of such tests for predicting developmental outcomes in individuals with SSC is unknown.

In this longitudinal study of children with SSC, we developed models to predict neurodevelopmental delay at age 3 using data collected at two time points during infancy: before and after cranial vault surgery. We sought to determine whether the predictive ability of a baseline model, including easily obtained, pre-surgery demographic and clinical variables, could be improved by adding post-surgery parent-reported behavioral data and/or data obtained by developmental testing both before and after surgery.

METHOD

Participants included children with SSC who were part of a larger multicenter longitudinal cohort study.⁶ Eligible participants were enrolled between January 2002 and September 2006 from the Seattle Children's Hospital, the Children's Memorial Hospital in Chicago, the Children's Health Care of Atlanta, and St Louis Children's Hospital. In 2006, we also enrolled children diagnosed at the Children's Hospital of Philadelphia.

Children were referred to the project by the treating surgeon or pediatrician at diagnosis, and were eligible if they met the following criteria: they had SSC (sagittal, metopic, unilateral coronal, or unilateral lambdoid) confirmed by computed tomography; they had not yet undergone reconstructive surgery; and they were 30 months old or younger at recruitment. Children excluded from the project included those born preterm (at less than 34wk gestation); those with major medical or neurological conditions; those with three or more minor extracranial malformations;¹² and those with other major malformations. Children with genetic variants (e.g. mutations associated with SSC or polymorphisms) were not excluded. The institutional review board at each participating institution approved the study. We identified 333 individuals with SSC, 318 of whom were eligible. Parents of 52 eligible children declined participation, and we enrolled 266 (84% of those eligible; Fig. S1, supporting information published online). Children with lambdoid synostosis were excluded from analyses owing to small sample size.

Participants were assessed at three time points: time 1 (T1), after initial diagnosis and before cranial vault surgery (mean age [SD] 7mo 14d [5mo 4d]); time 2 (T2), 6 months or more after surgery with a target age of 18 months (mean age [SD] 19mo 24days [3mo 2d]); and time 3 (T3), 6 months or more after assessment at T2 with a target age of 36 months (mean age [SD] 37mo 23d [1mo 5d]).

Assessments

Bayley Scales of Infant Development (second edition) and Preschool Language Scale (third edition)

Trained psychometrists applied the Bayley Scales of Infant Development (second edition) (BSID-II)⁹ to evaluate participants’ cognitive (mental development index) and motor (psychomotor development index) development, and the Preschool Language Scale, third edition (PLS-3),¹³ to measure expressive communication and auditory comprehension. Both tests are norm-referenced, validated measures with mean (SD) standardized scores of 100 (15).

All assessments were video recorded for reliability purposes. Agreement on individual items was 98% or greater for the BSID-II mental development index and PLS-3 scales, and 93% for the BSID-II psychomotor development index.⁶ Invalid scores (e.g. due to child non-compliance or illness on the day of testing) were removed from analyses (n=4).

Child Behavior Checklist: ages 1.5 to 5 years

Parents completed the Child Behavior Checklist (CBCL),¹⁴ which measures both internalizing and externalizing behavior problems, at T2 and T3. Only maternal CBCL internalizing and externalizing behavior scores are included herein.

Statistical analysis

Developmental delay was defined as a score of less than 85 on one or more of the BSID-II or PLS-3 assessments at T3. Participants lacking assessment data from all scales were excluded. Potential predictors of delay at T3 were grouped according to the timing (T1 or T2) and burden of assessment. Four groups of predictors in five different models were examined (baseline model and models 1–4; see Fig. 1 for details). Predictors were examined in the following order: baseline variables, including demographic information, SSC diagnosis, and age at cranial vault surgery; BSID-II and PLS-3 test scores from T1; the CBCL given at T2; and BSID-II and PLS-3 test scores from T2. Individuals lacking data from any one of these predictors were excluded.

Figure 1.

Predictors of delay among preschool children with single-suture craniosynostosis (SSC) and predictors retained at each model iteration using the forward automated stepwise selection procedure (p<0.1 for inclusion). Predictors in ‘bold’ are those with p<0.01 identified at each stage, and entered in all subsequent models. ^aAll predictors modeled continuously, with the exception of sex (male/female), race (white/non-white), and SSC diagnosis (nominal, sagittal, metopic, or unicoronal). ^bModel 4 was a post hoc analysis examining addition of only post-surgery assessment to baseline variables, so not all predictors selected in previous model iterations were included. SES, socio-economic status; BSID-II MDI, Bayley Scales of Infant Development (second edition) mental development index; BSID-II PDI, Bayley Scales of Infant Development (second edition) psychomotor development index; PLS-3 AC, Preschool Language Scale (third edition) auditory comprehension; PLS-3 EC, Preschool Language Scale (third edition) expressive communication; CBCL externalizing, child behavior checklist externalizing behavior; CBLC internalizing, child behavior checklist internalizing behavior.

Automated forward stepwise regression was used to select predictors of delay with p values of 0.1 or less, starting with the baseline variables and iteratively adding groups of covariates in the order described above. At each step, all variables selected through the previous model development iterations were forced into the model so that sequential models were nested. Although all predictors within a group were eligible for inclusion in their model development step, only those selected into the model based on the p value criterion were retained and used in subsequent iterations.

In the event that both T1 and T2 BSID-II and PLS-3 test scores improved prediction, we wanted to evaluate whether prediction was more accurate when based on testing either before or after surgery. In a post hoc analysis, we compared the predictive accuracy of the final model 1 (which included only pre-surgery testing) with that of a model excluding pre-surgery testing, but including previously selected baseline predictors (e.g. demographic information) and T2 CBCL scores (model 4). This analysis allowed for the selection of post-surgical psychometric test scores. We also planned to compare model 4 with the final model 3 (including both pre- and post-surgery assessment data).

For all models, we calculated the empirical receiver operating characteristic curve, the area under the receiver operating characteristic curve (AUC),¹⁵ and the corresponding 95% confidence interval (CI) estimated in 300 bootstrap samples.¹⁶ Both the AUC and the receiver operating characteristic curve are measures of predictive accuracy. The receiver operating characteristic curve is a plot of sensitivity versus specificity, and the AUC is the area under this curve. An AUC of 0.5 reflects an inability to distinguish children who are delayed from those who are not, and an AUC of 1.0 indicates complete discrimination between children who are delayed from those who are not. Furthermore, we estimated the difference in AUC values between model iterations.

We identified a threshold predicted probability (‘cut-off score’) corresponding to a level of sensitivity of 0.8 (i.e. among children with a developmental delay or impairment, 80% would be correctly classified as being at ‘high risk’ for delay, and 20% incorrectly classified as ‘low risk’). At this cut-off, we also estimated specificity (i.e. the proportion of children classified as ‘low risk’ for delay among those without developmental concerns at age 3). We calculated each model's positive and negative predictive values (PPV and NPV, respectively), which are influenced by the prevalence of delay in the population. The PPV and NPV estimate how often the results of the predictive model (i.e. high or low risk of delay) reflect the participant's true condition (i.e. delayed or not). We also calculated the positive and negative likelihood ratios, which represent the ratio of the probability of a specific finding (high or low risk of delay) in children with delay to the probability of that same finding in children without delay. We used optimism-corrected specificities and a sensitivity of 0.8 to calculate the PPV, NPV, and the positive and negative likelihood ratios.

Because the same data set was used to develop and validate models, the results may be subject to overoptimistic estimates of model performance. We used the 0.632 bootstrap method,¹⁷ with the number of sampled data sets equal to sample size, to optimism-correct all reported results and measures of model performance, including the AUC and the difference in AUCs.¹⁸ To further assess improvement in model performance over earlier iterations, we examined reclassification tables,¹⁹ which illustrate the number of participants being correctly or incorrectly reclassified with the next model iteration.

All analyses were conducted using Stata 12.0 (StataCorp LP, College Station, TX, USA).

RESULTS

Valid scores on all predictor and outcome variables were available for 178 participants (60 females, 118 males). Among this sample, the distribution of demographic and clinical characteristics, and of test scores at T3, was similar to that of the larger sample of children in whom some valid scores were lacking (data not shown). Participants for whom complete data were available were mostly male, white, and of average to high average SES (Table I). Average scores across the BSID-II and PLS-3 ranged from 90.9 to 97.9 (Table SI, supporting information published online), and 42.1% of participants were classified as having a developmental delay or impairment (Table I).

Table I.

Pre- and post-surgery characteristics of children with single-suture craniosynostosis

	n	%
Total	178	100.0
Pre-surgery
Sex
Female	60	34
Male	118	66
Age group
<6mo	88	49
6-9mo	45	25
>9mo	45	25
Race
Non-white	40	22
White	138	78
SES (Hollingshead Four-Factor Index20)
I (high)	42	24
II	71	40
II	37	21
IV	23	13
V (low)	5	3
Site
Seattle	66	37
Chicago	66	37
St. Louis	20	11
Atlanta	26	15
Suture site
Sagittal	86	48
Metopic	44	25
Right unicoronal	29	16
Left unicoronal	19	11
Post-surgery (3y)
Delayed (score<85)
BSID-II mental development index	40	22
BSID-II psychomotor development indexa	45	26
PLS-3 auditory comprehension	38	22
PLS-3 expressive communicationb	42	24
Any developmental delay or impairment	75	42

^aTwo participants missing.

^bThree participants missing.

SES, socio-economic status; BSID-II, Bayley Scales of Infant Development (second edition); PLS-3, preschool language scale (third edition).

Sex, SES, and age at surgery were included in the baseline model (Fig. 1), resulting in an AUC of 0.66 (95% CI 0.60–0.74; Table II; see Table S2, supporting information published online, for model parameterizations). In the iteration selecting from T1 test scores, model 1 included the PLS-3 expressive communication, PLS-3 auditory comprehension, and BSID-II psychomotor development index parameters, in addition to baseline variables, with an AUC of 0.74 (95% CI 0.66–0.80). This was an improvement of 0.07 (95% CI 0.01–0.13) compared with the baseline model (Table II and Fig. 2). Model 2 tested the addition of T2 CBCL scores. However, neither of these scores met criteria for inclusion, resulting in a model identical to model 1. Selection from T2 psychometric test scores in the final iteration (model 3) resulted in inclusion of the BSID-II mental development index (cognitive development) and all previous predictors. Model 3 had an AUC of 0.79 (95% CI 0.73 to 0.85), an improvement of 0.04 (95% CI –0.01 to 0.08) compared with model 2 (Table II).

Table II.

Performance characteristics of models predicting developmental delay in 3-year-old children with single-suture craniosynostosis

Model iteration	AUC	95% CI		0.80 Sensitivity
		LB	UB	Specificity	PPVa	NPVa	LR+	LR–
Baseline modelb	0.66	0.60	0.74	0.37	0.48	0.72	1.27	0.54
Model 1c	0.74	0.66	0.80	0.55	0.57	0.79	1.78	0.36
Addition of pre-surgery psychometric testing
Model 2d	0.76	0.69	0.83	0.59	0.59	0.80	1.95	0.34
Addition of post-surgery questionnaire
Model 3e	0.79	0.73	0.85	0.62	0.60	0.81	2.11	0.32
Addition of post-surgery psychometric testing
Model 4f	0.79	0.74	0.85	0.59	0.59	0.80	1.95	0.34
Addition of post-surgery psychometric testing to baseline model

Model iteration	Differences in AUC	95% CI
		LB	UB
Model 1 vs baseline model	0.07	0.01	0.13
Model 2 vs model 1	0.02	0.02	0.02
Model 3 vs model 2	0.04	−0.01	0.08
Model 4 vs model 1	0.06	−0.004	0.12
Model 4 vs model 3	0.001	−0.03	0.03

^aCorresponding to prevalence of delay of 42.1%.

^bSex, SES and age at surgery.

^cSex, SES, age at surgery, T1 PLS-3 auditory comprehension, T1 PLS-3 expressive communication, and T1 BSID-II psychomotor development index.

^dSex, SES, age at surgery, T1 PLS-3 auditory comprehension, T1 PLS-3 expressive communication, and T1 BSID-II psychomotor development index. Sex, SES, age at surgery, T1 PLS-3 auditory comprehension, T1 PLS-3 expressive communication, T1 BSID-II psychomotor development index, and T2 BSID-II mental development index. Sex, SES, age at surgery, and T2 BSID-II mental development index.

AUC, area under the receiver operating characteristic curve; CI, confidence interval; spec, specificity; PPV, positive predictive value; NPV, negative predictive value; LR+, positive likelihood ratio; LR–, negative likelihood ratio; LB, lower bound; UB, upper bound; SES, socio-economic status; PLS-3, preschool language scale (third edition); BSID-II, Bayley Scales of Infant Development (second edition).

Figure 2.

Receiver operating characteristic curves comparing models for predicting delay in preschool children with single-suture craniosynostosis. The baseline model includes sex of child, socio-economic status, and age at surgery (AUC=0.66). Models 1 and 2 (overlapping) include variables from the baseline model, pre-surgery Preschool Language Scale (third edition) auditory comprehension and expressive communication, and T1 Bayley Scales of Infant Development (second edition) psychomotor development index (AUC=0.74 and 0.76, respectively). Model 3 includes all Model 2 variables and post-surgery Bayley Scales of Infant Development (second edition) mental development index (AUC=0.79). Model 3 more closely approaches the upper left corner (perfect sensitivity and specificity), indicating slightly better sensitivity and specificity than the other model iterations.

When all the baseline model variables and only T2 psychometric test scores were allowed (i.e. no T1 variables), the model retained only the BSID-II mental development index parameters. This model (model 4) had an AUC of 0.79 (95% CI 0.74 to 0.85), which was an improvement of 0.06 (95% CI –0.004 to 0.12) compared with model 1 (baseline variables and selected T1 psychometric test scores; Table II). There was no difference between the AUC values of model 4 and model 3, which contained selected baseline predictors and both T1 and T2 psychometric test scores (95% CI –0.03 to 0.03; Table II).

For the baseline model, a sensitivity of 0.80 corresponded to a cut-off point of 0.30 (i.e. a 30% probability of developmental delay, at or above which an individual would be classified as high risk). The corresponding specificity was 0.37, indicating correct low-risk classification of 37% of participants who were not developmentally delayed by age 3 (Table II). The cut-off point for model 1 was 0.33, resulting in a specificity of 0.55. Therefore, introduction of pre-surgery psychometric testing led to substantial gains in specificity. Model 3 had the best specificity at 0.62, with a cutoff of 0.18.

Assuming 42% prevalence of developmental delay or impairment (as observed in this sample; Table I) and sensitivity of 0.80 with corresponding specificity of 0.37, the baseline model optimism-corrected PPV was 0.48 and NPV was 0.72 (Table II). Models 1, 2, and 3 had similar PPVs (0.57, 0.59, and 0.60, respectively) and NPVs (0.79, 0.80, 0.81, respectively), and these were slightly higher than the PPV and NPV of the baseline model. Likewise, positive and negative likelihood ratios were similar in models 1, 2, and 3, and reflected an improvement in diagnostic accuracy compared to the baseline model.

At 0.80 sensitivity, the proportion of participants correctly classified as low risk was 39%, 54%, 54%, and 65% for the baseline model and models 1, 2, and 3, respectively (data not shown).

DISCUSSION

Testing of all infants and young children with suspected elevated risk of neurodevelopmental delay or impairment is impractical because of time and financial constraints. However, like preterm infants, those with SSC represent a group at particularly high risk of developmental concerns, with 40% or more showing signs of delay or impairment by 3 years of age, and nearly half with learning, developmental, or behavioral problems upon school entry.¹ A number of previous studies support the notion that such problems can be predicted using infant and toddler developmental assessments.^21–24 Moreover, children with SSC have early and frequent contact with the healthcare system, as most are diagnosed in the first months of life, undergo surgery by age 12 months, and receive follow-up medical examinations throughout their preschool years.

This is the first longitudinal study to examine development in a large cohort of children with SSC. Within this population, we sought to develop a model that would accurately predict developmental delay or impairment in children at age 3 years using information collected at earlier ages, ranging from data obtained from medical chart reviews to those obtained using more extensive measures such as parent questionnaires or psychometric testing. A baseline model using information on sex of child, family SES, and age at cranial vault surgery did surprisingly well, with an AUC of 0.66. Specifically, risk of delay was highest for male children, those from lower SES households, and those who received surgery at a later age. Information on SSC diagnosis and race did not contribute to accurate prediction of delay. These findings suggest that easily obtained demographic and clinical characteristics may be used for triage, identifying children who warrant closer developmental monitoring.

Inclusion of pre- and post-surgery measures of neurobehavioral status improved model accuracy. The best-performing model included the baseline variables listed above, as well as several test scores: pre-surgery tests of auditory comprehension, expressive communication, and motor development, in addition to post-surgery assessment of cognitive development (AUC=0.79). With the exception of motor scores at T1, all test scores were inversely related to outcome (i.e. higher scores were associated with a lower risk of delay, as expected). Using a sensitivity level of 0.80 as a cut-off to minimize the risk of ‘missing’ individuals at risk of delay, this final model resulted in a specificity of 0.62 (with a PPV of 0.60 and an NPV of 0.81). Compared with the baseline model, this model would appropriately classify an additional 15 children as low risk for every 100 evaluated, but with 22 incorrectly classified as high risk. Although the best-performing model included both pre- and post-surgery psychometric testing, pre-surgery testing may not be feasible because of the typically brief interval between diagnosis and surgery (i.e. often only 1 or 2wk). Furthermore, conducting multiple assessments may be impractical for both families and clinical programs. We therefore compared the predictive accuracy of pre- and post-surgery data with that of post-surgery data alone. We observed similar AUCs across all models. Compared with the model with only post-surgery testing, pre- and post-surgery testing would translate to only two additional children appropriately classified as low risk for every 100 evaluated, a difference too small to warrant multiple evaluations.

These observations provide support for recently developed practice parameters for the multidisciplinary care of children with SSC,² which call for neurodevelopmental screening and, when indicated, referral for early psycho-educational interventions. Although there are no published studies of early intervention services for children with SSC, data from studies of other high-risk groups indicate better outcomes for preschool children who received early interventions.^25–27 Further investigation using randomized trials is required to determine whether these findings extend to children with SSC.

With respect to the limitations of this study, we were unable to control for the developmental interventions that children received. Families were provided with their child's test results and encouraged to share these with their child's pediatrician to determine whether or not further assessment or intervention was needed. This may have elevated the proportion of individuals receiving developmental interventions (about 25%), which in turn may have mitigated later delays and artificially reduced the apparent predictive value of early assessment. With a larger prevalence of delay than observed in our study sample, these models would have had increased PPV and decreased NPV.

Given the lack of previous studies in children with SSC, an exploratory, model-building approach using stepwise regression was used to identify a parsimonious set of neurodevelopmental predictors. However, this approach has several limitations. Because stepwise methods rely on p value criteria for including predictors, the models may be unstable; for instance, such models may be influenced by small changes in the study sample.²⁸ They also may have limited power to select important covariates in small data sets²⁹ and potentially include random ‘noise’ covariates.^30,31 Therefore, our findings are specific to the study sample and the regression strategy used; application of alternative strategies, or even alternative p value thresholds, could result in different predictors and parameter estimates. For example, using the same method with a higher p value threshold would lead to inclusion of more predictors, while a lower p value would result in fewer predictors. Despite these limitations, a simulation study comparing selection methods reported that stepwise procedures led to smaller AUCs than other methods.²⁹

An important next step in the evaluation of our models to determine their external validity is replication using independent samples of children with SSC. Future studies should also examine broader models containing biological variables, such as cranial morphology and genotype. Developmental evaluations of older children with SSC are also needed to determine whether they ‘catch up’ with typically developing peers, or whether they show more permanent developmental impairments. This work is currently under way in our laboratory, with follow-up assessments of individuals with SSC and an unaffected comparison group at age 7 years. In the meantime, this study highlights a prediction model that can identify those at risk of delay or impairment, and supports the notion that tests of infant and toddler development, administered as early as 18 months of age, improve the early detection of developmental problems in children with SSC.

What this paper adds

• It demonstrates that predicting developmental delay in children with SSC can be achieved with moderate to high accuracy.

• It shows that evaluation of neurobehavioral status can improve the prediction of developmental delay.

• It indicates that neurobehavioral status measured before or after surgery is as predictive as if measured both before and after surgery.

ABBREVIATIONS

AUC

Area under the receiver operating characteristic curve

BSID

Bayley Scales of Infant Development

BSID-II

Bayley Scales of Infant Development (second edition)

CBCL

Child behavior checklist

NPV

Negative predictive value

PLS-3

Preschool Language Scale (third edition)

PPV

Positive predictive value

SES

Socio-economic status

SSC

Single-suture craniosynostosis