Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Aug 1.
Published in final edited form as: J Child Adolesc Psychiatr Nurs. 2013 Aug;26(3):10.1111/jcap.12042. doi: 10.1111/jcap.12042

Grandchild of the NBAS: The NICU Network Neurobehavioral Scale (NNNS) A Review of the Research Using the NNNS

Ed Tronick 1, Barry M Lester 2
PMCID: PMC3839620  NIHMSID: NIHMS516890  PMID: 23909942

Abstract

A review of the research on the NICU Network Neurobehavioral Scale (NNNS) is presented. The NNNS has good psychometric properties and reliability. Standardized norms are available for it. It was found to be sensitive to a wide variety of medical, exposure and demographic variables and has robust predictive validity. It will be useful for nurses for detecting neurobehavioral problems and management of the young infant.

Keywords: Newborn, NNNS, NICU Network Neurobehavioral Scale, neurobehavior, prenatal, intrapartum and neonatal risk factors, predictive validity


Early detection of children with developmental delays is needed for effective intervention (Lester B.M., et al., 2008; Majnemer, Rosenblatt, & Riley, 1994; Law, et al., 2003; Lester, et al., 2002; Boyle, Boulet, Schieve, et al., 2008). In this paper our aim is to present a neurobehavioral scale, the Neonatal Intensive Care Unit Network Neurobehavioral Scale (NNNS) (Lester & Tronick, 2004) that is establishing itself a valid biomarker for detecting at-risk infants and adding to our ability to predict their outcomes (for a more general review see, Noble & Boyd, 2012) We present a review of studies on the NNNS, including studies on its psychometric properties, its normative values and studies of its validity. We also suggest how the NNNS can aid clinicians managing the care of infants, individualizing interventions and working with parents on appropriate caretaking.

The importance of the NNNS for nursing has been well described by Sullivan, Miller, Fontaine, & Lester (2012). They note that newborn assessment has become more critical with the increased acuity and complex care needed especially in the neonatal intensive care unit (NICU). Nurses caring for high-risk infants use advanced assessment skills to identify the nature of infant instability and provide appropriate and timely intervention. Although nurses have multiple ways to evaluate the infant, the NNNS offers nurses in clinical practice and research a comprehensive neurobehavioral assessment especially suited to high-risk and premature infants. Adding the NNNS would provide a more comprehensive assessment to the high-risk full-term infant or preterm infant.

History

Before reviewing the research on the NNNS it seems useful to briefly present some of the history of neurobehavioral assessment (Lester & Tronick, 2004, for a more detailed account). One of the earliest reports of neurobehavioral assessment comes from the Inca’s. They wanted to predict whether or not a male child would be a warrior. They dug a hole and placed infants of about a year of age in it. Infants who could not crawl out of the hole were left to die, or were not selected to be warriors. This seems somewhat all or none and costly approach, but the outcome of the test had perfect predictability. More recent history is less extreme, but not quite as successful.

The NNNS evolved from a rich tradition of previous infant assessments. At the turn of the century, infant functioning was associated with the model of reflexes developed by Sherrington (1906) and Pieper (1928) began his exploration of the newborn’s reflexes. Critical support came from the finding that anencephalic infants had reflexes; that they operated only at a spinal level. Reflex models became supplemented by models of more generalized motor functioning (Thomas, 1960; Dargassies, 1977) Perhaps the most important advance was Prechtl’s (Prechtl & Beintema, 1964) introduction of the concept of state. States were differentiated, structured organizations of the brain, behavior and physiology. The same stimulus was found to produce different responses in different states indicating that the infant’s brain was active; not only reflexive. State, the organization of its components and their sequential organization over time, as such also became an assessable feature of the infant’s neurologic status. An intact brain was seen as capable of organizing states, whereas a damaged brain could not.

Furthermore, in the 1950s developmental researchers “discovered” the competent infant – the infant with abilities to regulate its own level of arousal and states, to habituate, to attend and orient, and have organized motor acts. Rosenblith (1961) developed a scale that incorporated qualities of infant orientation and habituation as well as tone and reflexes. Brazelton (1973) with Tronick developed the Neonatal Behavioral Assessment Scale (NBAS). It included items focused on the infant’s capacity to self-regulate and to interact with animate and inanimate stimuli. The NBAS emphasized how the infant’s individual differences affected caregiving and development.

Despite these advances a gap remained. A critical function of an examination is to provide norms against which to judge the performance of a group with a particular characteristic (e.g., low birth weight) or an individual. One need only to think of the critical importance of exams such as the Bayley Scales to appreciate importance of standardized biomarkers. A lack of normed instruments leads to sole reliance on clinical judgment, lack of comparability in research studies, inadequate clinical and research characterization of the infant’s functional status, lack of a standard language amongst clinicians and researchers and standard empirically derived norms for evaluating an infant’s neurobehavioral status. None of the early examinations were normed. Even the NBAS, though widely used, did not have norms, perhaps because of its procedural demand for eliciting optimal performance. Eliciting optimal performance confounds behavior elicited from the infant with the skill of the examiner (i.e., it actually measures the interaction of the infant and the examiner). To quote Brazelton, the “utilization of optimal performance is not desirable when one wants to assess the integrity of the infant to organize his or her own neurobehavior in a more-objective fashion (i.e., to assess what is “in” the infant) with a more-standardized examination. Nor is NBAS’ optimal-performance criteria appropriate for research studies, especially multisite studies, because of the training demands on the examiners, the data’s reflection of the examiner-infant interaction, and attendant demands on reliability” (Lester & Tronick, 2004). Lester and Tronick (2004) developed the NNNS to fill the gap left by a lack of a standardized reliable, valid and predictive biomarker of infant neurobehavior.

DESCRIPTION OF THE NNNS

A full description and manual for the NNNS is available (Lester & Tronick, 2004). It includes information on scoring, analytic techniques and training requirements. Briefly, NNNS is a comprehensive assessment of both neurologic integrity and behavioral functioning, including signs of stress. It assesses the full range of infant neurobehavioral performance (orientation to auditory and visual stimuli); infant stress (color changes, tremors, startles), neurologic functioning (reflexes, tone); some features of gestational age; self-soothing capacities; states and their organization. The neurobehavioral, reflex and stress items making up the NNNS can be treated as individual items, but given their large number 13 summary scores (i.e., orientation, habituation, hypertonicity, hypotonicity, excitability, arousal, lethargy, non-optimal reflexes, asymmetric reflexes, stress, self-regulation, quality of movement, handling) are typically used to summarize a clinical examination and are used for statistical analyses in research studies (see below on the development of neurobehavioral profiles). Procedurally it frames the assessment of the neurobehavioral items within states, the state-dependent administration of different items, as well as assessing the organization and regulation of states. It can be used with low and extremely high-risk infants once they are stable and well into the postnatal period. It has a standardized administrative format that “removes” the examiner from the behavior assessed, but it does not require rigid conformity to the standardized format. Rather it allows for flexibility of administration in relation to infants’ states and behaviors but also evaluates the extent to which a given examination conforms to the standardized administrative format as an additional marker of an infant’s neurobehavioral coherence.

RELIABILITY NNNS

The internal consistency of the NNNS was evaluated by Tronick and his colleagues (Fink, Tronick, Olson, & Lester, 2012). They found that Cronbach’s alpha ranged from .87-.90 for the summary scores. This consistency is well within the acceptable range but might be lower in a less homogeneous sample.

STANDARIZED NORMS FOR THE NNNS

Tronick and his colleagues (Fink, et al., 2012) carried out a study to develop standardized norms for the NNNS. The sample was made up of 344 clinically healthy newborns and mothers recruited from the well-child nurseries of a Boston hospital. Infants had to be on the well-newborn nursery and discharged from the hospital within four days. Infants were excluded if they had a circumcision within 12 hours of examination, an admission to the NICU (>12 hours), a major physical or neurological anomaly, were HIV positive, or had a positive toxicology screen for cocaine or heroin. Maternal exclusion criteria included: cognitive deficits, personality disorders or identified psychosis. Given the goals of the study procedures for the administration and reliability of the NNNS exceeded typical procedures; every examination was administered by two certified psychometrics.

Fink et al. (2012) presented means, standard deviations and 5th – 95th percentiles for the summary scores of the NNNS healthy infants (mean age at testing 32.3 ± 13.6 hours) (see Table 1). Table 2 presents the cut-off values and descriptions of the 5th, 50th and 95th percentiles. Of course, no sample can possibly be representative or fully generalizable to the entire population of newborns, but the choice of a large random sample of clinically healthy newborns from a large urban hospital’s well-child nursery with typical variation of demographic and medical status variables of clinically healthy infants can be considered typical and the normative values generated considered benchmarks for other samples.

TABLE 1.

Descriptive Data and Percentiles for Summary Scores of NNNS for Newborns (adapted from Fink et al, 2012).

NNNS scales Descriptive Statistics Percentiles
N Mean Std Min Max 5 10 25 50 75 90 95
Habituation 147 7.00 1.82 1.33 9.00 3.00 4.00 6.00 7.50 8.33 9.00 9.00
Attention 1 262 5.97 1.10 2.25 8.00 3.86 4.50 5.29 6.14 6.71 7.29 7.50
Arousal 1 341 4.34 0.68 1.40 6.40 3.00 3.43 4.00 4.43 4.86 5.00 5.14
Regulation 1 309 5.08 0.74 3.27 6.92 3.80 4.00 4.60 5.14 5.64 5.93 6.21
Handling 1 337 0.25 0.19 0.00 0.88 0 0 0.13 0.25 0.38 0.50 0.63
Quality of movement1 338 4.09 0.57 2.17 5.83 3.17 3.33 3.80 4.00 4.50 4.83 5.00
Excitability 344 4.80 2.36 0 10.0 1.00 2.00 3.00 5.00 6.00 8.00 8.00
Lethargy 344 4.01 2.75 0 12.0 1.00 1.00 2.00 3.00 5.00 9.00 10.0
Non-optimal reflexes 344 3.24 1.86 0 10.0 0 1.00 2.00 3.00 5.00 5.00 6.00
Asymmetry total 339 1.46 1.43 0 7.0 0 0 0 1.00 2.00 3.00 5.00
Hypotonia total 344 0.57 0.79 0 4.00 0 0 0 0 1.00 2.00 2.00
Stress abstinence 1 341 0.14 0.05 0 0.29 0.06 0.08 0.10 0.14 0.16 0.20 0.22
Stress abstinence (sum
items) 1
341 6.84 2.38 0 14.0 3.00 4.00 5.00 7.00 8.00 10.0 11.0
1

Scales that require a minimum number of items. Items were scored as follows: 0 = item did not occur, 1 = item did occur.

A higher score on each scale means a higher level of the construct.

Range: 1-9 = Mean of the variables included (Likert scale for variables)

Range: 0-1 = yes / no

Range 0-10, 0-15, 0-16 = Sum of all variables that went along with the construct (e.g. Lethargy, Hypertonia)

Table 2.

NNNS Percentiles and Interpretation (adapted from Fink et al, 2012)

NNNS Scale 5th Percentile 50th Percentile 95th Percentile
Habituation Score: 3 Score: 7.50 Score: 9
These infants show some response
decrement to stimulus presentations
of the light, rattle and bell over the
10 trials, but shutdown was not
complete.
These infants are able to shut down
their reactions (e.g., movements and
startles) within 4 to 5 stimulus
presentations of the light, rattle and
bell.
Infants with a summary score of 9 have
a response decrement after 1 to 2
stimuli.
Attention
(Orientation) 1
Score: 3.86 Score: 6.14 Score: 7.50
These infants show little
spontaneous interest in inanimate
and animate stimuli and general
quieting, blinking, change in
respirations, stilling and
brightening but no attempt to locate
the visual / auditory stimuli.
These infants are able to alert and shift
their eyes and to turn their head to the
stimulus (smoothly follows for two 30°
arcs with eyes and head)
These infants follow with eyes and
heas at least 60° horizontally and
sometimes vertically and head turns to
follow.
Arousal 1 Score: 3.0 Score: 4.43 Score: 5.14
These infants reach state 4 only
briefly, are mostly in state 3 or
lower and show little fussing during
two of the packages or items and
none to slight spontaneous / elicited
activity.
These infants are predominately in
state 4 but may reach state 5, and
evidence irritable fussing during 3 to 4
of the packages/items and moderate
spontaneous / elicited activity.
These infants reach state 6 after
stimulation once or twice, but are
predominantly in state 5 or lower, and
are fussy in 4 of the packages of items
and continuous consolable and
inconsolable spontaneous / elicited
activity.
Regulation 1 Score: 3.80 Score: 5.14 Score: 6.21
These infants increase head and
shoulder tone during pull–to-sit;
bring up their head, not maintained
at midline, but there are efforts to
right up; eventually molde into
arms, but after a lot of nestling and
cuddling by examiner; jerky and
smooth movements half the time
each with arcs of 45°; they have
some tone half the time, respond
with average tone less than half the
time when handled; Tremulousness
seen 1 or 2 times in states 5 or 6;
they have approximately 2 startles,
excluding the morrow reflex;
quality of alerting was variable;
responsiveness is brief but not
delayed; they have 9 to 10 state
changes over the course of the
examination; one brief success of
quieting self for period of ≥ 5
seconds; are able to self-quiet with
a hand to mouth reflex twice, no
insertion.
These infants increase head and
shoulder tone during pull–to-sit; bring
up their head once to midline and
maintaine it for at least 1 to 2 seconds;
molde and relaxe with some delay
when cuddled; have an equal mixture
of jerky and smooth movements;
respond for moderate duration but with
some delay; have good tone when
handled approximately 75% of the
time; have smooth movements with
arcs of 60°; are tremulousness around
3 or more times in states 5 or 6; have
approximately 3 startles; achieve an
alert state that was sustained; have 11
to 13 state changes over the course of
the examination; make several brief (5
seconds) successful attempts at self-
quieting; are able to self-quiet with a
hand to mouth reflex around 3 times,
but without any insertion.
These infants brinh up their head twice
after seated, then can keep it in
position ≥ 2 seconds; molde and relax
on their own without delay when
cuddled; have smooth movements with
arcs predominantly 60°; have good
tone when handled approximately 75%
of the time; are tremulousness 1 or 2 in
state 4; have approximately 4 startles;
alerting is less variable, responsiveness
is moderately sustained and not
delayed; have 14 to 15 state changes
over the course of the examination;
make an attempt to quiet self that result
in a sustained, successful quieting with
the infant returning to state 4 or lower
(for at least 15 seconds); are able to
self-quiet with a hand to mouth reflex
with a brief insertion.
Handling 1 Score: 0 Score: 0.25 Score: 0.63
Infants at the 5th percentile require
no maneuvers.
Infants at the 50th percentile require
two maneuvers.
Infants in the upper 95th percentile
require 6 or more maneuvers,
including swaddling, rocking / walking
or sucking / pacifier.
Quality of
movement 1
Score: 3.17 Score: 4.0 Score: 5.0
Infants at the 5th percentile have
jerky movements with signs of
overshooting and only some
smooth movements.
These Infants have an equal mixture of
jerky and smooth movements with arcs
to 45°. Infants range between scores of
3 and 5 (5th – 95th percentiles),
suggesting relatively low motor
control.
At the 95th percentile infants have
predominantly smooth movements, 60-
degree arcs half the time and only
some jerkiness.
Excitability Score: 1.0 Score: 5.0 Score: 8.0
These infants score positive on 1 of
the items (e.g., general tone,
consolability with intervention,
peak of excitement, rapidity of
build-up, irritability, spontaneous /
elicited activity, etc.).
Infants at the 50th percentile score
positive on 5 of the items.
Infants at the 95th percentile have 9
positive scores.
Lethargy Score: 1.0 Score: 3.0 Score: 10.0
These infants have a positive score
on 1 of the items (e.g. pull to sit,
orientation, defensive response,
alertness, general tone, peak of
excitement, irritability, lability of
states, etc.).
Infants at the 50th percentile have
positive scores on 4 of the items.
Infants at the 95th percentile show 9
signs of lethargy.
Non-optimal
reflexes
Score: 1 Score: 3.0 Score: 6.0
Infants at the 5th percentile
evidence 1 non-optimal reflex.
Infants at the 50th percentile evidence 3
non-optimal reflexes.
Infants at the 95th evidence around 7
non-optimal reflexes.
Asymmetry
reflexes
Score: 0 Score: 1.0 Score: 5.0
Infants at the 5th percentile
evidence no asymmetrical reflexes.
Infants at the 50th percentile evidence
no asymmetrical reflexes.
Those at the 95th percentile have
about3 asymmetrical reflexes.
Hypotonia Score: 0 Score: 0 Score: 2
Infants at the 5% percentile
evidence no signs of hypotonia.
Infants at the 50% percentile evidence
no signs of hypotonia.
At the 95th percentile infants are
hypotonic on 2 items.
Stress /
Abstinence 1,2
Score: 3 Score: 7 Score: 11
Infants at the 5th percentile show 3
signs of stress/abstinence.
Infants at the 50th percentile have 7
signs of stress/abstinence.
Infants at the 95th percentile have 11
signs.
1

Scales that require a minimum number of items. Stress Scale is the sum of 49 items, 40 items required. Items were scored as follows: 0=item did not occur, 1=item did occur.

2

The absolute number of positive items of the 49 Stress/Abstinence signs (0 / 1, yes / no) was between 3 and 11 (5th – 95th percentiles).

Taking into consideration the directionality of the scales, Fink et al. (2012) suggested that scores below the 10th percentile or above the 90th percentile indicate poor performance, be it a lack of responsiveness or exaggerated responsiveness. A considerable proportion of newborns did not achieve an alert state and of those who did, 25% had poor or moderate performance. At least 10% and as many as 25% of the newborns evidenced limited arousal and excitability, moderate to high levels of lethargy, poor to moderate regulation and quality of movement, and a moderate number of stress signs. The newborns had few non-optimal reflexes or asymmetrical reflexes. Hypertonicity essentially was not seen and its presence in a newborn may be of concern. These levels of performance, especially for attention, regulation, and arousal, do not conform to the picture of the typical well newborn as alert, regulated, and interactive held by many clinicians and parents.

Despite the limited range of medical and demographic factors, the normative scores still cannot be applied without consideration of other factors. Age at testing was related to performance. Similar to a recent report (Xu, Yolton, & Khoury, 2011), newborns younger than 24 hours were less aroused and excited and showed more non-optimal reflexes and more signs of lethargy. Importantly, the “usual suspects” of gestational age, infant gender, complications during labor/delivery, Cesarean section, neonatal and intrapartum risk factors, age of the newborn at testing, and ethnicity affect neurobehavioral performance. Greater gestational age was independently related to greater maturity of movement, better regulation and less excitability. Complications during labor/delivery were associated with more non-optimal reflexes and stress signs. Newborns that were born with a C-section had significantly more signs of stress than newborns of mothers, who delivered vaginally. The relations of these factors speak to their robustness in affecting the neurobehavioral performance of healthy infants even when the factors are in the non-clinical range. Indeed the findings suggest that clinical cutoffs may mask important relations that would be better captured by evaluating continuous effects of medical and demographic factors (e.g., the range GA in weeks) rather than categorical groups (preterm).

As found by Yolton et al. (2009) there were differences in performance related to ethnicity. Newborns of mothers with minority status had lower excitability and arousal scores, but had fewer stress signs, were better regulated and had better attention, with covariates taken into account. Low arousal and low excitability could speak to a lower state of neurobehavioral activation. Further evaluation of the relations of ethnicity to neurobehavioral performance is warranted.

The NNNS has been used to examine the behavior of a large sample of infants (419 newborns of adolescent mothers; 12–14 years vs. 15–17 years vs. 18–19 years) and presented Mean, SD, and 5th to 95th percentiles for each of 13 NNNS variables (de Moraes Barros, 2008). However, because the sample is at high risk it is a poor choice as normative sample, but useful for comparison to the Fink et al. (2012) sample though the younger age ranges are not in the sample of Fink et al., (2012). It found that infants of younger adolescent mothers were less lethargic than infants of older ones. The study is an example of the kind of studies, (i.e., large and well characterized sample varying along a particular dimension) that would be useful for understanding the effects of different clinical variables.

Researchers on infant neurobehavior and the factors that affect it have lacked standardized comparison data for other groups of interest; the Fink et al. (2012) study fills that gap. From a clinical perspective, these standardized values can now frame and discipline the clinical picture of the healthy newborn. Indeed an integration of clinical experience and the normative data will help identify infants at-risk for neurobehavioral problems. Moreover, the findings may facilitate more subtle identification of the relations of medical and demographic factors which may underlie problems in neurobehavioral performance and developmental outcome. Clinically healthy newborns such as the ones in the Fink et al. (2012) study are presumed to be at low risk. Yet, many exhibited dysfunctional behaviors at a critical developmental transition. Thus the norms may help identify infants who develop problems, but who were were considered normal during the newborn period; and that “normal ” group is the source of the largest number of infants who evidence developmental problems (2008). The norms bring a broader and more nuanced view of the neurobehavior of the typical newborn; a view that can be useful in addressing parental concerns based on inappropriate expectations and in guiding interventions.

RESEARCH USING THE NNNS WITH DIFFERENT CONDITIONS

A growing body of research is finding the NNNS to be sensitive to the effects of medical status and exposures on infant neurobehavioral organization.

In Utero Exposures

A number of studies have evaluated the neurobehavior of infants who have in utero exposure to illicit drugs or other toxic substances and the effects of treatments for withdrawal. In one of the first studies by Lester’s group (Napiorkowski et al., 1996) the NNNS was used to examine a small sample of 20 infants with multiple in utero exposures to cocaine, alcohol, marijuana, and cigarettes, 17 infants exposed to alcohol and/or marijuana and cigarettes, and 20 drug-free infants. Cocaine-exposed infants had higher tone and motor activity, more jerky movements, startles, tremors, back arching, and signs of stress than unexposed infants. They also showed poorer visual and auditory orienting. Both neurobehavioral patterns of excitability and lethargy were observed. The small numbers precluded multivariate analyses so the findings may have been due to the synergistic effects of cocaine with alcohol and marijuana. It also limits identification of the putative factor affecting performance, but is likely to describe what clinicians observe: infants identified as “cocaine exposed infants” almost always have multiple exposures.

In a subsequent more powerful study, Lester et al. (2002), evaluated a sample of almost 1400 1-month-old infants with prenatal cocaine exposure. They found that in utero exposure was related to lower arousal, poorer quality of movement, self-regulation, higher excitability, more hypertonia, and more non-optimal reflexes. Most effects remained with adjustment for covariates, birth weight, and social class as well as other drugs (alcohol, marijuana, and tobacco). There were also effects in this study for heavy cocaine exposure and separate effects for opiates, alcohol, marijuana, tobacco, and birth weight. Though the effect sizes were small, this study was clear in pointing to specific effects associated with in utero cocaine exposure, but also suggested that cocaine interacted with a variety of other factors.

One factor studied the interaction with cocaine and maternal depression, a morbid condition common in the post-partum period (Salisbury, Lester, Seifer et al., 2007). There was an unexpected depression by cocaine group interaction such that a depression effect was found only in the non-cocaine exposed infants, who had poorer self-regulation and more stress signs, excitability, and arousal. The authors suggest that in utero cocaine exposure may serve to buffer the effects of postpartum maternal depression on infant behavior. More generally the study provides a precautionary note to the simple presumption that cocaine and other risk factors always lead to negative effects.

Like cocaine, methamphetamine has become a public health concern. LaGasse et al., (2011) evaluated in utero methamphetamine exposure in a prospective study, of US and New Zealand cohorts. Typical of studies of illicit exposures; alcohol, marijuana, and tobacco exposures were present in both groups. The cross cultural nature of the study is exceptional. Exposure was associated with poorer quality of movement, more total stress, physiological stress, and CNS stress with more non-optimal reflexes in NZ, but not in the USA. Heavy methamphetamine exposure was associated with lower arousal and excitability regardless of site. Speaking to the complexity of in utero exposure, first trimester methamphetamine use predicted more stress signs in the infants and third trimester use more lethargy and hypotonicity. A similar pattern of under arousal, low tone, poorer quality of movement and increased stress was found regardless of “site”, but some effects were specific to only culture; the “site” effects and the timing effects are not fully understood.

Though there does not appear to be a study of in utero exposure to opioids, a number of studies have studied the effects of neonatal opioid withdrawal and its treatment. In a very small study (n=3; Johnson et al., 2001), the safety and efficacy outcome measures during and following prenatal buprenorphine exposure were evaluated. Results showed that buprenorphine in combination with comprehensive prenatal care was safe and effective and was associated with a ‘relatively mild’ neonatal abstinence syndrome (NAS), comprised primarily of tremors and hyperactive moro. In a later short term longitudinal study Jones, O’Grady, Johnson, Velez, & Jansoon (2012) the functioning of neonates prenatally exposed to methadone or buprenorphine were examined with the NNNS on days 3, 5, 7, 10, and 14 post-delivery. Compared to neonates who did not require medication to treat NAS, neonates receiving pharmacotherapy for NAS showed differences over time in quality of movement, excitability, and lethargy. Coyle, Ferguson, LaGasse, Liu, & Lester, (2005) found that opiate exposed infants treated with diluted tincture of opium and phenobarbital were more interactive, had smoother movements, were easier to handle, and less stressed than infants treated with diluted tincture of opium alone. The findings from these studies suggest that different treatment regimens have differential effects on infant neurobehavior and that the NNNS can be effective in evaluating different management regimes as well as in monitoring the clinical status of infants with in utero exposures.

Other more common exposures have been examined with the NNNS. Marijuana (de Moraes Barros et al., 2006) has been found to be associated with differences in arousal, regulation, and excitability between exposed and unexposed infants. Smoking has been found to alter neurobehavioral performance and is of particular clinical importance because of the incidence of smoking in pregnancy. In a prospective study, (Law, Stroud, LaGasse, Niaura, Liu, & Lester, 2003) tobacco exposed infants were found to be more excitable and hypertonic, required more handling and showed more stress signs, specifically in the central nervous system, gastrointestinal, and visual areas. Indicative of a dose response relationship higher maternal salivary cotinine values related to more stress signs including CNS and visual stress and higher excitability scores, and cigarettes per day during pregnancy was related to more stress signs including CNS and visual stress. In another short term longitudinal study (Stroud et al., 2009) examining the effects of maternal smoking during pregnancy on newborn neurobehavior at 10 to 27 days. Smoking-exposed infants showed greater need for handling and worse self-regulation and trended toward greater excitability and arousal relative to matched, unexposed infants. They had poorer self-regulation and needed more external handling. These studies suggest neurotoxic effects of common exposures such as prenatal marijuana and tobacco exposure on newborn neurobehavior.

Given what appear to be somewhat ubiquitous effects of illicit and licit substances on infant neurobehavior and the public health concern for toxicant exposure, an understudied area is the effect of exposures to toxicants on newborn neurobehavior. An exception is a study by Yolton and colleagues on the association of prenatal exposure to bisphenol A and select common phthalates with infant neurobehavior measured at 5 weeks (Yolton, Xu, Strauss, Altaye, Calafat, & Khoury, 2011). Bisphenol A (BPA) and phthalates are found in many manufactured products. The study compared the concentration of maternal urinary metabolites of bisphenol A and phthalates at two distinct time points in pregnancy (16w, 26w). Prenatal exposure to BPA was not significantly associated with neurobehavioral outcomes at 5 weeks, but higher total di-butyl phthalate metabolites at 26w were found to be associated with decreased handling and arousal, and increased self-regulation. In males, non-optimal reflexes were related to higher total di-2-ethylhexyl phthalate (DEHP) metabolites at 26w. Prenatal exposure to DBP was associated with changes in behavioral organization in 5-week-old infants. The findings clearly suggest the need for research on these and other toxicants.

Medical Variables

As might be expected neurobehavioral differences have also been identified in relation to a number of medical status variables and conditions. Congenital heart disease is associated with poor neurobehavioral performance on the regulation and stress scales compared (Massaro et al., 2011; Brown, Doyle, Bear, & Inder, 2006) de Moraes Barros et al. (2008) found that small for gestational age infants of adolescent mothers showed poorer quality of movements, more excitability and more signals of stress in association with sex of infant and variables related to delivery.

Interestingly epigenetic research by Marsit may provide some insight into the mechanism underlying these neurobehavioral effects associated with SGA (Marsit, Padbury, & Lester, 2012; Marsit et al., 2011). In the placenta, the HSD11B2 gene encoding the 11-beta hydroxysteroid dehydrogenase enzyme is responsible for the inactivation of maternal cortisol. It is regulated by DNA methylation and has been shown to be susceptible to stressors from the maternal environment. The study found that the extent of HSD11B2 methylation was greatest in infants with the lowest birth weights, and increased methylation was associated with poorer scores for the quality of movement on the NNNS. In the second study the goal was to identify links between altered gene imprinting in the placenta and infant neurobehavioral profiles. Significant associations were identified between classes of expression and quality of movement and handling scores. Multivariate regression demonstrated an independent effect of gene expression classes on these neurobehavioral scores.

In more recent work, increased methylation of NR3C1 in the placenta related to decrements in the newborn infant’s ability to track and follow animate and inanimate stimuli (Bromer, Marsit, Armstrong, Padbury, & Lester, 2012). These results suggest that alterations in imprinted gene expression in the placenta are associated with infant neurodevelopmental outcomes, and suggest a role for the placenta and genomic imprinting in the placenta beyond intrauterine growth regulation. Studies such as these, promise to provide mechanisms for phenotypic characteristics such as weight.

The findings may also have relevance to preterm birth whose NNNS performance has also been evaluated. Montirosso’s group (Montirosso, Del Prete, Cavallini, Cozzi & Gruppo di Studio NEO-ACQUA, 2009) as part of the NEONATAL ADEQUATE CARE for QUALITY of LIFE multi-center, longitudinal study investigated possible differences in the neurobehavior between term and preterm infants (gestational age < 30th week and/or birth weight < 1500 gr.). Preterm showed a greater number of non-optimal reflexes, poorer quality of movements, less attention. They were less able to regulate stress and had higher levels of stress. In another study, preterm infants (<1250 g or <30 weeks’ gestation) displayed altered neurobehavior for the majority of the NNNS scores (Brown, Doyle, Bear, & Inder, 2006). These neurobehavioral changes were associated with the total number of days of ventilation, intraventricular hemorrhage, and necrotizing enterocolitis. Of interest the researchers found positive perinatal influences on neurobehavioral performance: maternal antenatal steroids, female gender, and infants receiving breast milk at discharge home.

The same group (Brown et al., 2009) examined the relationship between very preterm infant neurobehavior at term and concurrent magnetic resonance-defined cerebral abnormalities. Using a composite neurobehavioral score of the NNNS and the Revised Hammersmith Neonatal Neurological Examination the neurobehavioral scores related to the total grade of white matter abnormality, and poorer neurobehavior related most strongly to white matter signal abnormalities and reduction in white matter volumes. Neurobehavior was not related to the total grade of gray matter abnormality, but was to delayed gyral maturation.

More surprising than findings of the effects of medical variables in the clinical range are the findings from Fink et al. (2012). Fink et al. found that NNNS performance was related to a large number of variables (e.g., gestational age, birth weight) even though the values were in the normal range and that their range was limited. Performance in different domains of neurobehavioral performance was related to age in hours at testing, intrapartum and neonatal risk factors, marital status, ethnicity, and other variables. These findings suggest the importance of taking into account the effects of non-clinical values when examining newborn neurobehavior and in guiding clinical management.

PREDICTION FROM NEWBORN NNNS TO LATER DEVELOPMENT

Using the NNNS summary scores the predictive validity of the NNNS has been demonstrated in a number of studies. Studies have found relations between NNNS scores and fetal behavior (Salisbury, Fallone, & Lester, 2005) and “difficult” temperament ratings at 3 mos. (Bagner et al., 2005). Of note as regards temperament however, is a study by DeSantis, Harkins, Tronick, Kaplan, & Beeghly (2011) which found that the NNNS was not associated with either the Early Infancy Temperament Questionnaire (EITQ) or the Infant Sensory Profile (ISP) filled out by mothers in the newborn period. The EITQ and ISP were associated with each other. The findings suggest that the NNNS is detecting unique qualities not captured by the questionnaires and also that the concepts of temperament and sensory integration overlap to a large extent.

More recently, Lester, Tronick and their colleagues have developed NNNS profiles of neurobehavioral performance which are being used with success in predictive studies (Liu et al., 2010). Liu et al. (2010) used latent profile analysis on a sample that included 1248 mother-infant dyads (42% born <37 weeks’ gestational age) participating in a longitudinal study. Five discrete behavioral profiles were reliably identified. Profiles 1-3 represent infants who showed average to superior performance across summary scores. Profile 5 reflected the most extreme deficits and was found in 5.8% of the infants. These infants showed the poorest attention, handling requirements, self-regulation, arousal excitability, hypertonicity, quality of movement, and stress signs. Infants in profile 4 were either hypertonic or hypotonic, had the most non-optimal reflexes, poor movement quality, and high stress signs.

Profile 5 was related to infants’ medical status and outcome. They were more likely to be born <32 wks., to have an abnormal CUS reading at 1-month, chronic neurological abnormalities and brain related illness or CP by age 3, more likely to have impaired Bayley MDI scores at 1 and 2 yrs., behavior problems at age 3, concept problems in school readiness at age 4, and lower IQ at 4½ yrs. than infants with profiles 1-4. Infants with profiles 4 or 5 were more likely to show motor impairments (Bayley PDI<70) scores at age 2, and more motor, concept, and language school readiness problems at age 4. These findings remained statistically significant even after adjustment for birth weight and SES, supporting the idea that the NNNS can identify impaired infants not identified by more routinely collected measures.

In another study using the profile approach Sucharewa, Khourya, Xub, Succopc, & Yolton (2012) used NNNS summary scores to identify 3 discrete NNNS profiles: social/easy going infants (44%), hypotonic infants (24%), and high arousal/difficult infants (32%). Though the technique is similar, these 3 profiles do not map directly onto the 5 profiles scores described in Liu et al. (2010). However, the high arousal/difficult infants profile is similar to profile 5, and hypotonic is similar to profile 4. Likely the difference in profiles between the two studies is due to differences in the two samples: one a high risk sample and the other low risk, and one tested at term and the other at 1 mos. post term. At age 3, infants with the hypotonic profile had poorer motor development and lower externalizing behaviors than the social/easy going and high/arousal difficult infants. That some relations between these profiles were found from the newborn period to 3 year old outcome in a low risk sample is actually quite remarkable.

A conceptual strength of the profiles is that they integrate information about an infant’s neurobehavioral functioning across systems which fits to the developmental concepts that neonatal neurobehavior is less differentiated and different systems are more tightly linked than the neurobehavioral systems in older children. For example, visual orientation is not as differentiated from motor maturity as it will become, and problems of motor organization may disrupt a host of functions including orientation and self-regulation. Thus profiles as the research suggests may be more ‘valid’ in characterizing an infant’s performance for research purposes and compliment the use of standardized norms in clinical contexts. Certainly it would be useful to develop norms for profile scores once they are firmly established. More research developing and comparing profiles in different groups is clearly warranted.

NICU MANAGEMENT

In an important and exceptional study with the NNNS, the Montirosso group (Montirosso, Del Prete, Bellu, Tronick, & Borgatti, 2012) examined the relation between the neurobehavior of very preterm infants and management of their care. A questionnaire, the NEO-ACQUA Quality of Care Checklist was developed to evaluate the level of developmental care and pain management in each of 20 NICUs. Infants from NICUs with high scores on the infant-centered care evidenced higher attention and regulation, less excitability and hypotonicity, and lower stress/abstinence NNNS scores than infants from low-care units. Infants from NICUs with high scores on infant pain management evidenced higher attention and arousal, lower lethargy and non-optimal reflexes NNNS scores than preterm infants from low-scoring NICUs. These findings are clearly suggestive of the need to utilize developmental and pain management techniques in the care of preterm infants and the need for further use of the NNNS to evaluate quality of care.

CONCLUSION

This review of the research on the NNNS indicates that it is valid, predictive and powerful biomarker of infant neurobehavior. The researchers using the NNNS have used in a growing variety of contexts and have developed new ways of analyzing. As Brazelton wrote: “The NNNS, with its more-complete analysis of neurobehavior, neurologic and stress responses, and regulatory capacities, as well as its more-standardized format and wide application to clinically normal and compromised infants of different gestational ages, offers us an important window into the newborn baby’s organization and disorganization.” (Lester & Tronick, 2007). For nursing, the utilization of the NNNS can facilitate detection and management of neurobehavioral problems and serve as vehicle for communication with parents about the status and care of their infant.

Acknowledgments

Support was provided by the following grants: Standardization of the NRN-Neurobehavioral Scale, National Institute Child Health & Human Development (R01HD37138) to E. Tronick

Footnotes

Financial disclosure and conflict of interest: The authors have declared that they have no financial relationship and conflict of interests relevant to this article.

Contributor Information

Ed Tronick, Department of Psychology, University of Massachusetts Department of Newborn Medicine Brigham and Women’s Hospital, Harvard Medical School.

Barry M. Lester, The Warren Alpert Medical School of Brown University Women and Infants Hospital of Rohde Island.

References

  1. Andre’-Thomas CY. The Neurological Examination of the Infant. Little Club Clinics in Developmental Medicine. National Spastics Society; London, United Kingdom: 1960. [Google Scholar]
  2. Bagner DM, Sheinkopf SJ, Miller-Loncar C, LaGasse LL, Lester BM, Liu J, Bauer CR, Das A. The effect of parenting stress on child behavior problems in high-risk children with prenatal drug exposure. Child Psychiatry & Human Development. 2009;40(1):73–84. doi: 10.1007/s10578-008-0109-6. PMCID: PMC2861499. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Boyle CA, Boulet S, Schieve LA, Cohen RA, Blumberg SJ, Yeargin-…Kogan MD. Trends in the Prevalence of Developmental Disabilities in US Children, 1997-2008. Pediatrics. 2008 Jun;127:1034–42. doi: 10.1542/peds.2010-2989. [DOI] [PubMed] [Google Scholar]
  4. Brazelton TB. Clinics in Developmental Medicine. No 50. JB Lippincott; Philadelphia, PA: 1973. Neonatal Behavioral Assessment Scale. [Google Scholar]
  5. Bromer C, Marsit CJ, Armstrong DA, Padbury JF, Lester B. Genetic and epigenetic variation of the glucocorticoid receptor (NR3C1) in placenta and infant neurobehavior. Developmental Psychobiology. 2012 Jun 19; doi: 10.1002/dev.21061. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Brown NC, Inder TE, Bear MJ, Hunt RW, Anderson PJ, Doyle LW. Neurobehavior at term and white and gray matter abnormalities in very preterm infants. Journal of Pediatrics. 2009 Jul;155:32–8. doi: 10.1016/j.jpeds.2009.01.038. [DOI] [PubMed] [Google Scholar]
  7. Brown NC, Doyle LW, Bear ML, Inder TE. Alterations in Neurobehavior at Term Reflect Differing Perinatal Exposures in Very Preterm Infants. Pediatrics. 2006;118(6):2461–2471. doi: 10.1542/peds.2006-0880. [DOI] [PubMed] [Google Scholar]
  8. Carvalho De Moraes Barros M, Guinsburg R, Mitsuhiro S, Chalem E, Ramos Laranjeira R. Exposure to Marijuana During Pregnancy Alters Neurobehavior in the Early Neonatal Period. The Journal of Pediatrics. 2006;149:781–787. doi: 10.1016/j.jpeds.2006.08.046. [DOI] [PubMed] [Google Scholar]
  9. Coyle MG, Ferguson A, LaGasse L, Liu J, Lester B. Neurobehavioral Effects of Treatment for Opiate Withdrawal. Arch Dis Child Fetal Neonatal Ed. 2005;90:F73–F74. doi: 10.1136/adc.2003.046276. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. de Moraes Barros MC, Guinsburg R, de Araujo Peres C, Mitsuhiro S, Chalem E, Laranjeira RR. Neurobehavioral profile of healthy full-term newborn infants of adolescent mothers. Early Human Development. 2008;84:281–287. doi: 10.1016/j.earlhumdev.2007.07.001. [DOI] [PubMed] [Google Scholar]
  11. DeSantis A, Harkins D, Tronick E, Kaplan E, Beeghly M. Exploring an integrative model of infant behavior: What is the relationship among termperament, sensory processing, and neurobehavioral measures. Infant Behavior & Development. 2011;34:280–292. doi: 10.1016/j.infbeh.2011.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Fink N, Tronick E, Olson K, Lester B. Healthy Newborns’ Neurobehavior: Norms and Relations to Medical and Demographic Factors. Journal of Pediatrics. 2012 doi: 10.1016/j.jpeds.2012.05.036. DOI 10.1016/j.jpeds.2012.05.036; PMID 22727876a. [DOI] [PubMed] [Google Scholar]
  13. Johnson RE, Jones HE, Jasinski DR, Svikis DS, Haug NA, Jansson LM, Lester BM. Buprenorphine Treatment of Pregnant Opioid-Dependent Women: Maternal and Neonatal Outcomes. Drug and Alcohol Dependence. 2001;63:97–103. doi: 10.1016/s0376-8716(00)00194-0. [DOI] [PubMed] [Google Scholar]
  14. Jones HE, O’Grady KE, Johnson RE, Velez M, Jansson LM. Infant neurobehavior following prenatal exposure to methadone or buprenorphine: Results from the Neonatal Intensive Care Unit Network Neurobehavioral Scale. Substance Use & Misuse. 2012;45(13):2244–2257. doi: 10.3109/10826084.2010.484474. [DOI] [PubMed] [Google Scholar]
  15. Law KL, Stroud LR, LaGasse LL, Niaura R, Liu J, Lester BM. Smoking During Pregnancy and Newborn Neurobehavior. Pediatrics. 2003;111(6):1318–1323. doi: 10.1542/peds.111.6.1318. [DOI] [PubMed] [Google Scholar]
  16. LaGasse LL, Wouldes T, Newman E, Smith LM, Shah RZ, Derauf C, Lester BM. Prenatal methamphetamine exposure and neonatal neurobehavioral outcome in the USA and New Zealand. Neurotoxicol Teratology. 2011 Jan-Feb;33:166–75. doi: 10.1016/j.ntt.2010.06.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Liu J, Bann C, Lester B, Tronick E, Abhik D, Lagasse L, Bada H. Neonatal Neurobehavior Predicts Medical and Behavioral Outcome. Pediatrics. 2010;125(1):E90–E98. doi: 10.1542/peds.2009-0204. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Lester B, Tronick E. The Neonatal Intensive Care Unit Network Neurobehavioral Scale (NNNS) Pediatrics. 2004;113:631–99. [PubMed] [Google Scholar]
  19. Lester BM, Tronick E, LaGasse L, Seifer R, Bauer CR, Shankaran S, Maza PL. The maternal lifestyle study: effects of substance exposure during pregnancy on neurodevelopmental outcome in 1-month-old infants. Pediatrics. 2002 Dec;110:1182–92. doi: 10.1542/peds.110.6.1182. [DOI] [PubMed] [Google Scholar]
  20. Lester BM, Miller RJ, Hawes K, Salisbury A, Bigsby R, Sullivan MC, Padbury JF. Infant neurobehavioral development. Seminars in Perinatology. 2011 Feb;35:8–19. doi: 10.1053/j.semperi.2010.10.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Majnemer A, Rosenblatt B, Riley P. Predicting outcome in high-risk newborns with a neonatal neurobehavioral assessment. American Journal of Occupational Therapy. 1994 Aug;48:723–32. doi: 10.5014/ajot.48.8.723. [DOI] [PubMed] [Google Scholar]
  22. Massaro A, Glass P, Brown J, Chang T, Krishnan A, Jonas R, Donofrio M. Neurobehavioral Abnormalities in Newborns with Congenital Heart Disease Requiring Open-Heart Surgery. Journal of Pediatrics. 2011;158:678–81. doi: 10.1016/j.jpeds.2010.11.060. [DOI] [PubMed] [Google Scholar]
  23. Montirosso R, Del Prete A, Bellu R, Tronick E, Borgatti R. Level of NICU Quality of Developmental Care and Neurobehavioral Performance in Very Preterm Infants. Pediatrics. 2012;129(5):1–9. doi: 10.1542/peds.2011-0813. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Montirosso R, Del Prete A, Cavallini A, Cozzi P, Gruppo di Studio NEO-ACQUA Neurobehavioral profile in a group of healthy preterm infants. Utilization of the NICU Network Neurobehavioral Scale (NNNS) Child Development & Disabilities - Saggi. 2009;3:96–116. [Google Scholar]
  25. Marsit C, Padbury J, Lester B. Placental 11-Beta Hydroxysteroid Dehydrogenase Methylation is Associated with Newborn Growth and Neurobehavioral Outcomes. PLoS ONE. 2012;7(3) doi: 10.1371/journal.pone.0033794. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Marsit CJ, Lambertini L, Maccani MA, Koestler DC, Houseman EA, Padbury JF, Chen J. Placenta-Imprinted Gene Expression Association of Infant Neurobehavior. The Journal of Pediatrics. 2011;160(5):854–860. doi: 10.1016/j.jpeds.2011.10.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Napiorkowski B, Lester BM, Freier MC, Brunner S, Dietz L, Nadra A, Oh W. Effects of In Utero Substance Exposure on Infant Neurobehavior. Pediatrics. 1996;98:71–75. [PubMed] [Google Scholar]
  28. Peiper A. Die Hirntatigkeit des Sauglings. Springer; Berlin, Germany: 1928. [Google Scholar]
  29. Prechtl H, Beintema D. Clinics in Developmental Medicine. No 12. Heinemann; London, United Kingdom: 1964. The neurological examination of the full-term newborn infants. [Google Scholar]
  30. Rosenblith J. The modified Graham behavior test for neonates: test-retest reliability, normative data and hypotheses for future work. Biology of the Neonate. 1961;3:174–192. doi: 10.1159/000239814. [DOI] [PubMed] [Google Scholar]
  31. Saint-Anne Dargassies S. Neurological Development in the Full-Term and Premature Neonate. Elsevier North Holland; New York, NY: 1977. [Google Scholar]
  32. Salisbury AL, Fallone M, Lester B. Neurobehavioral assessment from fetus to infant: the NICU Network Neurobehavioral Scale and the Fetal Neurobehavior Coding Scale. Mental Retardation and Developmental Disabililites Research Reviews. 2005;11(1):14–20. doi: 10.1002/mrdd.20058. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Sherrington C. The Integrative Action of the Nervous System. Scribner’s Sons; New York, NY: 1906. [Google Scholar]
  34. Stroud L, Paster R, Papandonatos GD, Niaura R, Salisbury AL, Battle C, Lagasse LL, Lester B. Maternal smoking during pregnancy and newborn neurobehavior: Effects at 10 to 27 days. The Journal of Pediatrics. 2009;154:10–16. doi: 10.1016/j.jpeds.2008.07.048. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Sucharewa H, Khourya JC, Xub Y, Succopc P, Yoltonb K. NICU Network Neurobehavioral Scale profiles predict developmental outcomes in a low-risk sample. Pediatric and Perinatal Epidemology. 2012;26:344–352. doi: 10.1111/j.1365-3016.2012.01288.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Sullivan MC, Miller RJ, Fontaine LA, Lester B. Refining neurobehavioral assessment of the high-risk infant using the NICU Network Neurobehavioral Scale. Journal of Obstetric, Gynecologic, & Neonatal Nursing. 2012;41:17–23. doi: 10.1111/j.1552-6909.2011.01322.x. DOI: 10.1111/j.1552-6909.2011.01322.x. [DOI] [PubMed] [Google Scholar]
  37. Xu Y, Yolton K, Khoury J. Earliest Appropriate Time for Administering Neurobehavioral Assessment in Newborn Infants. Pediatrics. 2011;127(1) doi: 10.1542/peds.2010-1121. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Yolton K, Khoury J, Xu Y, Succop P, Lanphear B, Bernert JT, Lester B. Low-level prenatal exposure to nicotine and infant neurobehavior. Neurotoxicology Teratology. 2009 Nov-Dec;31:356–63. doi: 10.1016/j.ntt.2009.07.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Yolton K, Xu Y, Strauss D, Altaye M, Calafat AM, Khoury J. Prenatal exposure to bisphenol A and phthalates and infant neurobehavior. Neurotoxicology and Teratology. 2011;33:558–566. doi: 10.1016/j.ntt.2011.08.003. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES