Abstract
Purpose
We investigated to what extent early motor development problems predict a future diagnosis of neurodevelopmental disorders (NDDs)/Early Symptomatic Syndromes Eliciting Neurodevelopmental Examinations (ESSENCE) by using a Bayesian network model (BN).
Subjects and methods
Subjects were the children who had participated in the 18- and 36-month checkups in two cities in Japan between April 2014 and March 2015. Their motor development data at the 4-, 10- and 18-month-checkups were collected with ethical consideration. The diagnosis was confirmed at the age of six, after regular assessment in all developmental areas at a neurodevelopmental clinic. The accuracy of prediction of NDD based on posterior probabilities determined using the BN was evaluated using the area under the receiver operating characteristic (ROC) curve (AUC). The posterior probability (the optimal cut-off value) yielding the maximum Youden Index (sensitivity + specificity – 1) is determined with the ROC curve, and the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and utility index (UI) were computed.
Results
BN models showed associations between early motor items and developmental coordination disorders, borderline intelligence/intellectual disability, and speech and language disorder. The ROC curve for any NDD had an AUC of 0.735. The posterior probability with the maximal Youden Index was 0.138; at the optimal cut-off value, the sensitivity, specificity, PPV, NPV, UI+, and UI- were 0.619, 0.761, 0.250, 0.940, 0.155 and 0.715, respectively.
Conclusion
We utilized a novel approach in detailing the associations between certain early motor problems and specific NDDs. We showed that the presence of motor development problems early in development increases the probability of a future diagnosis of any NDD. Still, the sensitivity of early motor development problems as a screening tool was not high enough to be the sole instrument for detecting NDDs. The need for a broad, holistic ESSENCE perspective when looking at the course of motor development problems was stressed.
Keywords: early symptomatic syndromes eliciting neurodevelopmental clinical examinations, early screening, child health checkups, receiver operating characteristic, area under the curve, posterior probability
Background
Neurodevelopmental disorders (NDDs) affect various aspects of development, such as cognition, motor ability, language, sociability, imagination, emotion, self-regulation, and learning, leading to various learning, behavioral and emotional problems. Gillberg1 proposed the concept of Early Symptomatic Syndromes Eliciting Neurodevelopmental Clinical Examinations (ESSENCE) and suggested that concerns (parental or specialist) persisting for at least a few months in early childhood in areas of (1) general development, (2) motor development, (3) responses to sensory stimuli, (4) communication/language, (5) activity/impulsiveness, (6) attention, (7) social interaction, (8) behavior, (9) mood, (10) sleep, (11) feeding, and (12) history of consciousness disturbance should always be taken “seriously” as they might likely be early signs of NDD. ESSENCE comprises autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), tic syndrome (TS), speech and language disorder (SLD), developmental coordination disorder (DCD), intellectual developmental disorder (IDD), and borderline intellectual functioning (BIF). Previous studies have shown that if there are obvious concerns in two or more of the ESSENCE areas referred to above, a comprehensive neurodevelopmental assessment and follow-up should be undertaken.2,3 Gillberg also emphasized that clinicians and researchers should always consider the coexistence/comorbidity of NDDs (if there is one, there is likely to be at least, one more) and the sharing of early signs (such that those signs might signal, eg, ADHD, ASD, DCD – one, two or three).1,4
NDDs that are not detected at an early stage and are left without proper management are likely to have various psychiatric and mental health consequences.5 In addition, NDDs have been reported to be associated with physical problems, such as obesity, chronic pain, accidents, and premature death.6–8 The detection of NDD signs as early as possible, even before a specific diagnosis is confirmed, is a significant public/community health issue.1
Motor developmental problems are considered early signs of NDD,9–16 and are assessed on one item in the ESSENCE-Questionnaire (ESSENCE-Q), which is used as a screening tool for ESSENCE.2,17,18
In Japan, child health checkups provide meaningful NDD screening opportunities. Relevant studies and reports have been published for health checkups of children who are older than 18 months and younger than 24 months (the “18-month-checkup”) and for children who are older than three years and younger than four years (the “36-month-checkup”).19,20 Most municipal governments in Japan offer health checkups for infants aged approximately 3–4 months (the “4-month-checkup”) and those aged about 9–10 months (the “10-month-checkup”). However, no studies on NDD screening have been conducted based on the data from these earlier health checkups. Motor developmental problems are relatively easy to recognize in these early health checkups for infants. If it is suggested that this is an early sign of NDD, it would be possible to initiate an appropriate intervention for NDD at an early stage, leading to an improved prognosis.
In the present study, we investigated to what extent early motor development problems predict a future diagnosis of NDDs using a Bayesian network model (BN). BN is a probabilistic graph representing a set of variables and their conditional dependencies using a directed acyclic graph (DAG). The nodes in the graph represent random variables. If an edge connects two nodes, it relates the probability of dependence from one node to the other. Each node is also assigned a Conditional Probability Table (CBT), recording its probability distribution. Using BN, the relationship between disease/disorder and clinical symptoms/signs can be expressed probabilistically.21 For example, if the probability of a diagnosis of NDD increases when information is given about an abnormality on a motor development item in the 4-month-checkup (posterior probability) compared to the probability when such information is not given (prior probability), it can be assumed that the item contributes to the diagnosis.
Methods
Subjects and “Explanatory Variables”
This is a general population-based retrospective cohort study of 501 children (256 boys and 245 girls) and their parents who participated in the 18- and 36-month checkups in two cities (Kami and Aki) in Kochi Prefecture between April 2014 and March 2015. In Kami City, 143 (75 boys and 68 girls) of 152 eligible children (79 boys and 73 girls) (participation rate, 94%) and 150 (74 boys and 76 girls) of 158 eligible children (77 boys and 81 girls) (95%) participated in the 18- and 36 month-checkup, respectively. In Aki City, 100 (54 boys and 46 girls) of 112 eligible children (63 boys and 49 girls) (89%) and 108 (53 boys and 55 girls) of 116 eligible children (54 boys and 62 girls) (93%) in the 18- and 36-month-checkup participated, respectively. Data on children’s motor development were collected with the parents’ written informed consent from child development records at the 4-, 10- and 18-month-checkups. The child development records included items based on interviews with mothers by public health nurses and records of direct observation by pediatricians. Published data on typical development, along with the Japanese version of the Denver Developmental Screening Test, were used, at both sites, as references to develop the items for interviews with mothers.
“Motor development abnormalities” were identified from (1) the nurse interview items about developmental milestones that the children had not achieved at the time of the interview with their mothers, and (2) items that the pediatricians, who performed neurological examinations during the health checkups, considered abnormal. Sex, birth weight, gestational age at birth and newborn status (the Apgar score one minute and five minutes after birth) are known to be associated with a diagnosis of NDD,22 hence they were also included as explanatory variables in the analysis.
All explanatory variables are listed in Table 1. Note that the codes in parentheses after an item distinguish the health checkups at different months (C means checkup, and the numbers are age in months) and items (M for items based on interviews with mothers and P for items based on observation by pediatricians, followed by item number; item numbers are in the order listed in the records). The dependent variable was the diagnosis of NDD (see below).
Table 1.
Basic Attributes and Motor Development Items Used as Explanatory Variables
| Name of Variable | Variable Code for Motor Development-Related Items | |||
|---|---|---|---|---|
| Basic attributes | Sex [256 boys, 245 girls] | n. a. | ||
| Birth weight [M = 3016.5, SD = 420.6] (<2500 g)* | n. a. | |||
| Gestational age [M = 38.7, SD = 1.43] (<37 weeks) | n. a. | |||
| Apgar score one minute after birth [M=8.7, SD=0.76] (<7) | n. a. | |||
| Apgar score five minutes after birth [M=9.1, SD=0.56] (<7) | n. a. | |||
| Motor development | 4-month-old health check-up | Item based on interviews with mothers | When you carry the child in your arms, is his/her head wobbly? (“Yes”) | C4_M1 |
| When the child lies on his/her belly, does he/she support his/her body with his/her own arms and lift his/her head? (“No”) | C4_M2 | |||
| “Does the child play with his/her hands placed close together in front of him/her?” (“No”) | C4_M3 | |||
| Does the child bring his/her hands to his/her mouth? (“No”) | C4_M4 | |||
| When you carry the child in your arms, do you feel that his/her body is unstable? (“Yes”) | C4_M5 | |||
| When you carry the child in your arms, do you feel stiffness and hardness in his/her arms and legs? (“Yes”) | C4_M6 | |||
| Neurological findings in examinations by pediatricians | Head control (The child cannot hold his/her head up to 45° when traction is applied to him/her.) | C4_P1 | ||
| Posture (opisthotonos, frog-leg posture) | C4_P2 | |||
| Muscle tone (hypotonia, hypertonia) | C4_P3 | |||
| Traction response (When traction is applied, the child stretches his/her limbs; his/her upper limbs remain stretched and his/her head does not follow.) | C4_P4 | |||
| Moro reflex (retained) | C4_P5 | |||
| Tonic neck reflex (strong asymmetrical tonic neck reflex) | C4_P6 | |||
| Vertical holding (pronation/extension of both hands; rigid extension of lower limbs) | C4_P7 | |||
| Horizontal holding (inverted U-shape, retroflexion) | C4_P8 | |||
| Prone position (The child cannot raise his/her face.) | C4_P9 | |||
| Grasping (The child keeps both hands clenched tightly, and you cannot make him/her grasp anything.) | C4_P10 | |||
| 10-month-old health check-up | Item based on interviews with mothers | Does the child crawl? (“No”) | C10_M1 | |
| Does the child pull to stand independently? (“No”) | C10_M2 | |||
| Does the child cruise? (“No”) | C10_M3 | |||
| Does the child stand on his/her own? (“No”) | C10_M4 | |||
| Does the child walk when you hold his/her hands to guide him? (“No”) | C10_M5 | |||
| Neurological findings in examinations by pediatricians | Pulling to stand (The child cannot pull him/herself to stand, or even if he/she can, he/she stands on tiptoes with his/her lower limbs extended and uses only the upper part of his/her body to support his/her own weight.) | C10_P1 | ||
| Posture (The child cannot assume a sitting position and cannot crawl.) | C10_P2 | |||
| Muscle tone (insufficient hip flexion; incapable of ankle dorsiflexion; ankle clonus) | C10_P3 | |||
| Traction response (When traction is applied, the child stretches his/her limbs; his/her upper limbs remain stretched and his/her head does not follow; he/she cannot assume an ultimate sitting position.) | C10_P4 | |||
| Vertical holding (crossed extension of lower limbs; equinus; lower limbs do not touch the floor.) | C10_P5 | |||
| Parachute reflex (no forward parachute reflex; bilateral difference; hands kept clenched) | C10_P6 | |||
| Hopping reaction (no reaction) | C10_P7 | |||
| Horizontal holding (no extension/elevation of the neck/trunk; weakness of limbs) | C10_P8 | |||
| Prone position (incapable of transition from the prone position to crawling) | C10_P9 | |||
| Grasping (The child cannot grasp anything; even if you make him/her grasp something, he/she lets it fall; he/she uses the entire palm to grasp.) | C10_P10 | |||
| 18-month-old health check-up | Item based on interviews with mothers | Can the child walk well? (“No”) | C18_M1 | |
| Can the child crouch down to pick up objects? (“No”) | C18_M2 | |||
| Can the child walk up a staircase if you hold his/her hand lightly? (“No”) | C18_M3 | |||
| Neurological findings in examinations by pediatricians | Walking (bilateral difference in step length; equinus; the child lands on his/her bottom after 2–3 steps; the child does not walk.) | C18_P1 | ||
| Gross motor skills (The child cannot walk up a staircase while you hold his/her hands to guide him/her.) | C18_P2 | |||
| Fine motor skills (The child cannot stack 2–3 blocks, pinch a small ball or scribble with a pencil even after he/she sees an example.) | C18_P3 | |||
Note: *Numbers and categories in parentheses indicate values that are considered abnormal.
Diagnosis of NDD
Pediatricians with extensive experience in neurodevelopmental assessment and NDDs examined each child at the final step of the 18- and 36-month-checkup. Furthermore, they detailed all records on child development since birth and reviewed the ESSENCE-Q records by the mothers, public health nurses, and psychologists. They evaluated these data to determine if more detailed neurodevelopmental examinations would be considered indicated. If deemed necessary, children underwent a secondary health checkup by a child psychiatrist with extensive clinical experience with young children. At the secondary checkup, the psychiatrist reviewed all records in detail, interviewed the mothers, and observed the children’s behaviors according to the ESSENCE-Q structure. Based on the assessment results, children suspected of NDD were referred to a neurodevelopmental clinic.
All developmental areas were assessed at the neurodevelopmental clinic to confirm the diagnosis. The mothers completed the ESSENCE-Q again, and specialists completed the ESSENCE-Q based on the observation of behaviors of the children and interviews with the mothers; in addition to the information recorded at the health checkups, all developmental areas under the ESSENCE umbrella, ie, general development, motor development, reaction to sensory stimuli, communication/language, activity/impulsiveness, social interrelatedness, attention, behavior, mood, sleep, eating and history of consciousness disturbance, were evaluated.2,3 The Kyoto Scale of Psychological Development 2001 (KSPD2001),23 the psychometric property comparable to Bayley Scales of Infant Development second edition (BSID-II), was used to evaluate cognitive function. Diagnostic Interview for Social and Communication Disorders (DISCO-10)24 was used to assess the development of sociability and communication. The Japanese version of the Strengths and Difficulties Questionnaire (SDQ) for parents,25 a tool for evaluating the mental health and development in children, such as behavior, emotion, and interpersonal relationships, in terms of the effects and duration of symptoms, distress in children, disabilities in various settings, burdens on others and other aspects, was also used. Assessments with the ESSENCE-Q, KSPD2001, and SDQ were repeated every six months. In addition, ADHD-RS26 and ASSQ27 were used as regular assessment tools for children with hyperactivity, impulsivity, and inattention problems and those suspected to have ASD, respectively. To diagnose DCD, occupational therapists and the child psychiatrist assessed children’s motor-perceptual performance based on clinical observations of behaviors such as standing, walking, throwing, and retrieving a ball. After the initial evaluation, the examinations were repeated every 1–3 months. Specialists continued unstructured clinical observations consistently over the study period. In addition, interviews with and reports from parents and nursery school/kindergarten teachers were conducted in almost every examination. The diagnosis was reviewed every six months; during every review, all the information obtained up to that point was checked and re-evaluated. The regular assessments outlined above were repeated until the child was six. After these procedures were completed, the diagnoses described were based on the diagnostic criteria for pediatric mental and neurodevelopmental disorders in the International Statistical Classification of Diseases and Related Health Problems-10/Diagnostic and Statistical Manual of Mental Disorders-IV. The diagnoses included in the analysis were ASD, ADHD, DCD, SLD, BIF/IDD, and others (including tics, reactive attachment disorder, and social anxiety disorder).
Analysis
All children were included except those with no data on motor development at all. The number of children with data available for any of the checkups was 467. Table 2 shows the numbers of eligible children, participants, and participants for whom health checkup data were available. The analysis was performed according to the steps shown in Figure 1.
Table 2.
Numbers of Eligible Children, Participants, and Participants for Whom Health Checkup Data Were Available
| Kami | Aki | |||||
|---|---|---|---|---|---|---|
| 18-Month-Checkup | 3-Year-Checkup | 18-Month-Checkup | 3-Year-Checkup | Total | ||
| Children eligible for checkups | 152 (79boys) | 158 (77boys) | 112 (63boys) | 116 (54boys) | 538 (273boys) | |
| Participants in checkups | ||||||
| Without diagnosis | 130 (65boys) | 133 (65boys) | 91 (47boys) | 90 (38boys) | 444 (215boys) | 501 (256boys) |
| With diagnosis | 13 (10boys) | 17 (9boys) | 9 (7boys) | 18 (15boys) | 57 (41boys) | |
| Children with 4-month-checkup data available | ||||||
| Without diagnosis | 107 (53boys) | 98 (51boys) | 38 (16boys) | 39 (17boys) | 282 (137boys) | 317 (164boys) |
| With diagnosis | 12 (9boys) | 11 (8boys) | 4 (3boys) | 8 (7boys) | 35 (27boys) | |
| Children with 10-month-checkup data available | ||||||
| Without diagnosis | 110 (55boys) | 101 (51boys) | 83 (44boys) | 68 (27boys) | 362 (177boys) | 408 (213boys) |
| With diagnosis | 13 (10boys) | 10 (8boys) | 8 (6boys) | 15 (12boys) | 46 (36boys) | |
| Children with 18-month-checkup data available | ||||||
| Without diagnosis | 130 (65boys) | 104 (54boys) | 91 (47boys) | 77 (31boys) | 402 (197boys) | 457 (237boys) |
| With diagnosis | 13 (10boys) | 16 (9boys) | 9 (7boys) | 17 (14boys) | 55 (40boys) | |
| Children with data available for any of checkups | ||||||
| Without diagnosis | 130 (65boys) | 111 (57boys) | 91 (47boys) | 80 (33boys) | 412 (202boys) | 467 (242boys) |
| With diagnosis | 13 (10boys) | 16 (9boys) | 9 (7boys) | 17 (14boys) | 55 (40boys) | |
Figure 1.
Flow chart showing the analysis process.
Abbreviations: ROC, receiver operating characteristic; PPV, positive predictive value; NPV, negative predictive value; UI, utility index.
Before a BN was constructed, a test of independence (Fisher’s exact test) was conducted to analyze the relationships of the diagnosis of NDD with sex, birth weight, gestational age at birth, newborn status (the Apgar score one minute and five minutes after birth), and motor development-related items. The test’s purpose was to compare items found to have significant relationships with the diagnosis and items included in the BN and ensure that the network included the necessary and sufficient items. The significance level was set at p< 0.05. Items for abnormalities found in no more than one child were excluded even if p < 0.05 because the statistical power was considered insufficient.
Sex and all developmental items rated by mothers regarding motor development, NDD diagnosis, and diagnosis type (BIF/IDD, ASD, ADHD, DCD, SLD, others) were taken as candidate nodes for the BN. The bnlearn package in R statistical software was used to construct the DAG structure and derive the CPT. Bnlearn was configured so that only explanatory variable nodes that were found to be associated with any diagnosis type by chi-square test (p < 0.05) were included in the DAG.
Evaluation of the Accuracy of NDD Diagnosis Prediction
The predictive accuracy of the derived BN for the diagnosis of NDD was evaluated as follows;
The posterior probabilities of being positive for NDD diagnosis were determined using the BN.
These posterior probabilities were compared with the actual diagnosis results, and the sensitivity and specificity obtained when a certain posterior probability was used as a cut-off value were shown graphically as a receiver operating characteristic (ROC) curve.
The accuracy of prediction of NDD diagnosis based on posterior probabilities determined using the BN was evaluated using the area under the ROC curve (AUC). As a screening test, AUC > 0.7 and AUC > 0.9 indicate acceptable and very high levels of accuracy, respectively.28–30
The posterior probability (the optimal cut-off value) yielding the maximum Youden Index (sensitivity + specificity – 1) is determined with the ROC curve, and the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and utility index (UI) were computed. The UI is a measure of the clinical usefulness of screening; positive UI (UI+) calculated as the sensitivity × PPV and negative UI (UI-) calculated as the specificity × NPV are used to evaluate the “rule-in accuracy” and the “rule-out accuracy”, respectively. The evaluation criteria are as follows: poor, UI± < 0.2; fair, 0.2 ≤ UI± < 0.4; moderate, 0.4 ≤ UI± < 0.6; good, 0.6 ≤ UI± < 0.8 and very good, UI± ≥ 0.8.3,31
We searched for a combination of motor development abnormalities for which the posterior probability exceeded the optimal cut-off value described above.
Ethical Considerations
The study was conducted after approval from the Institutional Review Board of the Kochi Prefectural Medical and Welfare Centre for Handicapped (No. 24–473). Written informed consent was obtained from the subjects’ parents/caregivers. This study was conducted according to the principles of the Declaration of Helsinki.
Results
Table 3 shows percentage of those with normal and abnormal values for each variable. Considering the age in months at the time of the checkups, the results were largely consistent with those of previous studies.32,33
Table 3.
Percentage of Those with Normal and Abnormal Values for Each Variable
| All Cases | Total | ||||
|---|---|---|---|---|---|
| Diagnosis (+) | Diagnosis (-) | ||||
| N | % | N | % | ||
| Sex | |||||
| Boy | 41 | 8.2% | 215 | 42.9% | 51.1% |
| Girl | 16 | 3.2% | 229 | 45.7% | 48.9% |
| Birth weight(wt) [g] | |||||
| wt < 2500 | 4 | 0.8% | 43 | 8.9% | 9.7% |
| wt ≥ 2500 | 52 | 10.7% | 385 | 79.5% | 90.3% |
| Gestational age(ga) | |||||
| ga < 37 | 4 | 0.8% | 23 | 4.8% | 5.6% |
| ga ≥ 37 | 51 | 10.6% | 403 | 83.8% | 94.4% |
| Apgar score one minute after birth (Ap1) | |||||
| Ap1 < 7 | 1 | 0.4% | 4 | 1.5% | 1.9% |
| Ap1 ≥ 7 | 35 | 13.2% | 226 | 85.0% | 98.1% |
| Apgar score five minute after birth (Ap5) | |||||
| Ap5 < 7 | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Ap5 ≥ 7 | 23 | 15.3% | 127 | 84.7% | 100.0% |
| 4-month-old health check-up | |||||
| Item based on interviews with mothers | |||||
| C4_M1 | |||||
| Abnormal | 5 | 1.6% | 45 | 14.6% | 16.2% |
| Normal | 29 | 9.4% | 229 | 74.4% | 83.8% |
| C4_M2 | |||||
| Abnormal | 5 | 1.6% | 26 | 8.4% | 10.0% |
| Normal | 29 | 9.4% | 249 | 80.6% | 90.0% |
| C4_M3 | |||||
| Abnormal | 6 | 1.9% | 42 | 13.5% | 15.4% |
| Normal | 26 | 8.3% | 238 | 76.3% | 84.6% |
| C4_M4 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 33 | 10.5% | 282 | 89.5% | 100.0% |
| C4_M5 | |||||
| Abnormal | 0 | 0.0% | 8 | 2.6% | 2.6% |
| Normal | 33 | 10.5% | 272 | 86.9% | 97.4% |
| C4_M6 | |||||
| Abnormal | 1 | 0.3% | 13 | 4.2% | 4.5% |
| Normal | 30 | 9.6% | 267 | 85.9% | 95.5% |
| Neurological findings in examinations by pediatricians | |||||
| C4_P1 | |||||
| Abnormal | 0 | 0.0% | 1 | 0.3% | 0.3% |
| Normal | 32 | 10.3% | 279 | 89.4% | 99.7% |
| C4_P2 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 33 | 10.5% | 280 | 89.5% | 100.0% |
| C4_P3 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 35 | 11.1% | 279 | 88.9% | 100.0% |
| C4_P5 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 35 | 11.1% | 279 | 88.9% | 100.0% |
| C4_P6 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 35 | 11.1% | 279 | 88.9% | 100.0% |
| C4_P7 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 35 | 11.1% | 279 | 88.9% | 100.0% |
| C4_P8 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 35 | 11.1% | 279 | 88.9% | 100.0% |
| C4_P9 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 35 | 11.1% | 279 | 88.9% | 100.0% |
| C4_P10 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 35 | 11.1% | 279 | 88.9% | 100.0% |
| C4_P11 | |||||
| Abnormal | 0 | 0.0% | 0 | 0.0% | 0.0% |
| Normal | 35 | 11.1% | 279 | 88.9% | 100.0% |
| 10-month-old health check-up | |||||
| Item based on interviews with mothers | |||||
| C10_M1 | |||||
| Abnormal | 6 | 1.5% | 25 | 6.2% | 7.7% |
| Normal | 39 | 9.6% | 335 | 82.7% | 92.3% |
| C10_M2 | |||||
| Abnormal | 11 | 2.7% | 47 | 11.6% | 14.3% |
| Normal | 35 | 8.6% | 313 | 77.1% | 85.7% |
| C10_M3 | |||||
| Abnormal | 18 | 4.4% | 94 | 23.2% | 27.6% |
| Normal | 28 | 6.9% | 266 | 65.5% | 72.4% |
| C10_M4 | |||||
| Abnormal | 33 | 8.2% | 239 | 59.6% | 67.8% |
| Normal | 12 | 3.0% | 117 | 29.2% | 32.2% |
| C10_M5 | |||||
| Abnormal | 23 | 5.7% | 127 | 31.3% | 36.9% |
| Normal | 23 | 5.7% | 233 | 57.4% | 63.1% |
| Neurological findings in examinations by pediatricians | |||||
| C10_P1 | |||||
| Abnormal | 3 | 0.8% | 8 | 2.2% | 3.1% |
| Normal | 35 | 9.8% | 310 | 87.1% | 96.9% |
| C10_P2 | |||||
| Abnormal | 1 | 0.3% | 0 | 0.0% | 0.3% |
| Normal | 37 | 10.4% | 318 | 89.3% | 99.7% |
| C10_P3 | |||||
| Abnormal | 1 | 0.3% | 0 | 0.0% | 0.3% |
| Normal | 37 | 10.4% | 318 | 89.3% | 99.7% |
| C10_P4 | |||||
| Abnormal | 1 | 0.3% | 0 | 0.0% | 0.3% |
| Normal | 37 | 10.4% | 318 | 89.3% | 99.7% |
| C10_P5 | |||||
| Abnormal | 1 | 0.3% | 0 | 0.0% | 0.3% |
| Normal | 37 | 10.4% | 318 | 89.3% | 99.7% |
| C10_P6 | |||||
| Abnormal | 2 | 0.6% | 5 | 1.4% | 2.0% |
| Normal | 36 | 10.1% | 314 | 88.0% | 98.0% |
| C10_P7 | |||||
| Abnormal | 1 | 0.3% | 0 | 0.0% | 0.3% |
| Normal | 37 | 10.4% | 318 | 89.3% | 99.7% |
| C10_P8 | |||||
| Abnormal | 1 | 0.3% | 0 | 0.0% | 0.3% |
| Normal | 37 | 10.4% | 318 | 89.3% | 99.7% |
| C10_P9 | |||||
| Abnormal | 1 | 0.3% | 0 | 0.0% | 0.3% |
| Normal | 37 | 10.4% | 318 | 89.3% | 99.7% |
| C10_P10 | |||||
| Abnormal | 1 | 0.3% | 0 | 0.0% | 0.3% |
| Normal | 37 | 10.4% | 318 | 89.3% | 99.7% |
| 18-month-old health check-up | |||||
| Item based on interviews with mothers | |||||
| C18_M1 | |||||
| Abnormal | 4 | 0.9% | 4 | 0.9% | 1.8% |
| Normal | 51 | 11.2% | 398 | 87.1% | 98.2% |
| C18_M2 | |||||
| Abnormal | 1 | 0.2% | 1 | 0.2% | 0.4% |
| Normal | 54 | 11.8% | 401 | 87.7% | 99.6% |
| C18_M3 | |||||
| Abnormal | 1 | 0.2% | 9 | 2.0% | 2.2% |
| Normal | 53 | 11.7% | 391 | 86.1% | 97.8% |
| Neurological findings in examinations by pediatricians | |||||
| C18_P1 | |||||
| Abnormal | 8 | 1.8% | 6 | 1.3% | 3.1% |
| Normal | 46 | 10.3% | 387 | 86.6% | 96.9% |
| C18_P2 | |||||
| Abnormal | 1 | 0.2% | 5 | 1.1% | 1.4% |
| Normal | 49 | 11.1% | 387 | 87.6% | 98.6% |
| C18_P3 | |||||
| Abnormal | 1 | 0.2% | 0 | 0.0% | 0.2% |
| Normal | 48 | 10.9% | 391 | 88.9% | 99.8% |
Diagnoses
The overall prevalence of diagnosed NDD rates in the 18-month-and 36-month-old populations was 9.1% (9.1% in Kami City and 9.0% in Aki City) and 13.6% (11.0% and 16.7%), respectively. The male: female ratio was 3.4:1 in the 18-month-old population and 2.2:1 in the 3-year-old population.
Table 4 shows the diagnoses of the children with NDD.
Table 4.
The Diagnoses of the Children with NDD
| NDD Diagnoses | Number of Cases | Average Age in Months (SD) at the Diagnoses | Notes |
|---|---|---|---|
| ADHD | 12(7boys) | 55.8(19.9) | |
| ASD | 9(7boys) | 40.7(14.0) | Three boys had an ESSENCE diagnosis before the checkup. |
| ADHD+SLD | 7(6boys) | ADHD: 50.0(15.2), SLD: 56.1(10.9) | |
| IDD/BIF | 6(5boys) | 52.8(13.2) | |
| ASD+ADHD | 4boys | ASD: 56.3(10.0), ADHD: 56.3 (10.0) | One boy had an ESSENCE diagnosis before the checkup. |
| ADHD+BIF | 3(2boys) | ADHD: 47.0(7.9), BIF: 54.7(12.5) | |
| SLD | 3(2boys) | 36.0(8.5) | One boy had an ESSENCE diagnosis before the checkup. |
| ADHD+DCD | 3boys | ADHD: 44.0(6.2), DCD: 41.7(2.5) | |
| RAD | 1girl | 53 | |
| ASD+DCD | 1boy | ASD: 33, DCD: 28 | |
| ASD+IDD | 1boy | ASD: 30, IDD: 37 | |
| ASD+SLD | 1girl | Unknown | The girl had an ESSENCE diagnosis before the checkup. |
| DCD+ADHD+BIF | 1girl | DCD: 34, ADHD: 42, BIF: 50 | |
| ASD+DCD+Tics | 1boy | ASD: 48, DCD: 48, Tics: 48 | |
| DCD+Tics+social anxiety disorder | 1girl | DCD: 45, Tics: 45, social anxiety disorder: 46 | |
| NDD(precise diagnosis unknown) | 3 (2 boys) | Unknown | One boy and one girl had an ESSENCE diagnosis before the checkup. The relevant medical institution did not consent to provide information for one boy. |
| Total:57(41boys) |
Abbreviations: ADHD, attention-deficit/hyperactivity disorder; ASD, autism spectrum disorder; BIF, borderline intellectual functioning; DCD, developmental coordination disorder; IDD, intellectual developmental disorder; NDD, neurodevelopmental disorder; RAD, reactive attachment disorder; SD, standard deviation; SLD, speech and language disorder.
Results from the BN
Figure 2 shows the DAG constructed using sex, health checkup results, diagnostic data, and diagnosed NDD of the subjects. Among the 10-month-checkup items, C10_M1(“Does the child crawl?”) was connected by arcs to C10_M2(“Does the child grasp things to pull him/herself up to stand independently?”), C10_M3(“Does the child cruise?”), C10_M5(“Does the child walk when you hold his/her hands to guide him?”), and C10_P1(“pulling to stand”) because they had associations, and C10_P1 was connected to BIF/IDD because the former showed the strongest association with the latter. Among 18-month-checkup items, C18_M1(“Can the child walk well?”) was connected by arcs to C18_M2(“Can the child crouch down to pick up objects?”) and C18_P1(“walking”) to represent their associations, and C18_P1 and C18_M2 were connected to BIF/IDD and DCD, respectively. Associations were also found among the NDD diagnosed, and arcs connected DCD and ADHD to others and ASD, respectively. The same data were used to derive the CPT of the BN. As an example, the CPT of the BIF/IDD node is shown in Table 5. This table shows the probabilities of being negative or positive for BIF/IDD when C10_P1 and C18_P1 values are given. To evaluate the accuracy of NDD prediction by this BN, we computed the probabilities of NDD predicted from sex and health checkup results of the subjects using the BN and prepared a ROC curve, as shown in Figure 3. Notably, 98 children (53 boys and 45 girls) who had two or more missing data among the health checkup items included in the BN were excluded from the accuracy evaluation to prepare the ROC curve. This curve had an AUC of 0.735 (95% CI: 0.648–0.821). The posterior probability with the maximal Youden Index (optimal cut-off value) was 0.138; at this value, the sensitivity, specificity, PPV, NPV, UI+, and UI- were 0.619 (0.472–0.766), 0.761 (0.715–0.808), 0.250 (0.167–0.333), 0.940 (0.911–0.968), 0.155 (0.015–0.294) and 0.715 (0.679–0.752), respectively. All combinations of sex and health checkup results with posterior probabilities exceeding this cut-off value are listed in Table 6.
Figure 2.
The constructed Bayesian network (BN). Node names starting with C denote the following health checkup items (M, items based on interviews with mothers; P, neurological findings of examinations by pediatricians).
Note: *As an example, the conditional probability table (CPT) of the BIF/IDD node corresponding to dependencies in an area enclosed by dashed lines is shown in Table 5.
Abbreviations: ADHD, attention-deficit/hyperactivity disorder; ASD, autism spectrum disorder; BIF, borderline intellectual functioning; DCD, developmental coordination disorder; IDD, intellectual developmental disorder; NDD, neurodevelopmental disorders; SLD, speech and language disorder; C4_M3, 4-month-old-checkup “Does the child play with his/her hands close together in front of him/her?”; C10_M1, 10-month-checkup “Does the child crawl?”; C10_M2, 10-month-checkup “Does the child grasp things to pull him/herself up to stand independently?”; C10_M3, 10-month-checkup “Does the child cruise?”; C10_M5, 10-month-checkup “Does the child walk when you hold his/her hands to guide him?”; C10_P1, 10-month-checkup “Pulling to stand”; C18_M1, 18-month-checkup “Can the child walk well?”; C18_M2, 18-month-checkup “Can the child crouch down to pick up objects?”; C18_P1, 18-month-checkup “Walking”.
Table 5.
Conditional Probability Table for Borderline Intellectual Functioning/Intellectual Developmental Disorder
| C10_P1 | C18_P1 | BIF/IDD | Probability |
|---|---|---|---|
| – | – | – | 0.994 |
| – | – | + | 0.006 |
| – | + | – | 0.5 |
| – | + | + | 0.5 |
| + | – | – | 1 |
| + | – | + | 0 |
| + | + | – | 0 |
| + | + | + | 1 |
Note: –: normal (negative), +: abnormal (positive).
Abbreviations: BIF/IDD, borderline intellectual functioning/intellectual developmental disorder; C10_P1, 10-month-old health check-up “Pulling to stand”; C18_P1, 18-month-old health check-up “Walking”.
Figure 3.
Receiver operating characteristic curve when the constructed Bayesian network is applied to the analysis set (N = 369). The value on the curve is the cutoff value (specificity, sensitivity) at the maximal Youden Index.
Table 6.
Combinations Where the Posterior Probability of Neurodevelopmental Disorders is Above the Cut-Off Value
| Sex (Male) | + | x | x | + | x | x | + |
|---|---|---|---|---|---|---|---|
| C4_M3 | + | + | x | + | + | x | + |
| C10_M1 | x | x | x | x | x | x | x |
| C10_M2 | x | x | x | x | x | x | x |
| C10_M3 | x | x | x | x | x | x | x |
| C10_M5 | x | x | x | x | x | x | x |
| C10_P1 | x | x | x | x | x | x | x |
| C18_M1 | x | x | x | x | x | x | x |
| C18_M2 | x | x | x | + | + | + | x |
| C18_P1 | + | + | + | x | x | x | x |
| NDD posterior probability | ≥ 0.568 | ≥ 0.543 | ≥ 0.531 | ≥ 0.415 | ≥ 0.383 | ≥ 0.362 | ≥ 0.180 |
Note: +: positive, -: negative, x: either positive or negative.
Abbreviations: C4_M3, 4-month-old health check-up “Does the child play with his/her hands close together in front of him/her?”; C10_M1, 10-month-old health check-up “Does the child crawl?”; C10_M2, 10-month-old health check-up “Does the child grasp things to pull him/herself up to stand independently?”; C10_M3, 10-month-old health check-up “Does the child cruise?”; C10_M5, 10-month-old health check-up “Does the child walk when you hold his/her hands to guide him?”; C10_P1, 10-month-old health check-up “Pulling to stand”; C18_M1, 18-month-old health check-up “Can the child walk well?”; C18_M2, 18-month-old health check-up “Can the child crouch down to pick up objects?”; C18_P1, 18-month-old health check-up “Walking”.
Discussion
The current study showed that the presence of motor development problems early in development increases the probability of a future diagnosis of any NDD, which supports the need for an ESSENCE perspective when looking at the course of delayed motor development from a different angle to previous studies.
Based on the BN results, the items associated with DCD (“Can the child walk well?” (C18_M1) and “Can the child crouch down to pick up objects?” (C18_M2)) may suggest that posture transition from walking and standing, and the development of balance function underlie the development of coordinated movement and motor dexterity. Endorsed items that might increase the posterior probability of being diagnosed with IDD/BIF were C18_M1, C18_P1 (“walking”), and all items interviewed from mothers in the 10-month-checkup except for C10_M4 (“Does the child stand on his/her own?”). This is in line with previous research suggesting an association between IDD and delayed motor development before the age of one.34 In summary, the results could indicate that general intellectual development assessments should be considered necessary whenever motor development abnormalities are suspected at the 10-month-checkup.
The item associated with SLD (“Does the child play with his/her hands close together in front of him/her?” (C4_M3)) may suggest an association between motor control development and language development, as has been demonstrated in previous research. However, the diagnosis of ASD was found to have no association with delayed motor development. In a systematic review, Lim et al35 reported a result of a meta-analysis showing a significant association between the diagnosis of ASD and abnormal motor development in the first 18 months of life. However, there were variations among the different studies. In most studies included in their review, structured scales with established validity and reliability were used to evaluate motor development. The motor development-related items used in the health checkups here have not been standardized, which may have influenced the outcome. The later diagnosis of ADHD was also found to have no association with delayed motor development.
Further studies will be necessary to establish a clearer picture. Of the 11 cases diagnosed with SLD, three had SLD only, seven had SLD and ADHD, and one had SLD and ASD. Speculatively, in these cases of SLD and other NDD comorbidities, the association between motor developmental problems and ADHD or ASD may be mediated by language delay.36
The ROC curve had an AUC of 0.735, which means that NDD screening based on problems in motor development had acceptable, albeit much less than perfect, accuracy. However, the sensitivity with the optimum cut-off value was 0.619, which is not sufficiently high for a screening tool. The UI- value was 0.715, and the rule-out accuracy was also inadequate. This indicates that, while an abnormality among the motor development-related items in a health checkup suggests that the child would be at risk of an NDD and requires careful follow-up, the absence of abnormalities among the motor development-related items does not necessarily mean that the child is free of the risk of NDD and still may require examinations in all areas of development included under the ESSENCE umbrella. Given the paucity of data and a relatively high frequency of missing values, combinations giving the posterior probabilities exceeding the cut-off value shown in Table 6 should be carefully interpreted. Nevertheless, the following finding that when motor development is compromised under the age of one, careful follow-up on various developmental aspects, including non-motor aspects, would still be considered valid.
The novelty and strength of this study is the use of BNs to quantify the effects of motor development problems on the diagnosis of NDD in the form of the posterior probability. However, a considerable number of missing values in the data used in this study may have had a major impact on the results. Given that the data used were originally not recorded for research, it is possible that no values were recorded just because no abnormalities were found in these items. Nonetheless, we had to treat them as missing values because no data was recorded.
Conclusion
The study showed that motor development problems early in development increased the probability of a future diagnosis of any of the NDDs, but was not sufficiently sensitive as a screening tool for NDDs. The need for a broad, holistic ESSENCE perspective when looking at the course of motor development problems was emphasized.
Acknowledgment
JSPS KAKENHI Grant Number 20K11101 supported this work.
Disclosure
All authors declare that they have no conflicts of interest in this work.
References
- 1.Gillberg C. The ESSENCE in child psychiatry: early symptomatic syndromes eliciting neurodevelopmental clinical examinations. Res Dev Disabil. 2010;31(6):1543–1551. doi: 10.1016/j.ridd.2010.06.002 [DOI] [PubMed] [Google Scholar]
- 2.Hatakenaka Y, Ninomiya H, Billstedt E, Fernell E, Gillberg C. ESSENCE-Q - used as a screening tool for neurodevelopmental problems in public health checkups for young children in south Japan. Neuropsychiatr Dis Treat. 2017;13:1271–1280. doi: 10.2147/NDT.S132546 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hatakenaka Y, Maeda M, Ninomiya H, Hachiya K, Fernell E, Gillberg C. ESSENCE-Q obtained in routine Japanese public child health check-ups may be a valuable tool in neurodevelopmental screening. Acta Paediatr. 2020;109(4):764–773. doi: 10.1111/apa.15029 [DOI] [PubMed] [Google Scholar]
- 4.Gillberg C. The Essence of Autism and Other Neurodevelopmental Conditions: Rethinking Co-Morbidities, Assessment, and Intervention. London: Jessica Kingsley Publishers; 2021. [Google Scholar]
- 5.Nylander L, Holmqvist M, Gustafson L, Gillberg C. Attention-deficit/hyperactivity disorder (ADHD) and autism spectrum disorder (ASD) in adult psychiatry. A 20-year register study. Nord J Psychiatry. 2013;67(5):344–350. doi: 10.3109/08039488.2012.748824 [DOI] [PubMed] [Google Scholar]
- 6.Hirvikoski T, Mittendorfer-Rutz E, Boman M, Larsson H, Lichtenstein P, Bölte S. Premature mortality in autism spectrum disorder. Br J Psychiatry. 2016;208(3):232–238. doi: 10.1192/bjp.bp.114.160192 [DOI] [PubMed] [Google Scholar]
- 7.London AS, Landes SD. Attention deficit hyperactivity disorder and adult mortality. Prev Med. 2016;90:8–10. doi: 10.1016/j.ypmed.2016.06.021 [DOI] [PubMed] [Google Scholar]
- 8.Asztély K, Kopp S, Gillberg C, Waern M, Bergman S. Chronic pain and health-related quality of life in women with autism and/or ADHD: a prospective longitudinal study. J Pain Res. 2019;12:2925–2932. doi: 10.2147/JPR.S212422 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Allely CS, Gillberg C, Wilson P. Neurobiological abnormalities in the first few years of life in individuals later diagnosed with Autistic Spectrum Disorder: a review of recent data. Behav Neurol. 2014;2014:1–20. doi: 10.1155/2014/210780 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Billstedt E, Gillberg IC, Gillberg C. Autism in adults: symptom patterns and early childhood predictors. Use of the DISCO in a community sample followed from childhood. J Child Psychol Psychiatry. 2007;48(11):1102–1110. doi: 10.1111/j.1469-7610.2007.01774.x [DOI] [PubMed] [Google Scholar]
- 11.Dahlgren SO, Gillberg C. Symptoms in the first two years of life. A preliminary population study of infantile autism. Eur Arch Psychiatry Neurol Sci. 1989;238(3):169–174. doi: 10.1007/BF00451006 [DOI] [PubMed] [Google Scholar]
- 12.Fernell E, Hedvall A, Norrelgen F, et al. Developmental profiles in preschool children with autism spectrum disorders referred for intervention. Res Dev Disabil. 2010;31(3):790–799. doi: 10.1016/j.ridd.2010.02.003 [DOI] [PubMed] [Google Scholar]
- 13.Gillberg C, Coleman M. The Biology of the Autistic Syndromes. 3rd ed. Mac Keith Press; 2000. [Google Scholar]
- 14.Hatakenaka Y, Kotani H, Yasumitsu-Lovell K, Suzuki K, Fernell E, Gillberg C. Infant motor delay and early symptomatic syndromes eliciting neurodevelopmental clinical examinations in Japan. Pediatr Neurol. 2016;54:55–63. doi: 10.1016/j.pediatrneurol.2015.09.008 [DOI] [PubMed] [Google Scholar]
- 15.Teitelbaum P, Teitelbaum O, Nye J, Fryman J, Maurer RG. Movement analysis in infancy may be useful for early diagnosis of autism. Proc Natl Acad Sci USA. 1998;95(23):13982–13987. doi: 10.1073/pnas.95.23.13982 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Trevarthen C, Delafield-Butt JT. Autism as a developmental disorder in intentional movement and affective engagement. Front Integr Neurosci. 2013;7:49. doi: 10.3389/fnint.2013.00049 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Gillberg C. ESSENCE-Q (Questionnaire). Available from: https://www.gu.se/en/gnc/gncs-resources/screening-questionnaires/essence-q-screening-questionnaire. Accessed May 4, 2022.
- 18.Stevanovic D, Knez R, Zorcec T, et al. ESSENCE-Q: Slavic language versions for developmental screening in young children. Neuropsychiatr Dis Treat. 2018;14:2141–2148. doi: 10.2147/NDT.S171359 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hashimoto T, Nishimura M, Mori K, Miyazaki M, Tsuda Y, Ito H. Jiheiseisyougai [Autistic disorders]. No to Hattatsu. 2005;37(2):124–129. Japanese. [PubMed] [Google Scholar]
- 20.Honda H, Shimizu Y, Nitto Y, et al. Extraction and Refinement Strategy for detection of autism in 18-month-olds: a guarantee of higher sensitivity and specificity in the process of mass screening. J Child Psychol Psychiatry. 2009;50(5):972–981. doi: 10.1111/j.1469-7610.2009.02055.x [DOI] [PubMed] [Google Scholar]
- 21.Shojaei Estabragh Z, Riahi Kashani MM, Jeddi Moghaddam F, et al. Bayesian network modeling for diagnosis of social anxiety using some cognitive-behavioral factors. Netw Model Anal Health Inform Bioinform. 2013;2(4):257–265. doi: 10.1007/s13721-013-0042-x [DOI] [Google Scholar]
- 22.Schieve LA, Tian LH, Rankin K, et al. Population impact of preterm birth and low birth weight on developmental disabilities in US children. Ann Epidemiol. 2016;26(4):267–274. doi: 10.1016/j.annepidem.2016.02.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Tatsuta N, Suzuki K, Sugawara T, Nakai K, Hosokawa T, Satoh H. Comparison of Kyoto Scale of Psychological Development and Bayley Scales of Infant Development second edition among Japanese infants. J Spec Edu Res. 2013;2(1):17–24. doi: 10.6033/specialeducation.2.17 [DOI] [Google Scholar]
- 24.Wing L, Leekam SR, Libby SJ, Gould J, Larcombe M. The diagnostic interview for social and communication disorders: background, inter-rater reliability and clinical use. J Child Psychol Psychiatry. 2002;43(3):307–325. doi: 10.1111/1469-7610.00023 [DOI] [PubMed] [Google Scholar]
- 25.Matsuishi T, Nagano M, Araki Y, et al. Scale properties of the Japanese version of the Strengths and Difficulties Questionnaire (SDQ): a study of infant and school children in community samples. Brain Dev. 2008;30(6):410–415. doi: 10.1016/j.braindev.2007.12.003 [DOI] [PubMed] [Google Scholar]
- 26.Takayanagi N, Yoshida S, Yasuda S, et al. Psychometric properties of the Japanese ADHD-RS in preschool children. Res Dev Disabil. 2016;55:268–278. doi: 10.1016/j.ridd.2016.05.002 [DOI] [PubMed] [Google Scholar]
- 27.Ehlers S, Gillberg C, Wing L. A screening questionnaire for Asperger syndrome and other high-functioning autism spectrum disorders in school age children. J Autism Dev Disord. 1999;29(2):129–141. doi: 10.1023/A:1023040610384 [DOI] [PubMed] [Google Scholar]
- 28.Greiner M, Pfeiffer D, Smith RD. Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Prev Vet Med. 2000;45(1–2):23–41. doi: 10.1016/S0167-5877(00)00115-X [DOI] [PubMed] [Google Scholar]
- 29.Swets JA. Measuring the accuracy of diagnostic systems. Science. 1988;240(4857):1285–1293. doi: 10.1126/science.3287615 [DOI] [PubMed] [Google Scholar]
- 30.Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol. 2010;5(9):1315–1316. doi: 10.1097/JTO.0b013e3181ec173d [DOI] [PubMed] [Google Scholar]
- 31.Mitchell AJ. How to design and analyse screening studies. In: Holland JC, Breitbart W, Jacobsen P, Lederberg MS, Loscalzo M, McCorkle R, editors. Psycho-Oncology. 2nd ed. New York: Oxford University Press; 2010:648–651. [Google Scholar]
- 32.Ueda R. Standardization of the Denver developmental screening test on Tokyo children. Dev Med Child Neurol. 1978;20(5):647–656. doi: 10.1111/j.1469-8749.1978.tb15284.x [DOI] [PubMed] [Google Scholar]
- 33.Ueda R. Child development in Okinawa compared with Tokyo and Denver, and the implications for developmental screening. Dev Med Child Neurol. 1978;20(5):657–663. doi: 10.1111/j.1469-8749.1978.tb15285.x [DOI] [PubMed] [Google Scholar]
- 34.von Wendt L, Mäkinen H, Rantakallio P. Psychomotor development in the first year and mental retardation–a prospective study. J Ment Defic Res. 1984;28(3):219–225. doi: 10.1111/j.1365-2788.1984.tb01013.x [DOI] [PubMed] [Google Scholar]
- 35.Lim YH, Licari M, Spittle AJ, et al. Early motor function of children with autism spectrum disorder: a systematic review. Pediatrics. 2021;147(2):e2020011270. doi: 10.1542/peds.2020-011270 [DOI] [PubMed] [Google Scholar]
- 36.Wu YT, Tsao CH, Huang HC, Yang TA, Li YJ. Relationship between motor skills and language abilities in children with autism spectrum disorder. Phys Ther. 2021;101(5):pzab033. doi: 10.1093/ptj/pzab033 [DOI] [PubMed] [Google Scholar]