Validation of the Infant-Toddler Checklist as a Broadband Screener for Autism Spectrum Disorders from 9 to 24 Months of Age

Abstract

The purpose of this study was to examine the validity of the Infant-Toddler Checklist (ITC) as a broadband screener to detect infants and toddlers with communication delays including ASD from a general population sample. The ITC was used to screen 5,385 children from 6–24 months of age. Three surveillance methods were used to detect children with possible ASD and diagnosis was confirmed at a mean age of 3 years. Positive and negative predictive values support the validity of the ITC for children 9–24 months of age but not 6–8 months. Of 60 children diagnosed with ASD, 56 had a positive screen on the ITC; parent concern increased with child age from less than half reporting concern from 6–15 months and nearly three-fourths at 21–24 months. Implications for improving early detection of ASD are discussed.

There is a pressing need to improve early detection of autism spectrum disorders (ASD) so that families can access intensive, appropriate intervention services as early as possible because research suggests that earlier intervention maximizes children’s outcomes (National Research Council, 2001). Because there is currently no biological marker for ASD, screening and diagnosis must be based on behavioral features (Filipek et al., 1999; Johnson & Myers, 2007). In spite of the severity of the behavioral characteristics of ASD, the mean age for diagnosis in the US is between 3 and 5 years and many children are not diagnosed until school age (Filipek et al., 1999; Mandell, Novak, & Zubritsky, 2005). Most parents of children with ASD recall concerns about their child’s development in the first 2 years of life based on retrospective reports (Chawarska et al., in press; Young, Brewer, & Pattison, 2003). The average delay between initial evaluation and diagnosis of ASD was 13 months in a study of surveillance records (Wiggens, Baio, & Rice, 2006). Closing the gap in time between initial parent concern, initial evaluation, and age of diagnosis would greatly reduce the age of entry into intervention.

The American Academy of Pediatrics (AAP) recently published a policy statement and technical report for the identification and evaluation of children with ASD (Johnson & Myers, 2007). In addition to routine developmental surveillance at every preventative visit, this report recommended that all children be screened with a standardized broadband screening tool at 9-, 18-, 24- and 30-month visits, and an ASD-specific screening tool at the 18- and 24-month visits. There is a growing body of research on early red flags of ASD in the 1^st and 2^nd years of life from retrospective parent report and home video analyses and from prospective research on siblings of children with ASD and on general population samples. However, there is currently a paucity of research validating broadband screeners that detect children with ASD between 9 and 30 months or ASD-specific screeners at 18 and 24 months. The new AAP practice guidelines make the need for validated screening tools more pressing. This paper will provide a brief review of research on the accuracy of screeners for children with ASD that have been administered to general pediatric samples and then present results of a population-based study with a broadband screener to detect children with communication delays including children with ASD.

Challenges of Screening for ASD in Infants and Toddlers

Screening for ASD may target high-risk populations, such as children referred to early intervention systems and younger siblings of children with ASD, utilizing ASD-specific screeners. Screening for ASD may also target the general pediatric population utilizing either ASD-specific screening tools designed to identify children at risk for ASD or a two-stage approach using broadband screening tools designed to detect children at-risk for a variety of developmental disorders including ASD as the first stage and an ASD-specific screener as the second stage. Screeners may be based on parent report and/or interactive observational measures. It is important that the screening tool be validated on the population that it will be used with (i.e. general or high risk) because the accuracy will vary with the method used.

Studying the validity of screening tools for infants and toddlers with ASD poses unique and significant challenges since most children are not diagnosed until late preschool or school-age. The age of diagnosis of ASD reflects the under-identification of young children with developmental disabilities by the early intervention system under the Individuals with Disabilities Education Act. According to the 27th Annual Report to Congress (US DOE, 2005), 11.2% of elementary-school children receive special education services. In contrast, 5.8% of preschool children receive special education and 2.2% of infants and toddlers receive early intervention services. Thus, only 20% of children who qualify for special education at school-age are identified and receive early intervention under age 3. Therefore, ASD-specific screening of high-risk samples referred to the early intervention system would likely miss the majority of children with ASD as infants and toddlers. This underscores the critical role of the primary care provider in improving early detection of ASD as well as other developmental disorders.

Accuracy of Screening Tools for Infants and Toddlers with ASD

This section will provide a brief review of screening tools that have been administered to general population samples, include children less than 24 months of age, and have published psychometric information about the accuracy of detecting children with ASD. Any screening tool should have strong psychometric features to support its accuracy in identifying at-risk children who need further evaluation. Sensitivity (true positives), specificity (true negatives), positive predictive value, and negative predictive value provide particularly important information about the accuracy of screening tools. To be considered psychometrically sound, a screening tool would minimally need to report sensitivity and specificity. Meisels (1989) recommended that both sensitivity and specificity be no less than 80% for developmental screening of young children; however, he noted that a “75% sensitivity ratio is considerably less favorable than a 75% specificity proportion” (p. 579). Adjusting cutoffs to increase sensitivity will decrease specificity and vice versa, and therefore, one should not be considered without the other. There are not recommended standards for positive and negative predictive value because they are related to the base rate of a disorder. That is, the higher the prevalence rate of the disorder, the greater the probability that a positive result will be correct and the higher the positive predictive value. In screening a general population for relatively low incidence disorders such as ASD, even an instrument with a sensitivity and specificity of .80 will yield a poor positive predicative value (Clark & Harrington, 1999).

There are three autism-specific screening tools that have been used with a general population sample. The Checklist for Autism in Toddlers (CHAT; Baird et al., 2000; Baron-Cohen, Allen, & Gillberg, 1992; Baron-Cohen et al., 1996), consisting of 9 items reported by parents and 5 items observed by a health professional at the 18-month developmental checkup, was the first to be studied. Baird et al. (2000) reported on a follow-up at age 7 years of 16,235 children screened with the CHAT at a mean age of 18.7 months. At follow-up at age 7 years, 94 cases of ASD were identified. The CHAT correctly identified 33 children, which is a rate of 2.03 per 1,000, well below the expected prevalence rates. These findings indicate that the CHAT has a specificity of 97.7% but a sensitivity of 35.1% and positive predictive value of 8.1% (Baird et al., 2000), and missed more children at 18 months who were later diagnosed with ASD than it detected. The poor sensitivity and corresponding high false negative rate indicate that the CHAT is not a valid screening tool at 18 months. It should not be relied on as an accurate screener and likely does not merit the time in a pediatric practice.

The Modified Checklist for Autism in Toddlers (M-CHAT; Robins, Fein, Barton, & Green, 2001; Robins, & Dumont-Mathieu, 2006; Kleinman et al., in press) consists of 23 parent report questions using the original 9 items from the CHAT as a basis. The MCHAT can be downloaded from www.firstsigns.org or http://www2.gsu.edu/~wwwpsy/faculty/robins.htm. The M-CHAT was initially studied on 1,122 children from a general pediatric sample at age 16–30 months and had 3 positive screens (Robins et al., 2001). This is a rate of 2.7 per 1,000 based on the low-risk sample, which is only slightly better than the CHAT with children slightly older and still below the expected prevalence rates. They estimated that sensitivity was 97%, specificity was 99%, and positive predictive power was 80%, but more accurate measures cannot be determined until a follow-up study is conducted as with the CHAT. In a more recent replication study, Kleinman et al. (in press) administered the M-CHAT to 3,309 children from a general pediatric sample at a mean age of 20.5 months. A telephone interview was administered to caregivers of children with a positive screen to review failed items at a mean age of 22.7. They reported that 189 children had a positive screen initially and 31 after the phone interview, with a positive predictive value of 11% for the M-CHAT and 65% for the M-CHAT combined with the telephone interview. They detected 20 children later diagnosed with ASD from the general pediatric sample. This is a rate of 6.0 per 1,000, which is near current ASD prevalent estimates.

In conclusion, the M-CHAT questionnaire alone without the telephone interveiw, even at a mean age of 20.5 months does not appear to have better positive predictive value than the CHAT at a mean of 18.7 months with a general pediatric sample. Kleinman et al. concluded that the M-CHAT should only be used in combination with an interview with a general pediatric sample in order to reduce false positives and avoid unnecessary referrals and parent concern. It is noteworthy that the M-CHAT was more promising with a high-risk sample at a mean of 24.3 months and the interview did not improve positive predictive value sufficiently to warrant the time with the high risk sample. It is premature to judge sensitivity and specificity of the M-CHAT until a more thorough follow-up study is conducted to carefully detect possible missed cases.

The Early Screening of Autistic Traits Questionnaire (ESAT; Dietz, Swinkels, Daalen, Engeland, & Buitelaar, 2006) is a 14-item two-stage screening instrument designed for use at 14–15 months of age. A pre-screening instrument with 4 ESAT items was designed for use at well-baby clinics as the first stage of screening. A Dutch screening study pre-screened 31,724 children selected from a random population sample at a mean age of 14.91 months and 370 children screened positive; 255 or 69% agreed to participate in a second screening stage during a home visit using the 14-item ESAT. They detected 18 children with ASD from the positive screens, indicating a positive predictive value of 25% and a rate of 0.57 per 1,000, well below the expected prevalence rates. The false positives included children with other developmental delays. These findings do not provide support for the validity of the ESAT as an ASD-specific screener for a general population sample. The low number of children with ASD detected may be partly due to the young age that they were screened. Some children with ASD may not show detectable features at 14–15 months as documented in prospective studies of younger siblings (Landa, Holman, & Garrett-Mayer, 2007).

There is only one broadband screener that has been studied to detect children with ASD, albeit preliminary. The Infant-Toddler Checklist (ITC; Wetherby & Prizant, 2002; Wetherby et al., 2004) is one component of the Communication and Symbolic Behavior Scales Developmental Profile (CSBS DP; Wetherby & Prizant, 2002) and is designed as a broadband screener for communication delays. The ITC can be downloaded from www.firstsigns.org and http://firstwords.fsu.edu/toddlerChecklist.html. The ITC includes 24 questions with 3 to 5 choices about developmental milestones of social communication. It also asks the following question about concerns: “Do you have any concerns about your child’s development?”, and if yes, to describe the concerns. The Flesch reading ease score is 84.0, which would be easily understood by an average 12-year old and the Flesch-Kincaid grade level is 4.9, which is based on a U.S. grade level and corresponds to an age of 10 to 11 years. The ITC is a standardized tool that, in addition to screening cutoffs, has standard scores at monthly intervals from 6 to 24 months based on a normative sample of over 2,188 children (Wetherby & Prizant, 2002).

Wetherby et al. (2004) reported on a preliminary study of 3,021 children from a general population sample screened with the ITC between 6 and 24 months through the longitudinal research of the FIRST WORDS^® Project. Children performing in the bottom 10^th percentile on the ITC and randomly selected children performing within normal limits were invited for a communication evaluation using the CSBS DP Behavior Sample during the second year of life. This is an interactive structured observation of social communication that is norm-referenced and was videotaped. Red flags of ASD were rated from the behavior samples of 36 children with communication delays, 18 who received a diagnosis of ASD at 3 years of age and 18 with developmental delay (DD) in which ASD was ruled out, and 18 children with typical development (TD). Seventeen of the 18 children in the ASD group or 94.4% had a positive screen on the ITC, 15 in the DD group or 83.3%, and 2 in the TD group or 11.1%. Sensitivity of the ITC was estimated at 88.9% when the ASD and DD groups were combined and increased to 94.4% when only the ASD group was examined with the TD group. Specificity was 88.9%. These results suggest that the ITC has high sensitivity and specificity (both 88.9%) for catching toddlers at risk for ASD and other developmental delays from a general pediatric sample. However, a follow-up study is needed to examine the validity of the ITC on a larger sample with more systematic surveillance methods to determine how many children with ASD may have been accurately detected or missed. The ITC is a broadband screener, and therefore, a positive screen indicates that the child is at-risk for a communication delay but does not differentiate a child with ASD from a child with other developmental problems.

There is a critical need for further research to develop and validate screening tools for ASD in very young children. These findings suggest that it may be more accurate to use a broadband screener followed by an ASD-specific screener to detect children with ASD at 18–24 months from a general pediatric sample. If a broadband screener is to be used as a first stage to an ASD-specific screener, then further research is needed on broadband screeners to document that they actually catch children with ASD along with children with other developmental disorders. The first aim of this study was to estimate the positive and negative predictive value of the ITC to detect children with communication delays, including children with ASD, from a general population sample of 5,385 children. The second aim was to document the percentage of positive screens and parents with concern reported on the ITC and the developmental characteristics of children later diagnosed with ASD from this general population sample.

Method

Participant Recruitment

The participants were studied prospectively through the FIRST WORDS^® Project screening of a general population sample of 5,385 children recruited from healthcare and childcare agencies to identify children under 24 months with communication delays using the CSBS DP (Wetherby & Prizant, 2002). The participants were consecutive children screened who were born between June, 1997 and November, 2003 and were between 4 and 10 years of age at the time of this study. This is a cumulative sample and includes the sample reported by Wetherby et al. 2004. Informed consent was appropriately obtained from parents of all participants. Figure 1 illustrates the steps involved in screening and evaluation beginning with the pool of 5,385 children who were screened with the Infant-Toddler Checklist. Each step involved in detecting children with ASD from the community-based sample is described below.

Figure 1.

Steps in the screening and evaluation process used by the FIRST WORDS^® Project to identify children with autism spectrum disorders. (© 2008, Florida State University. All rights reserved. Reprinted with permission.)

Screening and Evaluation Procedures

Broadband Screening with ITC

Families were given a packet with a brochure about the project, the ITC, a family information form, and a consent form to participate in this research study at initial contact. The brochure indicated that the project was screening communication development from as many families as possible with children 6 to 24 months of age and was interested in both families with young healthy infants and families who may have concerns about their child’s communication development. The project screened about 1/4^th the birth rate of the region annually and the general population sample screened was representative of this region in regard to race, ethnicity, parent’s education, and parent’s age. Because the ITC was given to families by their primary care provider or childcare providers and asked to complete it onsite, or families requested a packet at public places such as health fairs, no information is available on families who were given the ITC but did not complete it.

All families who completed an ITC were sent a brief clinical report that indicated whether their child’s developmental milestones based on parent report were or were not as expected for their age (i.e., negative or positive screen respectively). The following criteria established by Wetherby and Prizant (2002) were used for a positive screen on the ITC: 1) the bottom 10^th percentile (i.e., 1.25 SD below the mean) on the Social composite, Symbolic composite, or Total score; or 2) the bottom 10^th percentile on the Speech composite on two consecutive ITCs. The ITC was used as a surveillance tool until 24 months and was mailed 3 months later to families of children with a negative screen, and families who did not agree to or whose children were too young to participate in the communication evaluation. A total of 8,563 ITCs were completed by the families of this sample of 5,385 children.

Communication Evaluation with CSBS DP Behavior Sample

Families of all children with a positive screen on the ITC and/or who reported concern about their child’s development on the ITC and randomly selected children with a negative screen were contacted by phone to invite them to bring their child in for a face-to-face communication evaluation between 12 and 24 months to conduct the CSBS DP Behavior Sample. Children who participated in a Behavior Sample under 18 months were invited for a second evaluation between 18 and 24 months. The Behavior Sample was collected in a small clinical room using the standard sampling materials and procedures (Wetherby & Prizant, 2002). A child’s caregiver was present during the full evaluation and was instructed to respond naturally, but not to direct the child’s behavior, in order to encourage spontaneous communication and play. The evaluation session began with a warm-up of about 10 minutes and lasted 30–40 minutes. The Behavior Sample uses a standard set of systematic procedures designed to encourage spontaneous behavior that range in degree of structure provided. The child is first presented with a series of communicative temptations to entice spontaneous communication using a windup toy, balloon, bubbles, jar with food, bag with toys and books designed for young children. The child is then presented with a feeding toy set and stuffed animal to play symbolically and blocks to play constructively. The sample includes probes of gaze/point following and comprehension of person name, body part, and object name. The Behavior Sample was videotaped and scored using the standard procedures by one of 5 trained examiners who were blind to the child’s diagnostic classification.

The Behavior Sample has normed scores at four age intervals from 12–24 months based on a normative sample of 337 children (Wetherby & Prizant, 2002). Standard scores include a Social composite (sum of Emotion and Eye Gaze, Communication, and Gestures clusters), Speech composite (sum of Sounds and Words clusters), and Symbolic composite (sum of Understanding and Object Use clusters). The standard scores were scaled to a mean of 10 and SD of 3 for the composites and a mean of 100 and SD of 15 for the Total score. The criterion for a communication delay established by Wetherby and Prizant is performance in the bottom 10^th percentile (i.e., 1.25 SD below the mean) on two clusters, one composite, or the total score.

Psychometric Features of the ITC and Behavior Sample

Information about the reliability and validity of the CSBS DP has been reported in Wetherby et al. (2002), Wetherby et al. (2003), and Wetherby and Prizant (2002). Based on the normative sample, the measures of the CSBS DP were found to have a high degree of internal consistency (α coefficients ranging from .86 to .92) and good test-retest reliability for standard scores over a 4-month interval, with significant increases in raw scores. Construct and concurrent validity has been supported by the developmental progression of scores from 6 to 24 months of age, intercorrelations among cluster and composite scores, and correlations between the parent report measures and the Behavior Sample. Wetherby et al. (2003) compared the accuracy of the ITC to standardized testing on 232 children between 12 and 24 months of age, half with language delays and half with typical development. Sensitivity was 87.4% and specificity was 75.2% using the bottom 10^th percentile or 1.25 standard deviations below the mean as criterion for risk, which is comparable to or better than other instruments gathered with infants and toddlers using 2 standard deviations below the mean as risk criterion (e.g., Squires, Potter, & Bricker, 1999; Glascoe, 1999). The three composites of the ITC and Behavior Sample were found to be a significant predictor of receptive and expressive language outcomes at 2 and 3 years of age and the Behavior Sample explained a significant amount of unique variance in language outcomes beyond the ITC. Thus, the ITC and Behavior Sample are appropriate screening and evaluation tools for identifying children with developmental delays at 12 to 24 months of age.

Follow-up Developmental Evaluation

Families of children who participated in the communication evaluation using the Behavior Sample in the 2^nd year of life were contacted by phone when their child turned two and three years of age to invite them to bring their child back for a follow-up developmental evaluation. Developmental level was measured with the Mullen Scales of Early Learning (MSEL; Mullen, 1995), which includes separate scales for nonverbal and verbal skills. The MSEL Early Learning Composite was used for this analysis, which is a standard score scaled to a mean of 100 and SD of 15 based on the sum of the T scores for the cognitive scales (visual reception, fine motor, receptive language, and expressive language).

Surveillance Methods for Detecting Children with ASD

Three surveillance methods were used in an effort to identify all children with ASD who were screened by the project and are indicated in the screening and evaluation process shown in Figure 1. First, the Systematic Observation of Red Flags of ASD (SORF), which is a rating of 13 red flags of ASD (see Wetherby et al., 2004), was used as an autism-specific screen at both the communication evaluation in the second year and the developmental evaluation at 2 and 3 years of age. The child’s record in the FIRST WORDS database was tagged to indicate red flags of ASD based on the clinician rating of the SORF. Second, all families who completed an ITC were mailed a one-page follow-up questionnaire in 2007 if their child was 4 years of age or older to inquire if their child has received a diagnosis of ASD or other developmental disorder. Third, families who register with the Florida State University Center for Autism and Related Disabilities (CARD), a state-funded program that serves children and adults with ASD, are asked to sign a release of information to share information with the FIRST WORDS Project.

Diagnostic Evaluation to Confirm or Rule out ASD

Families of all children suspected of having ASD from any of the three surveillance methods were invited to participate in a follow-up diagnostic evaluation at 3 years of age or older. The diagnostic team made a best estimate diagnosis based on the following measures: a) the MSEL to determine nonverbal and verbal developmental level; b) the Vineland Adaptive Behavior Scales (VABS; Survey Interview Form; Sparrow, Balla, & Cicchetti, 1984) to provide an index of adaptive behavior; c) a developmental history; d) the Autism Diagnostic Observation Schedule (ADOS; Lord, Rutter, DiLavore, & Risi, 1999) to provide a standardized assessment of communication, social interaction, and play or imaginative use of materials for the diagnosis of ASD; and e) the Social Communication Questionnaire (SCQ; Lifetime Version; Rutter, Bailey, Berument, Lord, & Pickles, 2001) to provide parent report of symptoms of ASD.

The diagnostic evaluation included the ADOS but not the Autism Diagnostic Interview-Revised (ADI-R; LeCouteur, Lord, & Rutter, 2003) for two reasons. First, research suggests that the ADI is not as accurate for children under 4 years of age as for older children (Chawarska & Volkmar, 2005; Cox et al., 1999; Lord, 1995). Second, the administration time for the ADI limited its feasibility for this study because of the challenge of getting families to agree to participate in the diagnostic battery, which already was very lengthy. The SCQ was collected to gather information about possible signs of ASD from parent report. This screening questionnaire was developed by the authors of the ADI and covers the same domains as the ADI. The diagnostic team used the information from the SCQ to make the diagnosis, which was particularly important for children who displayed minimal repetitive behaviors and restricted interests during the ADOS. It also helped to form consensus with the family on diagnosis.

Participant Characteristics

The participant and demographic characteristics of the children screened in the general population sample with the ITC, the subset of children who participated in a communication evaluation, and the subset of children later diagnosed with ASD are presented in Table 1. There were significant differences between the general population sample and the subgroup who participated in a communication evaluation on all characteristics, except first born, because of the very large sample size. However, all of these differences were very small or small effect sizes with only one exception. The mothers of the subgroup with a communication evaluation had significantly more years of education than the general population sample with a medium effect size (d=.58). The subgroup with ASD was significantly different than the subgroup with a communication evaluation on four characteristics. They were significantly higher on age when the ITC was completed (t=3.36, p=.001; d=.43), positive screens (X²=49.5, p=.000,Φ=.17), parents with concerns (X²=8.1, p=.004,Φ=.10), and males (X²=23.7, p=.000, Φ=.25). Although these differences were significant, they were all small effect sizes.

Table 1.

Summary of Participant Demographic Characteristics

Characteristic	General Population Sample	Subgroup with Communication Evaluation	Subgroup with ASD
Children Screened	5,385	813	60
Infant-Toddler Checklists Completed	8,563	1,274	78
Age at First Infant-Toddler Checklist	13.5 (4.7)	14.2 (4.8)	16.4 (5.4)
Positive Screen on ITC	18.0%	37.8%	93.3%
Parents with Concern on ITC	12.2%	28.1%	46.8%
Percentage of Males	51.1%	58.3%	88.3%
Percentage of First Born	49.4%	46.2%	39.3%
Race and Ethnicity
Caucasian	59.3%	71.1%	67.2%
African American	30.2%	17.6%	18.0%
Hispanic	3.1%	5.4%	8.2%
Asian	2.3%	3.3%	3.3%
Other	5.1%	2.6%	3.3%
Parent’s Education in Years Completed
Mother (M, SD)	14.3 (2.4)	15.7 (2.4)	15.4 (2.2)
Father (M, SD)	14.3 (2.7)	15.3 (2.8)	15.6 (2.6)
Parent’s Age at Child’s Birth in Years
Mother (M, SD)	28.2 (6.2)	30.8 (6.0)	31.7 (5.1)
Father (M, SD)	30.9 (7.0)	33.5 (6.8)	34.1 (6.5)

Results

Validity of the ITC to Detect Children with Communication Delays

Of 5,385 children screened with the ITC, 978 children participated in one or more Behavior Samples totaling 1,274 Behavior Samples. The results were divided into six age intervals between 6 and 24 months based on age when the ITC was completed. Each ITC was compared to the Behavior Sample closest in age to examine agreement classification. For younger children there was a larger age gap between the ITC and Behavior Sample because the ITC was completed as early as 6 months and children did not participate in the Behavior Sample until at least 12 months of age. The classification of positive/negative screen on the ITC and typical/delayed communication on the Behavior Sample are presented in Table 2. A 2×2 classification table was used to estimate positive predictive value and negative predictive value at each age interval. Only one ITC was included for each child at each age interval. Thus, the samples were independent within each age interval presented in Table 2 but not across age intervals. About 23% of the sample had more than one ITC and Behavior Sample and was included in more than one age interval. By including more than one ITC if available, the results reflect a developmental surveillance system as it would be used in a community-based program. The results support the validity of the ITC for detecting communication delays with estimates of positive and negative predictive value at or higher than 70% for all age groups except 6 to 8 months based on children who participated in a communication evaluation. Estimates of sensitivity and specificity were not calculated because they may be inflated due to the proportion of children with negative screens being lower than would be found in a general population sample.

Table 2.

Estimates of Positive and Negative Predictive Value for the Infant-Toddler Checklist Compared to Communication on the Behavior Sample

	6 to 8 months				9 to 11 months				12 to 14 months
		Communication				Communication				Communication
	ITC Screen	Delayed	Typical		ITC Screen	Delayed	Typical		ITC Screen	Delayed	Typical
	Positive	6	8	14	Positive	54	22	76	Positive	98	36	134
	Negative	11	75	86	Negative	12	171	183	Negative	2	194	196
		17	83	100		66	193	259		100	230	330
Age at ITC	7.6 (1.0)				10.7 (1.0)				13.1 (1.0)
Age at Behavior Sample	16.5 (3.3)				16.9 (3.5)				16.6 (3.1)
Positive Predictive Value	42.9%				71.1%				73.1%
Negative Predictive Value	87.2%				93.4%				99.0%
	15 to 17 months				18 to 20 months				21 to 24 months
		Communication				Communication				Communication
	ITC Screen	Delayed	Typical		ITC Screen	Delayed	Typical		ITC Screen	Delayed	Typical
	Positive	79	33	112	Positive	65	19	84	Positive	49	13	62
	Negative	7	119	126	Negative	14	121	135	Negative	8	58	66
		86	152	238		79	140	219		57	71	128
Age at ITC	16.2 (0.9)				19.3 (.09)				22.8 (1.2)
Age at Behavior Sample	18.8 (2.4)				20.8 (1.7)				23.4 (1.2)
Positive Predictive Value	70.5%				77.4%				79.0%
Negative Predictive Value	94.4%				89.6%				87.9%

The percentage of parents reporting concern about their child’s development on the ITC for children with a positive versus negative screen on the ITC who have delayed versus typical communication based on the Behavior Sample is presented in Table 3. The percentage of parents who reported concern is higher for children over 18 months of age both with delayed and typical communication. Across age intervals, a substantial percentage of parents of children with delayed communication do not report concern and a substantial percentage of parents of children with typical communication do report concern. It is important to note that parents with concern were invited for a Behavior Sample, whether the ITC screen was positive or not, and therefore, the values in Table 3 over-represent parents with concern from the general population sample.

Table 3.

Agreement of Parent Concern and Infant-Toddler Checklist Screen Compared to Communication on the Behavior Sample

	Age in Months when ITC was Completed
	6–8		9–11		12–14		15–17		18–20		21–24
Delayed Communicationa	n = 17		n = 66		n = 100		n = 86		n = 79		n = 57
	Parent Concern
	Yes	No	Yes	No	Yes	No	Yes	No	Yes	No	Yes	No
ITC Positive Screen	18%	18%	29%	53%	24%	74%	34%	58%	42%	41%	53%	33%
ITC Negative Screen	35%	29%	2%	17%	0%	2%	0%	8%	8%	10%	7%	7%
Typical Communicationa	n = 83		n = 193		n = 230		n = 152		n = 140		n = 71
	Parent Concern
	Yes	No	Yes	No	Yes	No	Yes	No	Yes	No	Yes	No
ITC Positive Screen	2%	7%	3%	9%	4%	11%	9%	13%	5%	9%	14%	4%
ITC Negative Screen	18%	72%	13%	76%	14%	70%	9%	70%	27%	59%	46%	35%
Kappa Statistic
Parent Concern & ITC Screen	.08		.19**		.10*		.28***		.15*		.08
Parent Concern and Behavior Sample	.27**		.16**		.06		.18**		.17*		.01
ITC Screen and Behavior Sample	.28**		.67***		.75***		.66***		.68***		.67***
Logistic Regression	Delay	Typical	Delay	Typical	Delay	Typical	Delay	Typical	Delay	Typical	Delay	Typical
Percentage of Correct Prediction	17.6	97.6	81.8	88.6	98.0	84.3	91.9	78.3	82.3	86.4	86.0	81.7
β (S.E.) for Parent Concern	−1.45 (.58)**		−0.26 (.46)		−0.22 (.43)		−0.22 (.41)		−0.53 (.39)		0.53 (.51)
β (S.E.) for or ITC Screen	−1.60 (.67)**		−3.52 (.40)***		−5.60 (.74)***		−3.77 (.46)***		−3.35 (.39)***		−3.40 (.51)***

^aBased on the CSBS DP Behavior Sample standard scores in the second year

^*p <.05

^**p <.01

^***p <.001

The kappa statistic for agreement between parent concern and ITC screen, parent concern and communication on the Behavior Sample, and ITC screen and communication on the Behavior Sample are presented in Table 3 for each age interval. These kappa values between screening outcome and parent concern were significant due to the large sample size but reflect a small to negligible relationship or near chance agreement (Szklo & Nieto, 2007). Kappas for agreement between parent concern and typical/delayed communication on the Behavior Sample indicate poor agreement. In contrast, kappas for agreement between positive/negative screen on the ITC and typical/delayed communication on the Behavior Sample indicate good agreement. Overall, parent report of concern is important to consider but is not as accurate a screening measure of communication delay as parent report about developmental milestones on the ITC.

Logistic regression was conducted to assess whether parent concern and ITC screen results significantly predicted a child’s communication status on the Behavior Sample. The model with both variables together significantly predicted communication outcomes on the Behavior Sample at all age intervals. The percentage of children who had communication delays versus typical communication and were predicted correctly with this model containing both variables is shown in Table 3. The β and SE for the unique contribution of parent concern and ITC screen are also presented in Table 3. The results show that the ITC screen is a significant predictor, but not parent concern, when both are considered together, with only one exception. Both parent concern and ITC were significant predictors at 6–8 months.

Detection of Children with ASD from a General Population Sample

At the time of this study, 60 children who were 4 years of age or older completed the diagnostic evaluation and received a best estimate diagnosis of ASD from our general population sample of 5,385 children screened between 6 and 24 months. Some parents completed more than one ITC, totaling 78 ITCs completed between 6 and 24 months for the 60 children with ASD. The ITC positive and negative screen results are presented in Table 4 by subject number for each of six age intervals when the ITC was completed. Of the 60 children with ASD, 56 received a positive broadband screen on the first and/or subsequent ITC, which is a sensitivity or true positive ratio of 93.3%. Subjects 4, 5, 45, and 60 did not receive a positive screen. All 4 of these false negatives had participated in a Behavior Sample. Subjects 4 and 5 did not display a communication delay on the Behavior Sample but red flags of ASD were reported by the clinician at their developmental evaluation at 3 years of age. Subject 45 and 60 were found through CARD. Both had displayed communication delays at the time of the Behavior Sample.

Table 4.

Infant-Toddler Checklist Positive (+) or Negative (−) Screen Results for 60 Children with ASD from 6 to 24 Months

	Age in Months when ITC was Completed
Child with ASD	6–8 (n = 5)	9–11 (n = 13)	12–14 (n = 11)	15–17 (n = 17)	18–20 (n = 13)	21–24 (n = 19)
1	+			+
2	−	+		+
3	−	−	+	+
4	−
5	−	−
6–10		+ (5)
11–12		+ (2)				+ (2)
13		+	+
14		+	+	+
15		−	+
16–20			+ (5)
21			+	+
22			−	+
23–32				+ (10)
33				+	+
34–42					+ (9)
43–44					+ (2)	+ (2)
45					−
46–59						+ (14)
60						−

Note: Number of children if more than one with positive screen in parentheses.

The follow-up questionnaires had been returned from 25.9% of the sample screened with the ITC whose children were at least 4 years of age and no additional children with possible ASD were reported. The demographic characteristics of the children whose family returned the follow-up questionnaires were similar to the general population sample with 63.0 % Caucasian, 31.3% African American, 2.4% Hispanic, 2.0% Asian, and 1.3% other; 46.8% were male, and 47.85 were first born. The mother’s education level was 14.4, father’s education level was 14.3, mother’s age at the child’s birth was 28.4 years and the father’s age was 31.2%

The percentage of positive screens and parents with concern reported on the ITC and the standard scores on the ITC, Behavior Sample, and MSEL for the 60 children with ASD are presented in Table 5 for each of the six age intervals. Children with more than one ITC were included more than once in the corresponding age interval but only once per age interval. The percentage of true positive screens or sensitivity on the ITC is low at 6–8 months but increased with age and is above 90% at 12–14 months through 21–14 months for children with ASD. The proportion of parents with concern is less than half under 15 months and increases to more than two-thirds at 18–20 months and approaches three-fourths at 21–24 months.

Table 5.

Measures for 60 Children with ASD Screened with the Infant-Toddler Checklist from 6 to 24 Months

	Age in Months when ITC was Completed
Measure	6–8	9–11	12–14	15–17	18–20	21–24
Sample Size	5	13	11	17	13	19
ITC
Age	7.2 (1.2)	10.3 (1.1)	13.2 (0.9)	16.4 (0.9)	19.2 (0.8)	23.2 (1.4)
Positive Screen	20%	77%	91%	100%	92%	95%
Parents with Concern	40%	31%	45%	53%	69%	74%
Social Compositea	9.2 (4.4)	6.5 (2.8)	5.6 (2.6)	5.4 (2.1)	4.8 (2.3)	4.2 (2.3)
Bottom 10th %ile	20%	62%	82%	82%	85%	79%
Speech Compositea	9.4 (1.3)	7.5 (2.9)	6.1 (1.8)	5.4 (2.1)	5.5 (1.6)	5.5 (2.0)
Bottom 10th %ile	0%	39%	46%	59%	77%	63%
Symbolic Compositea	9.0 (2.6)	7.3 (3.1)	7.6 (2.8)	7.0 (2.7)	6.5 (2.5)	5.4 (2.1)
Bottom 10th %ile	20%	39%	46%	59%	46%	58%
Totalb	95.8 (17.7)	82.0 (13.8)	78.4 (8.5)	75.4 (8.0)	74.2 (7.1)	71.1 (7.5)
Bottom 10th %ile	20%	46%	73%	77%	92%	90%
Behavior Sample
Age	17.2 (2.1)	18.0 (4.4)	15.7 (2.9)	18.3 (1.7)	20.3 (1.0)	23.8 (1.3)
Communication Delay	60%	77%	82%	100%	92%	100%
Social Compositea	7.4 (2.7)	5.4 (2.5)	5.8 (2.4)	4.9 (2.0)	5.1 (1.8)	4.0 (1.2)
Bottom 10th %ile	40%	77%	82%	88%	85%	100%
Speech Compositea	9.0 (3.4)	6.8 (3.0)	7.2 (1.6)	6.4 (2.8)	5.6 (1.8)	4.6 (1.8)
Bottom 10th %ile	20%	39%	18%	59%	54%	74%
Symbolic Compositea	7.2 (3.7)	6.1 (3.6)	7.6 (2.8)	6.6 (3.4)	6.5 (3.1)	4.4 (2.3)
Bottom 10th %ile	60%	62%	46%	65%	62%	79%
Totalb	86.4 (17.2)	76.3 (14.4)	77.6 (9.6)	75.5 (13.2)	74.2 (10.9)	68.5 (5.5)
Bottom 10th %ile	40%	77%	73%	71%	85%	95%
Mullen Scales
Age	34.3 (5.0)	43.0 (10.1)	36.7 (1.8)	38.8 (7.0)	35.4 (9.6)	40.9 (9.9)
Early Learning Compositeb	106.8 (21.6)	79.6 (31.0)	87.4 (30.1)	83.5 (32.2)	74.9 (23.8)	63.1 (15.7)
Bottom 10th %ile	0%	54%	46%	47%	62%	84%

^aStandard Scores based on a M of 10 and SD of 3

^bStandard Scores based on a M of 100 and SD of 15

This sample of 60 children with ASD is very heterogeneous in developmental level on the MSEL and had a mean Early Learning Composite of 73.3 (SD=25.0) with 63% in the bottom 10^th percentile. The standard scores on the ITC, Behavior Sample, and MSEL are lower for the ASD children at older age intervals with a larger percentage falling in the bottom 10^th percentile likely reflecting sampling bias. That is, more children with ASD who had more significant delays participated at older ages, likely due to greater parent awareness of delay and concern, and therefore, higher motivation to participate in the screening and evaluation process. Some parents did not participate in the Behavior Sample until their child had a positive screen on at least 2 ITCs. In contrast, all 5 children with ASD in the 6–8 month age interval were high functioning, with a mean Early Learning Composite of 106.8 and all above the 10^th percentile. As shown in Table 4, child 1 had a positive screen at 6–8 months, child 2 and 3 had a subsequent positive screen on the ITC, child 2 at 9–11 months and child 3 at 12–14 months. Child 4 and 5 did not have a positive screen and did not display a communication delay on the Behavior Sample in the second year. For both the ITC and Behavior Sample, more children with ASD performed in the bottom 10^th percentile on the Social composite than on the Speech or Symbolic composite, making this composite an important early indicator of the need to conduct an ASD-specific screen.

Discussion

Accuracy of the ITC as a Broadband Screener of Communication Delay

The results of this study support the validity of the ITC as a broadband screener of communication delays including ASD in children 9–24 months of age but not 6–8 months. Positive and negative predictive values were all above 70% for children 9–24 months of age based on those evaluated with a Behavior Sample. These ratios were based on a 1 to 6 month time gap between the ITC and Behavior Sample, because communication evaluations were not conducted until 12 months of age or older, but are comparable to screening tools validated on concurrent measures (e.g., Glascoe, 1999; Squires et al., 1999). In spite of a good negative predictive value, positive predictive value was very poor with a corresponding high false negative rate for children in the 6–8 month age interval. This is likely due to the small number of typical developmental milestones measured on the ITC that are achieved at this age. This may also reflect the smaller sample size at this age interval and the longer time delay between the ITC and Behavior Sample.

The ITC consists of 24 questions about typical social communication milestones and a question about parent concern. The findings of this study suggest that parents of children 9–24 months are fairly accurate at reporting current developmental milestones but are not very accurate in reporting concern about their child’s development relative to what should be expected at their child’s age. This finding is important to consider because parent concern can influence whether parents seek out or agree to participate in screening and evaluation. The majority of parents of children under 20 months with delayed communication do not report concern. In contrast, a substantial percentage of parents of children 21–24 months report concern whether their child is typical or delayed.

Accuracy of the ITC to Detect Children with ASD

The FIRST WORDS Project screening process with the CSBS DP and surveillance methods for detecting children with ASD identified 60 children with ASD from 5,385 children screened, which is a rate of 11.1 per 1,000. This rate is high compared to recent prevalence estimates based on record reviews of 8 year-olds (CDC, 2007). We found that 18% of our general population sample had a positive ITC screen, which is higher than expected since cutoffs are based on the 10^th percentile. This may reflect more children being screened whose doctor or parent had concerns about the child. This was more pronounced at 21–24 months, which is when parents were more likely to notice and report concern about their child’s communication. The mean composites on the CSBS DP and MSEL were lower in this age interval for the children with ASD, reflecting a sampling bias at this age. This pattern of heightened parent concern at 21–24 months is likely to be encountered in community-based screening programs and may encourage parent participation in screening and evaluation at this age.

The ITC identified 56 of the 60 children with ASD in this general population sample or 93.3%, which is consistent with Wetherby et al. (2004) but on a larger sample screened that was followed longer with more careful surveillance methods. However, it is important to point out that a positive screen on the ITC does not distinguish children with ASD from children with other communication delays. Children with ASD who had a positive screen on the ITC were most likely to perform in the bottom 10^th percentile on the Social composite. Therefore, an ASD-specific screen should be considered particularly for children showing this pattern on the ITC. This sample of 60 children with ASD had a higher mean and more variance on the MSEL Early Learning Composite (M=73.3; SD=25.0) than the children with ASD detected by the M-CHAT reported by Kleinman et al. (in press) for the general population sample (M=56.31; SD=10.0). This suggests that the ITC and our surveillance methods are detecting a more heterogeneous sample of children with ASD including relatively more children who are higher functioning.

Research on parent concerns of children with developmental disabilities has suggested that concerns raised by the majority of families are warranted (Glascoe, 1999). However, there is limited research on parent concerns of children in the first or second year of life except for retrospective research asking parents to remember back when their child was an infant or toddler. Less than half of the parents of children with ASD under 15 months of age in this study had concerns about their child’s communication development, at least when asked in the context of a checklist about communication milestones. Parent concern increased with the child’s advancing age to three-fourths by 21–24 months. The percentage of parents with concern was higher by 12–14 months for children with ASD than the inclusive group with communication delays and this was consistent across the second year of life. The percentages of parents of children with ASD who reported concern on the ITC in this prospective study were very consistent with those reported from retrospective studies which have found that 30% of parents of children with ASD recall developmental problems before the first birthday, 50% before 18 months, and 80% by 2 years (Chawarska et al., 2007). This study illustrates that many parents of young children are concerned about their child’s communication development. Further research comparing the specific concerns reported may help distinguish children with ASD.

Limitations of this Study

This study reflects an ambitious effort to implement a community-based screening and evaluation process beginning with the ITC. A limitation of this study is the sampling bias of children whose families completed the ITC. Based on the 18% positive screen rate, more children with communication delays are included in this sample than would be expected in a general population sample. The sampling bias is most evident at the outer edges of the age intervals, which have the smallest number of children (100 and 128 children respectively). More parents were concerned at 21 to 24 months than any other age interval and this age interval includes a larger proportion of children with communication delays. The 6–8 month age interval had the smallest proportion of children with communication delay.

Another limitation of this study is that it was not possible to gather information on families who did not agree to fill out the ITC. The sample screened with the ITC reflects the demographics of the community suggesting that it is representative. Comparing the demographics of the general population sample with the families who participated in the communication evaluation reveals a reluctance of some African American families to participate in the communication evaluation although they comprised a substantial proportion of the screening sample. These findings may explain the later age of diagnosis of African American children (Mandell et al. 2005) and are important to consider in community-based early intervention programs. In contrast, a slightly higher proportion of Hispanic families participated in the communication evaluation than the general population sample and was included in the ASD group. The demographic characteristics of the ASD children were similar to the subgroup who participated in the communication evaluation except the ASD subgroup had a higher proportion of positive screens, parents with concern, and males.

The study was also limited by the low return rate of the follow-up questionnaire. It is possible that more children with ASD are in the general population sample and were not detected. However, the other surveillance methods did catch the missed children among those whose family returned the questionnaire, and therefore, are important to consider in a community-based screening program. It is also noteworthy that the subgroup returning the follow-up questionnaire was very representative of the general population sample screened based on demographic characteristics. This in an ongoing longitudinal study and a second mailing of the questionnaire will be conducted in the future and may increase the response rate.

Improving Early Detection of ASD

Healthcare providers are in a pivotal role to detect developmental problems including ASD earlier by listening to families concerns, conducting developmental surveillance regularly as recommended by the AAP (Johnson & Scott, 2007), and making referrals for a developmental evaluation so that families access intervention earlier. These findings add to the growing body of research documenting the accuracy of parent report of developmental milestones to screen young children. Using a parent report tool, such as the ITC, minimizes the time required of healthcare providers, maximizes the role of the family, and provides reasonably accurate information about whether to refer a child for a communication evaluation or consider an ASD-specific screen. By 24 months of age, an ASD-specific parent report screener such as the M-CHAT with a follow-up phone interview looks very promising. However, at this point in time research does not support the use of an ASD-specific screener based only on parent-report in a pediatric practice for children under 24 months of age. Interactive ASD-specific screeners may hold more promise (Wetherby et al., 2004; Stone, McMahon, Yoder, & Walden, 2007). Ongoing prospective studies of young children with ASD have documented an unfolding of social communication deficits over the second year of life (Landa et al., 2007; Wetherby, et al., 2007; Zwaigenbaum et al., 2007) and will be critical to inform future screening efforts. For now, extreme caution is needed to not rule out ASD prematurely in children with social communication delays less than 24 months.