Completeness of Case Ascertainment for Surveillance of Autism Spectrum Disorders Using the Autism Developmental Disabilities Monitoring Network Methodology

Abstract

The Autism and Developmental Disabilities Monitoring Network (ADDM), sponsored by the Centers for Disease Control and Prevention, is the largest-scale project ever undertaken to identify the prevalence of Autism Spectrum Disorders (ASD) in the United States. The objective of the present study was to examine the accuracy of the ADDM methodology in terms of completeness of case ascertainment; that is, to assess the success of the ADDM Network in identifying and accurately classifying all existing cases of ASD among 8-year-old children in the target study areas. To accomplish this objective, the ADDM methodology was applied to a selected region of South Carolina for 8-year olds in 2000 (birth year 1992) and again seven years later for the same region and birth year. For this region and birth year, completeness of case ascertainment was high, with prevalence estimates of 7.6 per 1000 at both ages 8- and 15-years. For children common to both surveillance years, concordance in case status was also high (82%). Given that prevalence did not change within this region and birth year, continued research is needed to better understand the changes in prevalence estimates being found by the ADDM network across surveillance groups.

INTRODUCTION

Autism Spectrum Disorders (ASDs) encompass a range of behaviorally defined conditions characterized by atypical development in socialization, communication, and behavior. The Autism and Developmental Disabilities Monitoring (ADDM) Network is a Centers for Disease Control and Prevention (CDC) sponsored project developed with the authorization of the Children’s Health Act of 2000. Spanning much of the past decade, ADDM is the largest scale effort ever undertaken to determine and monitor the prevalence of ASD. The ADDM Network’s methodology aims to identify all 8-year-old children in the target study area who have an ASD, including those who may not have previously been diagnosed with an ASD. Since the network’s inception in 2000, participants using the same records-based methodology have included 15 sites in 16 states—Alabama, Arizona, Arkansas, Colorado, Florida, Georgia, Illinois, Maryland, Missouri, New Jersey, North Carolina, Pennsylvania, South Carolina, Utah, West Virginia, and Wisconsin. Overall prevalence estimates across successive surveillance years for 8-year-old children are 6.7 per 1000 in study year 2000 [1], 6.6 per 1000 in study year 2002 [2], 8.0 per 1000 in study year 2004 [3], and 9.0 per 1000 in study year 2006 [3]. These estimates indicate that approximately 1% of 8-year-old children in 2006 had an ASD [3].

Accurate reporting of ASD prevalence is important for researchers, health care providers, policy makers, and families. Prevalence estimates and changes in prevalence identified by the ADDM network are widely cited by the scientific and stakeholder communities to establish the critical needs for service support for people with ASDs.

Previously, accuracy in terms of sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the assignment of case status (case versus non-ASD case) using ADDM record-based methodology was studied in a group of 177 children by comparison to clinical examination [4]. Estimated specificity (0.96), PPV (0.79), and NPV (0.91) were high (comparable to clinical screens in similar samples of studies), with sensitivity lower (0.60) [4].

The objective of the present study is to determine accuracy of the prevalence estimates obtained by ADDM by examining completeness of case ascertainment, in other words, by examining the extent to which children with ASD may have been missed or inaccurately classified by the case identification method. Because the ADDM design is a period prevalence design, a sub-region of the 2000 surveillance area (birth year 1992) was selected for re-application of the methodology at a later time point, specifically 2007, when children born in 1992 were 15 years old. The sub-region selected was a three-county region within South Carolina containing 8069 8 year olds in 2000 and 8786 15 year olds in 2007. South Carolina was a suitable location for this study because of its continuous membership in the ADDM Network since 2000, its stable study team, and its full cooperation from all surveillance sources for all surveillance years.

METHODS

The ADDM Network methodology and prevalence results have been previously described [1] [2] [3] [5] [6] [7]. Briefly, the ADDM Network is an active, population-based surveillance program monitoring the prevalence of developmental disabilities among 8-year-old children through retrospective review of records from both school and clinical sources (e.g., clinics, disability centers). Children are classified as having an ASD if behaviors documented in evaluation records are consistent with the Diagnostic and Statistical Manual of Mental Disorders, 4 th Edition, Text Revision (DSM-IV-TR) [8] criteria for autistic disorder, pervasive developmental disorder-not otherwise specified (PDD-NOS) (including atypical autism), or Asperger disorder [3].

Sites participating in the ADDM Network were selected through a competitive federal award process and consisted of the CDC and academic researchers working on behalf of their state health departments [1] [6] . For surveillance year 2000, the number of children aged 8 years residing in each site ranged from 21,532 to 45,322 [1]. To begin case ascertainment, data sources for health and/or education providers are identified. Providers at health sources are requested to provide a list of all children whose medical records are associated with a diagnostic or billing International Criteria for Diagnosis, 9 th Revision (ICD-9) [9] code for child neurodevelopmental disorder (e.g., 299.0 for autistic disorder or 315.3 for developmental speech or language disorder). In addition, South Carolina (among several other sites) includes education sources and the special education evaluation records of children receiving school services (for example, for the 2006 surveillance year, evaluation records were requested for 2005–2006 and/or 2006–2007). Each child is assigned a unique identifier to prevent duplication and to link information across multiple data sources. Records are selected for abstraction if the child meets the residency requirement and the record contains a diagnosis of an ASD, a special education eligibility classification of autism, and/or a description of social difficulties associated with an ASD diagnosis (e.g., poor eye contact, failure to respond to name). All abstracted evaluations are then reviewed and scored by ASD clinician reviewers with specialized training and experience in autism assessment and diagnosis. Clinician reviewers use a coding guide based on the DSM-IV-TR criteria to determine whether each abstracted child meets the ASD case definition [3]. Interrater reliability was established among clinician reviewers to standards of 80%-85% agreement for individually scored items and 90% for overall case status, with ongoing checks conducted on a random (10%) sample of records [1]. Of note, the case ascertainment process does not limit surveillance to previously diagnosed cases [3]. South Carolina has had 100% cooperation from all surveillance sources for all surveillance years.

In the ADDM Network’s first study year, the prevalence of ASDs among 8 year olds was calculated for children born in 1992 and living in the surveillance area in 2000. These children were 15 years old in 2007. To determine the completeness of case ascertainment for study year 2000, the South Carolina site applied the same ADDM Network methodology to 15 year olds living in a three-county sub-region of its surveillance area in 2007. This sub-region was selected because it is demographically similar to the total surveillance area ( Figure 1 ). When comparing prevalence estimates between the two age groups, all calculations for the 8-year-old group were re-calculated for the same sub-region. In addition to comparing prevalence between the two age groups, records for those children abstracted in both years were evaluated for consistency in case status assignment. Among children identified as cases, age groups were compared in regard to sex, race/ethnicity, school service categories, co-occurring intellectual disability, age at first diagnosis, and median number of evaluations.

Figure 1.

South Carolina Autism Developmental Disabilities Monitoring Study Area (SC ADDM). Light gray area represents the entire SC ADDM surveillance area. Dark gray area represents the three-county sub-region used in this analysis.

Reprinted from Nicholas JS, Carpenter LA, King LB, Jenner W, Charles J. Autism spectrum disorders in preschool aged children: prevalence and comparison to a school aged population. Annals of Epidemiology 19:808–814, 2009, with permission from Elsevier.

Statistical analysis

ADDM Network prevalence rates are calculated using as the denominator the number of children aged 8 years residing in the study area. The numerator is the number of cases in the study area as identified by the methodology described previously. Poisson approximation to the binomial distribution is used to calculate 95% confidence intervals (CI) for prevalence rates [3]. Prevalence rates reported here were calculated using this same methodology. The denominator for study year 2000 was based on the number of 8-year-old children residing in the region according to US Census Bureau estimates. The denominator for study year 2007 was based on the number of 15-year-old children residing in the region according to CDC’s National Center for Health Statistics vintage 2007 postcensal estimates [10]. Descriptive statistics included means, standard deviations, frequencies, and percentages for variables of interest. All p values are two-sided.

RESULTS

The prevalence of ASDs among 8-year-old children residing in the three-county sub-region in 2000 (birth year 1992) was 7.6 (61 identified cases from a total population of 8069 8 year olds), 95% CI: 5.7–9.5. The prevalence of ASDs among 15-year-old children (birth year also 1992) residing in the same sub-region in 2007 was 7.6 (67 identified cases from a total population of 8786 15 year olds), 95% CI: 5.8–9.4. These results indicate no change in overall prevalence of ASDs for children born in 1992 and residing in the sub-region in 2000 and/or 2007.

Special education service categories for identified cases in each year are given in Table 1. The percent of children served decreased from 87% at age 8 years to 75% at age 15 years; however, of those served, the percent served under the Autism category increased from 21% at age 8 years to 58% at age 15 years (highly significant difference). Also in the older group, more were served under the category Other Health Impaired, and fewer under the category Intellectual Disability. (The category Other Health Impaired is used for a wide range of diagnoses that adversely affect a child’s educational performance).

Table 1.

Birth year 1992, special education eligibility for all cases residing in the region in surveillance year 2000 (age 8 years) and all cases residing in the region in surveillance year 2007 (age 15 years)

	Age 8 in 2000 N=61	Age 15 in 2007 N=67	p-value
% of ASD cases in special education	87%	75%	0.08
Autism	21%	58%	0.00†
Intellectual Disability	30%	10%	0.02†
Emotional Disorder	8%	4%	0.50
Learning Disorder	13%	10%	0.90
Speech and Language Disorder	9%	2%	0.10
Orthopedic Impairment	2%
Hearing Impairment	·	·
Vision Impairment	2%	·	0.30
Cross-categorical	2%	·	0.30
Other Health Impairment	·	16%	0.00†
Traumatic Brain Injury	·	·
Pre-school child with Disability	·	·
Other/Not Stated*	11%	·	0.01†

^†Statistically significant

^*In 2000, some children’s IEPs did not clearly state a special education eligibility. Formatting of IEPs in the study area became standardized between 2000 and 2007 so that all children had special education eligibilities clearly stated in their records in 2007.

Table 2 gives demographics and additional information for identified cases in each year. Race and the median number of evaluations did not differ significantly between children identified as cases at age 8 years in 2000 and cases at age 15 years in 2007. In the older group, fewer had an intellectual disability, a greater percentage was male, and the median age at first diagnosis was higher (the higher median age at first diagnosis among the older group is in part because of the inclusion of older children). (In this study, intellectual disability is defined as a full scale, or global, intelligence score of less than 70 on the most recent intelligence test. Because scores are derived from a range of tests at a range of ages, these data do not permit us to more closely examine individual strengths and weaknesses, such as spatial versus language skills, which can be disparate in children with ASD.) Because the ADDM methodology is a period prevalence design, as opposed to a prospective cohort design, children included in the two surveillance years were not necessarily the same. Therefore, several additional analyses were performed to explore variation in case status assignment at the child level.

Table 2.

Birth year 1992, demographics, intellectual disability, and evaluation information for all cases residing in the region in surveillance year 2000 (age 8 years) and all cases residing in the region in surveillance year 2007 (age 15 years)

	Age 8 in 2000 N=61	Age 15 in 2007 N=67	p-value
Sex			0.05†
Male	41(67%)	55(82%)
Female	20(33%)	12(18%)
Race*			0.26
White	41(67%)	41
Black	20(33%)	15
Other	-	2
Intellectual isability**			0.02†
IQ<70	33(63%)	27(42%)
IQ>70	19(37%)	37(58%)
Median number of evaluations	6	6	0.33
Median age at 1st diagnosis	44 months	104 months	0.00†

^†Statistically significant

^*Percentages do not add to 100 because of unknown race.

^**Intellectual disability reported only for children with available IQ testing scores (90% of 8 year olds, 96% of 15 year olds).

Children whose records were abstracted in both years

Although results indicate no change in overall prevalence of ASDs across time in the sub-region, individual variation in case-status assignment was examined. Among those children whose records were abstracted at either time point, a subgroup of 105 was identified as being abstracted at both time points within the sub-region. The change in case status between age 8 years and age 15 years is shown in Table 3 . Concordance in case status was 82%, as indicated by the 32 children identified as cases in both years and the 54 children identified as non-cases in both years. Of the eight children who moved from non-case to case status, seven accumulated more evaluations. Similarly, fewer records were identified for seven of the 11 children who moved from case to non-case. Of the remaining five children for whom case status changed, three narrowly missed or achieved case status based on one diagnostic criterion, and two were cases in which the child met criteria to be coded as a case, but was removed as a case in response to evidence that the child was not a case (following a group consensus meeting, which is part of the ADDM protocol).

Table 3.

Birth year 1992, change in case status between age 8 years and age 15 years for 105 children residing in the region during both surveillance years (8 in 2000, 15 in 2007). concordance 82%

	Case Yes at Age 15	Case No at Age 15
Case Yes at Age 8	32	11
Case No at Age 8	8	54

Children identified as cases in 1 year but not abstracted in the other

Some children were identified as cases in 1 year but not abstracted in the other. One reason for this could be in- or out-migration from the selected region. Because information on 15 year old children was only collected at one time point, it is not possible to track this migration for all children. Another difficulty in examining children not abstracted in a given year is the absence of recorded information for that year. Therefore, to examine these children, we used information from the year in which these children were identified as cases to look for predictors of abstraction (or not) in the other year.

In 2000, 61 children in the sub-region were identified as cases, with 43 ( Table 3) re-abstracted in 2007. Using study year 2000 data, those children identified as cases in 2000 and re-abstracted in 2007 were compared with those identified as cases and not re-abstracted. No significant differences were found in terms of race, intellectual disability, median number of evaluations, or age at first diagnosis.

In 2007, 67 children in the sub-region were identified as cases, with 40 previously abstracted in 2000. A similar examination of study year 2007 data revealed no significant differences between cases identified in 2007 who were or were not previously abstracted in 2000.

DISCUSSION

The results of this study provide evidence supporting the consistency of the ADDM Network’s prevalence estimates among children born in the year 1992 when the same methods are applied to the same geographical area 7 years apart. ASD prevalence among children born in 1992 was virtually identical for those residing in the selected region in 2000 (age 8 years) and those residing in the same region in 2007 (age 15 years) (7.6 per 1000 for both study years), suggesting a high level of completeness of case ascertainment using the CDC’s ADDM Network case identification methodology.

Concordance in assignment of case status was 82% for children abstracted in both years, suggesting good reliability of case assignment methodology. When changes in case status occurred among those abstracted in both years, they were generally associated with increases or decreases in evaluations available for review, with more evaluations at a specific time point being associated with a greater likelihood of an individual child being an ASD case. In addition, we found no significant predictors of re-abstraction among children abstracted in 1 year but not the other.

These results indicate that the ADDM methodology does produce stable prevalence estimates for children born in 1992 and residing in the region in 2000 (age 8 years) and children born in 1992 and residing in the region in 2007 (age 15 years). Given that ADDM methodology aims to identify life-time prevalence of ASD (rather than children currently meeting DSM-IV criteria) and that symptoms of ASD are generally apparent in the first years of life, we would expect to have stable prevalence estimates over time between ages 8 and 15.

In terms of Special Education services through public schools for these groups, children were more likely to have a special education eligibility of autism in 2007 at age 15 (58%) than in 2000 at age 8 (21%). This may be due in part to dramatic changes in community awareness of autism during the time period of interest (2000–2007). There was also an increase in the use of the category Other Health Impaired, which is used for a very wide range of diagnoses from Asperger’s Disorder to attention deficit hyperactivity disorder (ADHD) to Sickle Cell Disease. All other categories of disability decreased over the same period (including a significant drop from 30% to 10% in children whose primary classification was Intellectual Disability). These results suggest that diagnostic substitution may play a role in the changes in special education eligibility. Of note, significantly fewer children identified as ASD cases at age 15 years compared with age 8 years were also identified as intellectually disabled (IQ < 70). Other changes, although not statistically significant, contributing to the shift in eligibility classification included decreases in the proportions of children with emotional disorder (8% to 4%), learning disorder (13% to 10%), and speech and language disorder (9% to 2%). The proportion identified as Other Health Impaired increased significantly from 0% to16%. A definitive reason for this change is not known. We speculate that the cause is likely to be multi-factorial and may include factors such as increased awareness of autistic symptoms in children with normal cognitive functioning during the same time period, and increased likelihood of early referral for children with cognitive impairments. More of the 15-year-old cases were male, which when taken with the reduction of Intellectual Disability in that age group, suggests an increased identification in the older group of higher functioning males. The drop in those served in any category (87% at age 8, 75% at age 15) may indicate that some children did not require special education services at the later age. It should be noted that South Carolina uses a fairly narrow interpretation of the federal definition of Autism, including in that category primarily children with Autistic Disorder, and not those with the broader spectrum (PDD-NOS and Asperger’s). Inclusion of children in the broader spectrum would likely increase the percentage of children served with Autism-specific services.

The ADDM project is the largest-scale project ever undertaken to identify the prevalence of ASD in the United States. Results of the current analysis suggest that within the region and birth year studied, completeness of case ascertainment was high, with prevalence estimates of 7.6 per 1000 for children at both age 8 years and at age 15 years. For children common to both surveillance years, concordance in case status was also high (82%). Given that prevalence did not change within this region and birth year, continued research is needed to better understand the changes in prevalence estimates being found by the ADDM network across birth years.