Three-item Direct Observation Screen (TIDOS) for autism spectrum disorder

Abstract

We compared ratings on the Three-Item Direct Observation Screen test for autism spectrum disorders completed by pediatric residents with the Social Communication Questionnaire parent reports as an augmentative tool for improving autism spectrum disorder screening performance. We examined three groups of children (18–60 months) comparable in age (18–24 month, 24–36 month, 36–60 preschool subgroups) and gender distribution: n = 86 with Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.) autism spectrum disorders; n = 76 with developmental delay without autism spectrum disorders; and n = 97 with typical development. The Three-Item Direct Observation Screen test included the following (a) Joint Attention, (b) Eye Contact, and (c) Responsiveness to Name. The parent Social Communication Questionnaire ratings had a sensitivity of .73 and specificity of .70 for diagnosis of autism spectrum disorders. The Three-Item Direct Observation Screen test item Joint Attention had a sensitivity of .82 and specificity of .90, Eye Contact had a sensitivity of .89 and specificity of .91, and Responsiveness to Name had a sensitivity of .67 and specificity of .87. In the Three-Item Direct Observation Screen test, having at least one of the three items positive had a sensitivity of .95 and specificity of .85. Age, diagnosis of autism spectrum disorder, and developmental level were important factors affecting sensitivity and specificity. The results indicate that augmentation of autism spectrum disorder screening by observational items completed by trained pediatric-oriented professionals can be a highly effective tool in improving screening performance. If supported by future population studies, the results suggest that primary care practitioners will be able to be trained to use this direct procedure to augment screening for autism spectrum disorders in the community.

Autism spectrum disorders (ASDs) are a heterogeneous set of conditions that present with various degrees of impairments in social interaction, restricted interests/stereotypic behavior, and communication. For most of the subjects, problems begin in early childhood and even in infancy (Landa et al., 2007). It has been shown that in particularly high-risk infants, social interaction problems are evident during the first year of life (Zwaigenbaum et al., 2005). By 2 years of age, toddlers on the autism spectrum may have problems in social communication, play, language and cognition, as well as sensory and motor impairments (Zwaigenbaum et al., 2009). It has been increasingly emphasized that early intervention is critical in improving the outcome of ASD (Dawson and Burner, 2011). The pattern of onset of ASDs and benefits of early intervention for them, taken together, underscore a pressing public health need for more efficiently screening young children for ASDs. Despite a consensus on this issue and availability of nondirect methods for screening, there remain a number of important problems that need to be surmounted in order to make screening more effective.

The prevalence of ASD is now considerably higher than once thought and is estimated to be around 0.5 to 1% (Baird et al., 2000; Bertrand et al., 2001; Chakrabarti and Fombonne, 2001, 2005; Fombonne, 2003; Fombonne et al., 2001; Tidmarsh and Volkmar, 2003). Despite such a multiplicative increase in prevalence, probabilistically, a diagnosis of ASD remains a rare outcome. An important logistical concern, therefore, is the need to improve the accuracy of screening measures with high-enough sensitivity and specificity (Al-Qabandi et al., 2011) as well as meaningful predictive values.

Several parent-administered questionnaires have been investigated as possible screening tools for ASD, including the Checklist for Autism Spectrum Disorders in Toddlers (CHAT) (Baron-Cohen et al., 1992), the Modified CHAT (M-CHAT) (Robins et al., 2001), Pervasive Developmental Disorders Screening Test–2 (Siegel, 2004), and the Social Communication Questionnaire (SCQ) (Berument et al., 1999), among others. The sensitivity and specificity of these tests have ranged between .18 and 1.0 (reviewed by Johnson and Myers, 2007). When the relatively low prevalence is combined with low sensitivity and specificity of screening tools, the positive and negative predictive values for any screening measure have been low, resulting in unacceptably high false-negative and false-positive rates.

Paradoxically, in screening for ASD, the need for identifying cases at all levels of severity is salient, since early intervention has been in particular most optimal for children with less severe forms of ASD. The early diagnosis of ASD among these less severely affected children has been particularly challenging.

A two-stage systematic approach involving screening and subsequent clinical diagnosis is critically important. Currently, the best diagnostic method is to use expert clinical judgment as the “gold standard” for confirming the diagnosis combined with a standardized observational schedule, such as the Autism Diagnostic Observation Schedule (ADOS) (Lord et al., 1999), and a semistructured parent interview, such as the Autism Diagnostic Interview–Revised (ADI-R) (Lord et al., 1994). Substantial training is necessary to use these tools effectively in specialized clinical centers. One major challenge has been the considerable expenditure of time associated with the use of these tools. Furthermore, although the ADOS and ADI-R have now been translated into number of languages with a wide ranging effort by Western Psychological Services (WPS) in collaboration with the authors to validate them in various cross-national settings, the universal usage of these tools especially in low and middle income countries (LMICs) has significantly lagged behind. It has been very difficult to evaluate the results of parent-based screening measures with relatively low predictive values in resource-poor settings. From an ethical perspective, the use of poorly performing screening measures in the context of limited availability of second-stage clinical diagnosis in LMIC settings has been particularly problematic. It is essential that we develop cost-effective cross-nationally comparable approaches to high income country settings for screening for ASD for research and clinical services in LMICs to make any significant mark in the near future.

The purpose of this project was to develop a direct observational augmentative screening test that can be administered by trained pediatric-oriented professionals. An overarching goal of this innovative approach that has not been previously used was to lower the likelihood of false-positive results and make screening otherwise more meaningful for public health authorities in the intended LMIC setting. We proposed to compare the examination of children completed by trained pediatric psychiatric professionals with the use of direct observational items that included the following: (a) Joint Attention (following examiner’s cues in observing an object with direct gaze or pointing gesture), (b) Eye Contact, and (c) Responsiveness to Name (called by examiner on four occasions). We selected the three direct observation measures based on their (a) relevance to social interaction domain, (b) ease of administration particularly with young children, and (c) early developmental nature of the observational tasks not influenced by child’s education and early environmental opportunity. We did not select the items based on language/communication and restricted, repetitive, and stereotyped patterns of behavior and interests (Johnson and Myers, 2007; Maestro et al., 2006; Oner et al., 2012; Zwaigenbaum et al., 2005).

We also obtained parent reports on the SCQ across the ASD index and developmentally delayed (DD) and typically developing (TD) comparison groups of children. There are several studies that have investigated the sensitivity and specificity of SCQ in various populations. The original study in the United Kingdom reported good psychometric properties (Berument et al., 1999). However, that study was conducted with much older subjects. In later studies conducted in younger children (Allen et al., 2007; Corsello et al., 2007; Eaves et al., 2006; Lee et al., 2007; Oosterling et al., 2009, 2010; Snow and Lecavalier, 2008), sensitivity of the SCQ was between .40 and .89, and specificity was between .28 and .98. In a previous study, it has been shown that the Turkish form of the SCQ is valid and reliable in this age group (Oner et al., 2012). Our hypothesis was that augmentation with direct observation items and SCQ would be more sensitive and specific for ASD diagnosis than the SCQ alone.

Method

Index subjects

The study was conducted in a large child psychiatric outpatient unit of a public children’s hospital. The diagnoses of autistic disorder (AD) and pervasive developmental disorders–not otherwise specified (PDD-NOS) were made according to Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV) criteria; there were altogether 86 consecutively referred subjects in the ASD index group (ASD, n = 62; PDD-NOS, n = 24). Clinical diagnosis of ASD was made by two experienced child psychiatrists who directly examined the children using a structured Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR) checklist; subjects for whom there was independent agreement on presence of ASD by both raters were included in the study group. Distinction between AD and PDD-NOS was based on presence of full versus subthreshold symptom number in the affected ASD domains according to well recognized DSM-IV-TR rules. The inclusion criteria were the following: (a) age of 18–60 months, (b) DSM-IV-TR diagnosis of AD or PDD-NOS assigned by both raters, and (c) parental consent. Subjects were screened for sensory hearing and/or visual impairments for exclusion, but none were detected. All ASD subjects with expressive language level above 16 months based on Vineland Adaptive Behavior Scale (VABS) were evaluated with the ADOS, and DSM-IV diagnosis was in accordance with ADOS diagnosis in subjects who were evaluated with ADOS.

Comparison groups

We included two groups of comparison children from the same outpatient setting: (a) speech/language and/or cognitive impairments (DD without ASD or “DD,” n = 76) and (b) TD (n = 97).

Assessments

All the ASD index and DD and TD comparison children were assessed by the VABS (Sparrow et al., 1984) and the SCQ (Berument et al., 1999). The VABS was administered by trained staff psychologists to mothers as the primary caregivers in all cases. The SCQs were self-administered by parent informants (predominantly mothers) in the clinical setting with help from staff psychologists and residents as requested.

SCQ

The SCQ is a 40-item rating scale filled by the parents (Berument et al., 1999) and based on ADI (LeCouteur et al., 1989). It covers communication, reciprocal social interaction, and repetitive behavior and interest domains. Each item is filled as “yes” or “no,” and abnormal behaviors are assigned “1” point, while absence of those behaviors is assigned “0” point. The first item that indicates whether the child has sufficient verbal ability is not scored; the following six language items are also skipped in nonverbal children. The points are summed to obtain a total score. In this study, we used the current approved Turkish WPS version of SCQ (Oner et al., 2012).

VABS

The VABS was used to assess adaptive behavior (Sparrow et al., 1984). VABS is a 297-item interview consisting of three domains (communication, daily living skills, and socialization). The communication domain includes items concerning receptive and expressive communication. Socialization includes interpersonal relations, play and leisure time, and coping skills. Previous studies have shown that in subjects with ASD, VABS is sensitive to the severity of impairment (Carter et al., 1998; Klin et al., 2007; Saulnier and Klin, 2007). The Turkish translation is shown to be valid and reliable (Alpas and Akcakin, 2003).

The Three-Item Direct Observation Screen (TIDOS) measures were Joint Attention (following the examiner’s verbal cues and pointing), Eye Contact, and Response to Name (called by the examiner four times, see Appendix 1 for operational definitions of the scores). Joint Attention and Response to Name were scored as 0 if the response was normal, 1 if the response was not optimal, and 2 if the response was absent. Eye Contact was scored as normal (0) and abnormal (1). The pediatrics residents, who had a 1-month training period, were trained to score the observation items for a single day by the first author (P.O.).

ADOS

All ASD subjects with expressive language level above 16 months based on VABS were evaluated with the ADOS (Lord et al., 1999). The ADOS (Lord et al., 1999) is a standard observation schedule designed to assess ASD-related behaviors in a social setting that is suitable for the child’s developmental level. In the present study, we used Modules 1 and 2. The ADOS is coded by trained observers; the inter-rater reliability, which is tested periodically, for the total score was more than 0.90%.

Procedure

All the children were evaluated by means of physical and psychiatric examination, sociodemographic questionnaire, DSM-IV-TR ASD criteria checklist, the VABS, the SCQ, and the TIDOS augmentation measures. For subjects with ASD and DD, additional audiometric, neurological evaluations as well as genotyping measures were routinely obtained. The TIDOS measures were completed by the pediatric residents blind to diagnostic status of the subject who were trained in the use and scoring of each observation item. Pediatric residents were trained on the administration of the items by observing two administrations of the authors followed by the supervision of two other administrations conducted by themselves. Observation items were filled before ADOS administration and independently. The training was conducted by P.O. who has completed research training in the use of ADOS and ADI-R in the United States. This training included demonstration and practice in the rating of each observation item with 80% or higher reliability for each rating. The observation began with a free play period of 5 min; during this period, parent interview is conducted. All children were evaluated while their parents were present. The observation took 15 to 20 min. In summary, the whole procedure was as follows: Step 1—TIDOS; Step 2—Physical examination, SCQ, VABS, laboratory workup; Step 3—ADOS if possible; and Step 4—DSM-IV diagnosis made by P.O. and O.O.

Data analysis

Categorical variables were compared with the chi-square test. Continuous variables were compared by using analysis of variance and post hoc Tukey test. Initial analysis of the whole group was followed by separate analysis of three age groups (18–24 months, 24–36 months, and 36–60 months). Another analysis was conducted to compare the subjects with different developmental levels. Receiver operating characteristic (ROC) analysis was used to compute the sensitivity, specificity, and cut-off values. All p-values are two-sided.

Results

Tables 1 and 2 summarize the gender, age, VABS, SCQ, and results of three direct observation items of the ASD, DD, and TD groups. Figure 1 shows the distribution of VABS and SCQ variables of the groups. There were fewer number of girls in the ASD group, but age distributions were similar. VABS scores were lowest for the ASD group, followed by the DD group, and highest for the TD group. In contrast, SCQ total score was highest in the ASD group followed by the DD group. Some of the SCQs were administered prior to diagnosis and some after it (n = 26). There were no significant differences in terms of SCQ scores between these two groups.

Table 1.

Gender and observation item distribution of subjects with autism spectrum disorder (ASD), subjects with developmental delay (DD), and typically developing (TD) children with chi-square analysis.

	ASD	DD	TD	χ2; p
Gender (boys%)	77.9%	69.7%	60.8%	6.5; .05
Response to Name	1:17.4%; 2:50.0%	1:5.3%; 2:5.3%	1:1.0%; 2:0%	119.9; <.001
Joint Attention	1:28.2%; 2:54.1%	1:5.3%; 2:5.3%	1:0%; 2:0%	165.2; <.001
Eye Contact	1:89.0%	1:9.2%	1:0%	170.7; <.001
Any item positive	95.3%	18.4%	1.0%	181.1; <.001
All items positive	60.0%	5.3%	0%	113.8; <.001

Table 2.

Mean and standard deviation of age, Social Communication Questionnaire (SCQ), and Vineland Adaptive Behaviors Scale (VABS) scores of subjects with autism spectrum disorder (ASD), subjects with developmental delay (DD), and typically developing (TD) children with analysis of variance.

	ASD	DD	TD	F; p
Age (months)	35.6 ± 11.03	39.4 ± 10.95	38.7 ± 10.6	3.0; .054
VABS Adaptation Level	16.8 ± 5.7	23.1 ± 9.4	33.5 ± 9.7	79.8; <.001
VABS Expressive Language	10.6 ± 6.0	18.4 ± 9.1	32.8 ± 11.2	122.1; <.001
SCQ Factor 1	14.3 ± 5.4	7.9 ± 5.1	4.6 ± 4.1	89.7; <.001
SCQ Total	19.3 ± 5.5	12.9 ± 6.1	9.6 ± 5.5	50.1; <.001

Figure 1.

Error-bars representing the distribution of (a) SCQ and (b) VABS scores of typically developing, autism spectrum disorder, and developmentally delayed groups.

VABS: Vineland Adaptive Behavior Scale; SCQ: Social Communication Questionnaire.

When the three direct observation items were taken into account, each was significantly more commonly endorsed in the ASD group.

Table 3 summarizes sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios for individual direct observation items when any item was positive, when all items were positive, for SCQ total score more than 14, when SCQ Social Interaction Problems score was more than 8, when both SCQ Social Interaction Problems score was above cut-off and at least one observation item was positive, and when SCQ Social Interaction Problems score was above cut-off or at least one observation item was positive.

Table 3.

Sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios in the whole group and three age bands.

		Sensitivity	Specificity	Positive predictive value	Negative predictive value	Positive likelihood ratio	Negative likelihood ratio
Response to Name	Total	.67	.94	.87	.85	11.1	.35
	18–24 months	.75	.83	.88	.67	5.9	.30
	24–36 months	.62	.94	.80	.86	10.3	.40
	36–60 months	.68	.97	.90	.88	22.7	.32
Joint Attention	Total	.82	.95	.90	.92	16.4	.19
	18–24 months	.90	.92	.95	.85	11.2	.11
	24–36 months	.79	.97	.91	.91	26.3	.21
	36–60 months	.85	.95	.87	.93	17.0	.16
Eye Contact	Total	.89	.96	.91	.95	22.2	.11
	18–24 months	1.0	.92	.95	1.0	12.5	—
	24–36 months	.81	.97	.91	.93	27.0	.21
	36–60 months	.90	.96	.90	.96	22.5	.10
Any item positive	Total	.95	.91	.85	.98	10.5	.05
	18–24 months	1.0	.83	.91	1.0	5.9	—
	24–36 months	.92	.92	.83	.97	11.5	.08
	36–60 months	.95	.96	.83	.98	23.8	.05
All items positive	Total	.60	.98	.93	.83	30	.40
	18–24 months	.75	.92	.94	.69	9.3	.27
	24–36 months	.50	.99	.93	.83	50	.50
	36–60 months	.59	.98	.92	.85	29.5	.41
SCQ Total (cut-off = 14.5)	Total	.80	.74	.60	.87	3.1	.27
	18–24 months	.90	.71	.81	.83	4.7	.14
	24–36 months	.70	.77	.57	.85	3.0	.39
	36–60 months	.83	.70	.55	.90	2.8	.57
SCQ Factor 1 (cut-off = 8.5)	Total	.89	.81	.66	.92	4.7	.14
	18–24 months	.95	.75	.86	.90	3.8	.06
	24–36 months	.77	.74	.55	.88	3.0	.31
	36–60 months	.90	.78	.64	.95	4.1	.12
SCQ cut-off or at least one item positive	Total	.99	.78	.69	.99	4.5	.01
	18–24 months	1.0	.66	.83	1.0	2.9	—
	24–36 months	1.0	.77	.63	1.0	4.3	—
	36–60 months	.98	.81	.68	.99	5.2	.02
SCQ cut-off and at least one item positive	Total	.69	.98	.94	.87	34.5	.31
	18–24 months	.89	.92	.94	.77	11.1	.12
	24–36 months	.62	.97	.88	.86	20.7	.39
	36–60 months	.80	.95	.86	.92	16.0	.21

SCQ: Social Communication Questionnaire.

Table 3 also summarizes these variables for three age groups. In general, sensitivity was higher and specificity was lower in the 18–24 months age group.

Table 4 compares the groups with different developmental levels as assessed by VABS. A DD greater than 9 months for children under 24 months of age and a delay greater than 12 months for older children in VABS were labeled as significant delay. In general, there was little difference between the delayed and nondelayed groups in terms of TIDOS items; however, sensitivity was higher and specificity was lower in the delayed group when SCQ was taken into account. In fact, SCQ total score was significantly higher in ASD subjects with significant delay (16.3 ± 6.1 vs 20.4 ± 5.1, F = 8.8, p = .004).

Table 4.

Sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios in the group with significant developmental delay versus others.

		Sensitivity	Specificity	Positive predictive value	Negative predictive value	Positive likelihood ratio	Negative likelihood ratio
Response to Name	Delay−	.63	.95	.82	.92	7.4	.38
	Delay+	.69	.92	.94	.76	3.9	.34
Joint Attention	Delay−	.84	.98	.83	.97	42	.16
	Delay+	.84	.91	.95	.86	9.3	.18
Eye Contact	Delay−	.89	.97	.89	.98	29.7	.11
	Delay+	.89	.92	.92	.90	11.1	.12
Any item positive	Delay−	1.0	.94	.79	1.0	16.6	—
	Delay+	.93	.85	.85	.93	6.2	.08
All items positive	Delay−	.52	.99	.91	.90	52	.48
	Delay+	.65	.95	.93	.74	13	.37
SCQ Total (cut-off = 14.5)	Delay−	.63	.83	.46	.91	3.7	.45
	Delay+	.85	.55	.66	.78	1.9	.27
SCQ Factor 1 (cut-off = 8.5)	Delay−	.79	.89	.62	.95	7.2	.23
	Delay+	.90	.56	.67	.85	2.0	.17
SCQ cut-off or at least one item positive	Delay−	1.0	.79	.52	1.0	4.8	.01
	Delay+	.98	.50	.66	.97	1.9	.04
SCQ cut-off and at least one item positive	Delay−	.63	.98	.86	.92	31.5	.37
	Delay+	.80	.91	.89	.83	8.9	.22

SCQ: Social Communication Questionnaire.

ASD cases with DSM-IV diagnosis of PDD-NOS were significantly more frequently missed by the first two items (Response to Name: PDD-NOS 66.7% vs autism 19.4%, χ² = 17.6, p < .001; Joint Attention: PDD-NOS 47.8% vs autism 6.5%, χ² = 17.6, p < .001; Eye Contact: PDD-NOS 18.2% vs autism 8.3%, χ² = 1.6, p = .24). For TIDOS items, missed cases in terms of Response to Name had lower SCQ Social Interaction and higher VABS General Adaptation scores, suggesting higher functioning. Cases with false-negative Joint Attention item have a significantly higher VABS Interaction Domain score. Cases missed at Eye Contact item had a trend of higher VABS Expressive Language score. On the contrary, false-positives on all three items were more likely to have lower VABS General Adaptation scores (F = 5.9–9.5, p = .017–.003), showing that subjects with significant DD were at increased risk of being incorrectly classified as “at risk” on the measurement.

In the whole group, all observation items were highly (more than .90) specific. Having any item positive was both sensitive and specific with a very high negative predictive value. SCQ Social Interaction Problems score over 8 was more specific and sensitive than SCQ total score. Using both SCQ cut-off and observation items increased sensitivity and using any of the two increased specificity. When the likelihood ratios were taken into account, using both SCQ cut-off and observation items led to a very high positive likelihood ratio and using either of these led to a very high negative likelihood ratio.

Discussion

Although TIDOS was intended as a possible augmentative tool to help improve screening in the community in the LMIC context, our results clearly showed that TIDOS measures were sensitive and specific in distinguishing ASD, DD, and TD in young children. Furthermore, the direct TIDOS measures outperformed the SCQ. Nevertheless, the sensitivity and specificity measures for the SCQ in the present study were also high and consistent with a previous study (Oner et al., 2012) and showed that the SCQ Social Interaction Problems score was most sensitive and specific for ASD diagnosis.

As elaboratively discussed by Al-Qabandi et al. (2011), screening for ASD posits serious challenges. Nevertheless, evidence-based early therapies now do exist, and more optimal screening for ASD needs to be made available to improve the response time in the healthcare system. In the present study, our aim was to examine whether direct observation items that comprised surrogate measure of “social interaction” could provide a more sensitive and specific means of screening for ASD compared to the use of parent-based rating scales such as SCQ. Since our study was conducted in a LMIC setting, we emphasized the need for a more direct approach as a cost-effective means of better identifying problem behaviors in the community, given the relative insufficiency of education and awareness of developmental factors among parents. Even one positive item in the TIDOS assessment led to a highly specific and sensitive differentiation and negative predictive value.

Previous studies have shown that social interaction problems emerge early and are more specific to ASD when compared to language-based concerns (Johnson and Myers, 2007; Maestro et al., 2006; Zwaigenbaum et al., 2005). ASD subjects have significantly much lower interest in sharing their emotions and to respond to others (Rogers and Benetto, 2000). Furthermore, Joint Attention impairments can be noted very early among ASD children (Charman, 2003; Chawarska et al., 2003; Mundy and Markus, 1997). Declarative pointing emerges early in development, around 12–14 months, and is closely associated with language development (Lord, 1995). Eye Contact problems can also be detected early in subjects with ASD (Johnson and Myers, 2007). Responding consistently to his or her name can be deficient in children with ASD. In fact, all of the behaviors we selected have their developmental origins quite early in development. Children follow their parent’s cues approximately at 10–12 months of age and “look when their names are called” by 8–10 months of age, and 5- to 6-months-old infants have clear Eye Contact (Johnson and Myers, 2007). On the contrary, speech problems are common in children with DD, as was reflected in our results. The SCQ ratings obtained from the parents indicated that 86% of the subjects with ASD and 63.5% of the subjects with DD did not have speech. This was only true for 15.5% of the control children. Lack of speech did not permit us to study more specific types of speech problems across the study groups. Nevertheless, our results suggest that speech problems may not be suitable to differentiate between DD and ASD subjects in this age group. This is consistent within the newly envisioned Diagnostic and Statistical Manual of Mental Disorders (5th ed.; DSM-5) diagnostic classification of ASD, which emphasizes social interaction items as a central distinguishing factor in the diagnosis of ASD. The more specific nature of social interaction problems was also reflected and supported by our SCQ results. It has been reported that Social Interaction Problems factor of SCQ was more sensitive and specific than the total score (Oner et al., 2012).

It has been argued that a combination of parent interview and direct observation is optimal for the diagnostic assessment of ASD (Zwaigenbaum et al., 2009). Therefore, it can be reasonable to argue that a combination of parent ratings with direct observation examination of the child by means of a structured assessment of performance on social interaction items may be more optimal. In the present study, we found that taking SCQ Social Interaction Problems score and presence of at least one positive observation item into account led to very high specificity and positive predictive value. On the contrary, having SCQ Social Interaction Problems score above cut-off or having one positive observation item led to very high sensitivity and negative predictive value. Therefore, depending on the objectives of the study, both can be used.

It must be noted that in this study, we decided to compare the observation test items with the SCQ, as this rating scale has concurrence with the ADI-R items. The specificity and sensitivity of the SCQ were similar to what has been reported in previous studies. The items that were chosen for observation also align well with items on the M-CHAT (Robins et al., 2001) as well as on the SCQ. As noted, our original goal was to use the direct observational items to augment and to improve the screening performance of the SCQ. An underlying rationale for this has been the need for development of more effective screening tool to empower primary care practitioners in a middle income country with a reasonably efficient network of primary care centers and to build a country-wide framework for launching an epidemiological study of ASD. At the very least, the observational test items selected would provide practitioners an opportunity for training in direct observation of children with respect to increased awareness of ASD symptoms.

Our aim was to develop a simple, easy-to-learn screening observation, which tapped into a relatively intuitive and family-friendly assessment of social interaction impairments in young children with ASD. Our results indicated that the social interaction problems can be observed during a short-term examination by physicians after a brief training, which can also be completed online or by following training DVDs. We believe that nurses and other health professionals who have experience in working with young children can also be effectively trained to examine social interaction problems. Indeed, training primary care physicians or nurses to rate these items may be a useful means to screen ASD, in particular in the LMIC settings. For example, in the country the study was conducted, each child is evaluated by the family practitioners and almost all children can be screened for ASD.

We would like to emphasize that our results do not indicate that the use of parent-based ratings have not been helpful but rather that there is further value in a structured assessment of the children as part of an improved and more interactive screening program. It took only 1 day for the residents to effectively use the observation items suggesting that a web-based training module has the potential of reaching out to a large number of practitioners in the community.

Finally, in terms of applications in other low resource settings, it is conceivable that such augmentative direct observations of children within the primary care context can improve ASD awareness and screening. A major goal is decreasing false-positive rates where there is such limited access to full diagnostic evaluations with the use of standardized diagnostic measures. Increasing the specificity of ASD screening while not having a detrimental effect on sensitivity is a highly important public health goal.

Limitations

The present study needs to be viewed in the light of number of limitations. First, we could not investigate the specificity and sensitivity of the observation items in differentiating between DSM-IV AD and PDD-NOS since our sample size was small in the ASD index group. Nevertheless, this differentiation between AD and PDD-NOS subtypes of ASD is less likely to be salient with the newly proposed merged classification in DSM-5. On the contrary, data suggested that subjects with PDD-NOS were more likely to be misclassified with the observation measure. This is important since in a population study, more subjects are likely to show milder or subthreshold clinical symptoms, as in PDD-NOS, which may lead to a higher rate of false-negatives than found in the present study. Second, the DD comparison subjects were older than those in the ASD index and the TD groups, which might lead to better ratings in the VABS. Nonetheless, age was not significantly correlated with SCQ, suggesting that this may not be a significant factor for the severity of ASD symptoms. However, developmental level was a significant factor for misclassification and seemed to be even more salient for SCQ scores. Again, in a nonclinical sample with better developmental level, a higher rate of false-negatives seems possible. Age was another important factor affecting the sensitivity and specificity of the measures; however, general findings still hold after we reanalyzed the data for age groups. It must also be kept in mind that when stratified by age subgroups, sample size in each group was smaller, and this might increase the chance of statistical error. Third, it is impossible to conclude whether the observation items can be used in population-based screening in a case-control study like the one presented here, which was conducted in a well-supported outpatient clinic with trained staff. On the contrary, case-control approach is necessary as an initial phase to determine whether the proposed method is sensitive and specific enough to test its use in the general population. Particularly, when the prevalence of a disorder is still relatively low, like ASD, the magnitude of prevalence is one of the most important factors in determining positive and negative predictive values of a screening instrument. At the subsequent phase, the usefulness of the observation ratings can then be evaluated in a general population context.

Appendix 1

Three observation items

(Before beginning the observation, place 2–3 toys that can attract the child)

1. Call the name of the child while he or she is interested in something else and not looking at you. (You can try up to four times if the child is not responding. Call only the child’s name. Do not say anything else.)

   • 0If the child looks at you at the first two calls.
   • 1If the child looks at you at your third or fourth call or the child looks at his or her parents/caregivers in the first two calls.
   • 2If the child does not look at you or at his or her parents/caregivers.

2. By using your index finger, point to an object (that must be out of reach for both you and child), which can interest the child, say “Ali, look …” When doing this, first look at the child, and then the object and again to the child and observe whether the child follows your gaze. If the child does not respond repeat once more.

   • 0If the child follows your gaze and turns to the target object.
   • 1If the child turns directly to the object without making eye contact with you.
   • 2No interest.

3. When scoring this item, you must distinguish eye contact, which is clear, flexible, socially oriented, and repeated several times in different contexts during the examination and interview from restricted, rare, and inappropriate ones. If the child seems shy and his or her behaviors change when he or she relaxes, score according to the later behaviors. However, if the child’s behavior persists, score what you observe.

   • 0Clear, flexible, appropriate eye contact accompanying and consistent with other gestures and mimics.
   • 1Does not use or rarely uses eye contact to initiate, sustain, or regulate social interaction.