Cross-Cultural Adaptation, Reliability, and Validity of The Autism Treatment Evaluation Checklist in Persian

Abstract

Objective

The objectives of the current study were to translate and adapt Autism Treatment Evaluation Checklist (ATEC) into Persian language and to investigate its reliability and validity in an Iranian autistic sample.

Methods

A total sample of 134 children with autism spectrum disorders aged 6-15 years were assigned to the study. The process of cross-cultural adaptation was performed according to international methodological steps as following: translation, back-translation, revision by an expert committee and pretest. A sample of 20 primary caregivers of autistic children were pretested. The content validity of the ATEC was reviewed by the expert committee all through the stages. The construct quality of the questionnaire was evaluated by comparison of the adapted version of the instrument with similar tests assessed similar factors. Moreover, the reliability of the questionnaire was evaluated through stability and homogeneity assessments.

Findings

The results showed good content validity and internal consistency (Cronbach's alpha: 0.86-0.93). In relation to construct validity, there was significant correlation between ATEC subscales and raw data obtained from Autism Diagnostic Interview-Revised (ADI-R) (r=0.38-0.79). The Intraclass Correlation Coefficient for the test–retest reliability was excellent for all the subscales and also for total scores (ICC: 0.79 - 0.93).

Conclusion

Cross-cultural adaptation of ATEC was successful. The psychometric properties were verified and indicated that the adapted questionnaire is valid and reliable to use in Iranian culture.

Introduction

Autism is a complex developmental disorder which has lifelong effects on several aspects of an individual^[1]. Although, the autism spectrum disorder (ASD) is known to be neurogenetic in origin^[2], its diagnosis is primarily based on behavioral and clinical signs and symptoms^[2]. According to Diagnostic and Statistical Manual of Mental Disorders, 4^th edition, text-revision (DSM-IV-TR)^[3], and International Classification of Diseases (ICD)^[4], there are three main diagnostic criteria for autistic disorders: impairments in social interaction, impairments in communication and language and restricted, stereotyped behaviors, interests and activities^[5]. Regarding the neurologic and genetic basis of autism, a comprehensive diagnostic evaluation is supposed to include psychological and genetic evaluations. However, so far regularly standard diagnostic tests are based on psychological and behavioral assessments such as Autism Diagnostic Interview-Revised (ADI-R)^[6], Autism Diagnostic Observation Schedule (ADOS-G)^[7] and Childhood Autism Rating Scale (CARS)^[8].

Recently, autism studies have shown an increased interest in examining the effect of different developmental, educational and behavioral interventions on children with ASD after their diagnosis is confirmed by diagnostic instruments^[9]. Several researchers have administered longitudinal studies to monitor the long lasting improvement of autistic children^[10–12]. As an example Glen et al evaluated outcomes including cognitive, language, adaptive, social and academic measures after 4 years of intensive behavioral treatment^[10]. They found that about half of their autistic children achieved average post treatment scores and succeeded in regular education classrooms by showing rapid learning. However, till recent, to choose a valid and proper measure assessing the changes and progress in autistic populations is a major controversial issue^[9]. For years, there was no specific measure while instead, measures such as ADOS, ADI-R or CARS have being used to examine the symptoms, evaluate changes and improvement in response to different interventions. Although these measures display overall stability over time, they are primarily designed as diagnostic tools but not sensitive enough to examine symptom severity and also intra-subject changes. Moreover, the assessment tools provided for typically developing (TD) children have been used to evaluate autistic individuals. However, these measures are not often suitable for autistic children since they commonly show a too far different developmental pattern compared to their TD peers.

Another problem in assessing long-term changes is the lack of an instrument which can evaluate autism severity along with developmental deficits. Those cannot cover a broad age range from childhood to adolescence and beyond^[9]. Although, some of them such as Vineland Adaptive Behavior Scales (VABS)^[13] as an informant based measure do cover a wide age range, it is not appropriate for comparing autistic with normative data. Hence this schedule focuses mainly on developmental profile which in autism is very delayed and deviant^[9].

Increasing number of children classified as ASD over the past decade^[14], alarms a vital need for early life interventions^[15]. Hence, in order to establish a reliable baseline for tracking the trajectory of early treatments, sensitive monitoring tools are becoming more mandatory^[9]. However, given several constraints of families, schools and service providers of individuals with ASD, it is important to provide symptom severity profile in a time-efficient, economic and practical way^[9].

Autism treatment evaluation checklist (ATEC) is a one paged checklist designed to be completed by parents, teachers or caretakers and is a simple but effective tool to assess the severity of symptoms as well as developmental aspects of autism^[61]. Furthermore, it can measure the effectiveness of various autism treatments. In other words ATEC also fulfills the need for a valid, easy to administer and sensitive to change instrument.

The ATEC which totally contains 77 items, covers four main impairment areas of ASDs including communication, sociability, sensory-cognitive awareness and health-physical-behavior (for more details on the scale, see methods). The ATEC is freely available and can be scored online with minimal training and resources.

A considerable amount of literature administered autism treatment evaluation checklist, has described that ATEC is sensitive to intra-subject changes after treatment programs^[17–19]. In a recent investigation, Megiati et al^[9] examined the internal consistency and predictive validity of ATEC. Given their findings, they introduce ATEC as a potentially useful and promising tool for gathering reliable data on current behaviors and skills as well as general functioning of children with autism spectrum disorders. Moreover, other findings have shown that ATEC data were significantly correlated with other equivalent diagnostic tools ((e.g. Pervasive Developmental Disorders Behavior Inventory (PDD-BI): r=0.87 or Severity of Autism Scale (SAS): r=0.7))^[20]. Given together, one can argue that ATEC is a potentially reliable and valid tool for monitoring progress over time.

It is noteworthy that exploring the standardization of a questionnaire is a continuing dynamic process and different investigations with different samples are supposed to examine the validity and reliability of the instrument. In addition, concerned for the fact that most of the health related questionnaires and scales have been developed in English speaking countries, the need to adapt the questionnaires in other than the source language has grown rapidly^[21]. Cross-cultural adaptation is a valid process through which reliable health status measures may be obtained in order to be used in different countries in spite of different sociocultural conditions. The process of adaptation provides a ground to measure a same phenomenon using a same instrument across different cultures. So far, however, there have been few studies applying ATEC in different countries in order to be implemented in other languages than English as well as monitoring the validity of the instrument. Thus, the purpose of the present study is to investigate the cross-cultural adaptation, validity and reliability of ATEC questionnaire in an Iranian autistic sample.

Subjects and Methods

Participants

This project used a convenience sample of 134 children and adolescents with autism spectrum disorders (111 boys and 23 girls) aged 6-15 years (mean: 9.6, SD: 1.97). Participants were invited from autism-specific schools in Tehran.

All children met criteria for a diagnosis of autism on both the DSM-IV^[3], and ADI-R^[6], while the diagnosis was established in a previous assessment by either a child psychiatrist or psychologist. The child's parent or caregiver completed written informed consent before they were assigned to the study. The study was approved by the Medical Ethics Committee of Tehran University of Medical Sciences.

Procedure

The current study used the essential methodological steps suggested by internationally recognized publications for the procedures involved in the cultural adaption of measurement instruments^[21–23]. Cross-cultural adaption stages were followed as below:

Forward translation: In this stage, the questionnaire was translated from original language (English) to target language (Persian) by 2 bilingual translators whose mother tongue was Persian. One of the translators was aware of the concepts examined by the instrument and had translated such medical questionnaires before. The other translator (also called native translator) was neither aware nor informed of the concepts and aims of the measurement instrument and had no medical background. In order to approach to a conclusive data, the results produced by both translators were compared with each other by the translators and a recording observer (one of the researchers involved in the present study). The final translation draft was also compared with the original English source.

Back translation: The final Persian translation was again back translated into English by 2 bilingual, native English-speaking translators who were totally blind to the original version. They were neither aware of the concepts and aims of the questionnaire nor had academic training in autism.

Expert committee: The final Persian translation and the back translation were compared and reviewed by a multidisciplinary, expert committee to obtain a final version. The committee was composed of the translators, a psychologist, a methodologist, and a medical doctor. On the way to guarantee accurate comprehension, the members of the committee evaluated and reviewed the topics of each section while taking into account the semantic, idiomatic and cultural equivalents and the intelligibility of the items. The committee also reviews all the ATEC drafts (i.e. the original, forward and back translation drafts) and reveals their suggestions about each item; finally the last version of the draft was produced.

Pretesting: The last stage of cross-cultural adaptation is the pretest. A sample of 20 primary caregivers of autistic children and adolescents was pretested in order to verify the comprehensibility of the statements and questions, to assess the equivalence of the instrument within the Iranian culture and to recognize the errors in the final version.

In order to assess the psychometric properties of the translated version of the instrument, the standardized factors were evaluated:

Content validity: The content quality of the autism treatment evaluation checklist was reviewed by the expert committee all through the cultural adaptation procedure. The items or questions would also have been revised if 15% of the participants had difficulty in the comprehension of the items in the pretest stage.

Construct validity: The construct quality of the questionnaire was evaluated in order to ascertain that the Persian version of the ATEC really measures what it is expected to measure by comparison of the adapted version of the instrument with similar tests that assess similar factors. The comparison was made against ADI-R. Each subscale of ATEC was examined with its equivalent from ADI-R; for example ATEC subscale 1 entered the analysis paired with verbal subscale of ADI-R. In this way, there was more assurance that the adapted version is measuring a construct comparable to the original^[22].

Reliability: The reliability of the questionnaire was evaluated by measuring the internal consistency of all the items within each subscale of the questionnaire and stability of the instrument as well (test-retest).

Measures: Autism treatment evaluation checklist ATEC consists of 4 subscales: 1: speech/language/ communication (14 items; maximum score: 28); 2: sociability (20 items; maximum score: 14); 3: sensory/cognitive/awareness (18 items; maximum score: 36), and 4: health/physical/ behavior (25 items, maximum score: 75). Items on subscales 1-3 are scored from 0 (not descriptive) to 2 (very descriptive). Scoring on subscale 4 ranges from 0 (not a problem) to 3 (serious problem). The total maximum score is 179 with a higher score representing higher severity of autistic behaviors and poorer social developmental skills, a decrease in scores indicates progress and improvement in autistic problems.

ADI-R^[6] which provides a comprehensive assessment of individuals suspected to have autism spectrum disorders, was also used in this project. ADI-R consists of 93 items and focuses mainly on three functional domains: language and communication, reciprocal social interactions, restricted, repetitive and stereotyped behaviors and interests. The interview also addresses other clinical factors like aggression, self-injury and possible epileptic features. For administrating ADI-R an experienced interviewer interviewed a parent or caregiver following highly standardized procedures. Data were scored and interpreted by using a diagnostic algorithm or a current behavior algorithm.

Data analysis

Construct validity of the instrument was evaluated by demonstrating the correlation between ATEC and ADI-R according to Pearson correlation; the set point for P. value was 0.05. Internal consistency (reliability) of the instrument was confirmed by Cronbach's coefficient alpha and Guttman split-half coefficient; an acceptable internal consistency was defined as a value >0.7. Furthermore, the stability of the instrument was evaluated using the test-retest reliability method; the data obtained in first test session and retest session (separated by 2 weeks) were analyzed using the Intraclass Correlation Coefficient to assess the reliability of all the scales measured.

Findings

Table 1 shows descriptive data for ASD symptoms evaluated by ATEC questionnaire.

Table 1.

Descriptive data and internal consistency coefficients for each ATEC subscale and total scores

ATEC	Number of items	Range	Mean (SD)	Cronbach's alpha	Guttman coefficient
Speech/ Language/ Communication	14	0 - 23	10.9 (5.9)	0.89	0.77
Sociability	20	0 - 32	12.2 (6.4)	0.86	0.78
Sensory/ Cognitive/ Awareness	18	1 - 33	17 (6.1)	0.86	0.81
Health/ Physical/ Behavior	25	4 - 58	23.4 (12)	0.86	0.74
Total ATEC	77	1 - 128	62.4 (23)	0.93	0.77

ATEC: Autism Treatment Evaluation Checklist / SD: Standard Deviation

Cross-cultural adaptation process

The procedures of translation, back translation and submission of the instrument to the expert committee showed that there was no need for significant changing of the meaning of items or adding and removing the statements. During the pretest stage, a direct interview with the participants was performed in order to appraise the difficulties in completing the questionnaires or to identify any misunderstanding in items or statements. Participants who were interviewed in this stage reported no difficulties in comprehending the content of each items; however two complained that they had difficulties in recognizing the extent to which their child had problems during answering to some items (in language and communication subscale).

Following further discussion with the expert committee, authors found no need to significant change in wording and the reported problem seemed to be related to the information reminiscence situation. However, according to the expert committee discretion, only the statements of two items were refined by adding a word. In the sociability subscale, the word "his/her inside" was added to the statement of the item 12 in parentheses. Moreover, in the sensory/cognitive/ awareness subscale an exemplification for the word "tuning in" was added to the statement of the item 17. The participants reported that the added words and expression made the sentences more comprehensible.

Reliability-Internal consistency (Homogeneity)

The internal consistency was evaluated by means of Cronbach's alpha and Guttman split-half methods. The results showed a high internal consistency of the instrument for total score in both methods (Cronbach's coefficient alpha: 0.93; Guttman split-half: 0.77). Internal consistency of the four ATEC subscales was also excellent in both methods (Table 1). Stability (test-retest)

Test-retest reliability of the ATEC was calculated using Intraclass Correlation Coefficient. The stability was excellent for all the subscales and also for total scores (Table 2).

Table 2.

Test-retest reliability scores of the ATEC (Persian version)

ATEC	Number of items	R	95% CI	P. Value
Speech/ Language/ Communication	14	0.87	0.83-0.90	<0.001
Sociability	20	0.93	0.89-0.96	<0.001
Sensory/ Cognitive/ Awareness	18	0.92	0.89-0.94	<0.001
Health/ Physical/ Behavior	25	0.79	0.75-0.84	<0.001
Total ATEC	77	0.89	0.85-0.93	<0.001

ATEC: Autism Treatment Evaluation Checklist / CI: Confidence Interval

Validity

Data obtained for construct validation were submitted to statistical analysis using Pearson correlation. The achieved values of the ATEC subscales and related ADI-R subscales are shown in Table 3. Results showed significant positive association between each pair variables, language and behavioral subscale indicated highest interrelation (r=0.7 and 0.79).

Table 3.

Construct validity of the ATEC: correlations between ATEC subscales and ADI-R raw data

ADI-R Raw data		Verbal subscale		Social subscale		Non-verbal subscale		Behavior subscale
ATEC		R	P. Value	R	P. Value	R	P. Value	R	P. Value
Speech/Language/Communication		0.7	<0.001	-	-	-	-	-	-
Sociability		-	-	0.54	<0.001	-	-	-	-
Sensory/Cognitive/Awareness		-	-	-	-	0.38	<0.001	_	_
Health/Physical/Behavior		-	-	-	-	-	-	0.79	<0.001

ATEC: Autism Treatment Evaluation Checklist / ADI-R: Autism Diagnostic Interview-Revised

Discussion

The current study was set out to adapt "Autism Treatment Evaluation Checklist" to Persian language. Given the complex nature of autism spectrum disorders, there is a subsequent need for a comprehensive battery of diagnostic and monitoring instruments. Hence, cross-cultural adaptation of the ASD questionnaires is worthy to be investigated. In such a way, a wide range of data can be provided using different language versions of a questionnaire for ASD in different societies. Alongside other formal evaluation tools for autism spectrum disorders, ATEC can be a routine measure in school or for clinic based monitoring procedures^[9]. However, to our knowledge there were already few published studies investigating cultural adaptation of the ATEC in languages other than English. Although a few ASD scales have been translated into Persian and being used in autism schools and clinics (e.g. Autism Scaling Questionnaire (ASQ) or Childhood Autism Rating Scale (CARS)); we were never informed about their validation procedure through the literature.

Given together, the current project provides a reliable and valid translation of ATEC which remained stable via the cross-cultural and cross-linguistic processes. Small changes made in the questionnaire by the expert committee, were introduced to smooth the progress of items comprehension and the association of the responses.

Regarding the psychometric properties of an instrument, the confirmation of its validity in other cultures boosts the validation of the original one. In relation to the construct validity of the Persian version of ATEC, our results showed that ATEC measures were significantly correlated with the data obtained in ADI-R interview. Results from ADI-R have been proven to support a thorough evaluation of core symptoms of autistic disorders listed in diagnostic and statistic manual of mental disorders (DSM–IV-TR)^[3]. Thus the current study indicated that the Persian version of the ATEC had acceptable properties for evaluating autistic symptoms in individuals with ASD. Moreover, these results accord with the findings of Magiati et al^[9], which showed that ATEC is a promising instrument for gathering reliable and valid information on autistic individuals functioning. Furthermore, there are several studies in which ATEC has been used as a tool for measuring severity of ASD and the authors reported that this questionnaire was successfully able to do so^{[9, 20, 24]}.

Despite the popularity of ATEC to evaluate and monitor progress in autistic individuals over time, to date few data on the reliability (internal consistency) of the questionnaire have been published. Rimland and Edelson cite a few data (reliability: 0.94 for the total scores and 0.8-0.9 for subscale scores) on over 1300 online completed ATECs^[16]. Moreover, recently in a cohort investigation, Magiati et al^[9] supplemented the previous limited literature on the value of ATEC and reported a high internal consistency at their two time points of assessment (When the children were aged about 5.5 years and 5-6 years later). Our study also confirms the previous findings by showing high values of internal consistency for ATEC. All subscales of the questionnaire as well as total scores had similar range of high internal consistency (Cronbach's alpha >0.80).

Given the primary role of ATEC which is to measure factors that are expected partially to change over long term period or under treatment conditions, Megiati et al^[9] reported that ATEC total scores obtained in the baseline significantly predicted the extent of improvement after 5-6 years. On the other hand, providing additional value to Megiati findings, current study showed excellent stability (i.e., test-retest reliability) for all the subscales over a short time period.

Nevertheless, there are a few limitations and remained questions on ATEC validation that current study could not address. Factor structure of inventory should be revised and rechecked in future studies. However, a larger sample study is needed to conduct a factor analysis on ATEC. To address the discriminant validity of the questionnaire, it should be used to differentiate ASD from normally developing children or other developmental disorders. Furthermore, future studies could use other standard measures of autism symptom severity along with ATEC to examine other possible association among subscales.

A few studies which used ATEC to examine severity and monitor symptoms in individuals with ASD indicated the value of ATEC in autism research^{[19, 20, 25–27]}.

Conclusion

In conclusion the current study also indicated that the Persian version of the ATEC is a reliable and valid tool for evaluating ASD symptoms in an Iranian sample with ASD. This finding has important implication for developing effective therapeutic programs as well as ongoing longitudinal research projects in ASDs.

Conflict of Interest

None