Usability and Reliability of a Remotely Administered Adult Autism Assessment, the Autism Diagnostic Observation Schedule (ADOS) Module 4

Abstract

Introduction: The Autism Diagnostic Observation Schedule (ADOS) Module 4 is an autism assessment designed for verbally fluent adolescents and adults. Because of a shortage of available clinical expertise, it can be difficult for adults to receive a proper autism spectrum disorder (ASD) diagnostic assessment. A potential option to address this shortage is remote assessment. The objective of this study was to examine the feasibility, usability, and reliability of administering the ADOS Module 4 remotely using the Versatile and Integrated System for Telerehabilitation (VISYTER). Materials and Methods: VISYTER consists of computer stations at the client site and clinician site for video communication and a Web portal for managing and coordinating the assessment process. Twenty-three adults with an ASD diagnosis participated in a within-subject crossover design study in which both a remote ADOS and a face-to-face ADOS were administered. After completing the remote ADOS, participants completed a satisfaction survey. Results: Participant satisfaction with the remote ADOS delivery system was high. The kappa value was greater than 0.61 on 21 of 31 ADOS items. There was substantial agreement on ADOS classification (i.e., diagnosis) between assessments delivered face-to-face versus assessments delivered remotely (interclass coefficient=0.92). Non-agreement may have been due to outside factors or practice effect despite a washout period. Conclusions: The results of this study demonstrate that an autism assessment designed to be delivered face to face can be administered remotely using an integrated Web-based system with high levels of usability and reliability.

Introduction

Autism spectrum disorder (ASD) is a developmental disorder characterized by deficits in social communication and social interaction and restrictive repetitive and stereotyped patterns of behavior, interests, and activities.¹ For 2008, the Autism and Developmental Disabilities Monitoring Network estimated the prevalence of ASD at 11.3 per 1,000 (1:88) children 8 years of age; this is an increase in prevalence from previous reports.² Symptoms of ASD emerge in infancy or childhood, and although intensive therapy may decrease severity, there is no cure; it is a lifelong disability.

An ASD diagnosis is made on the basis of behavioral observations combined with evidence from a detailed developmental history.³ Identifying ASD in adults can be challenging if a diagnosis has not been made in childhood because information regarding developmental history is often unavailable. Caregiver reports may be flawed because of incorrect memory recall, recall bias, and distortion of events. Furthermore, responses can be influenced by a variety of reliability-reducing factors, including alertness in recognizing behaviors, socioeconomic status, personality, intelligence, and mental health.³ It is possible that adults with ASD are currently undiagnosed (especially high-functioning adults) or misdiagnosed (with emotional or psychiatric disorders).^4–6 Accurately diagnosing ASD in adults is important for a variety of reasons, such as implications for treatment, public policy (planning for needs and development of services), and granting access to resources for qualified recipients.

Although there are many ASD assessment and diagnostic tools, few are designed for use with adolescents and adults. The Autism Diagnostic Observation Schedule (ADOS), considered an essential part of the “gold standard” diagnosis,^2,7–10 does have a module (Module 4) specifically designed for verbally fluent adolescents and adults. The ADOS is a semistructured, standardized assessment of communication, social interaction, and play or imaginative use of materials.¹¹

Over the past 30 years, technologists and clinicians have investigated the use of advanced telecommunications and information technologies as a way of bridging the gap between individuals with specialized medical needs living in remote areas and the source of specialty care that is often distal.^12–15 The ADOS is a critical part of ASD assessment in adults, but there is a lack of available clinical expertise to meet need,¹⁶ especially in poorly served areas. A possible solution is the use of technology for remote assessment. Boisvert et al.¹⁶ found through a systematic review that telepractice has potential to be a viable means to address the need for improved access to services for individuals with ASD.

An ADOS Module 4 remote assessment system was developed, and its technical and procedural usability was assessed. The system integrates videoconferencing, presentation of stimuli, scoring, data storage, report generation, and report sharing into an integrated and intuitive Web portal environment.¹⁷ The objective of this study was to determine the feasibility of remotely administering the ADOS Module 4, including usability from a client perspective, and reliability compared with face-to-face administration. Researchers hypothesized that participants would find the remote administration system to be an acceptable way to receive diagnostic services and that standard (face-to-face) administration and remote administration scores would have a high degree of agreement.

Materials and Methods

Instrumentation

ADOS Module 4

The ADOS is a semistructured, standardized assessment of communication, social interaction, and play or imaginative use of materials. The ADOS consists of standard activities that allow the examiner to observe behaviors that have been identified as important to the diagnosis of ASD at different developmental levels and chronological ages. Structured activities and materials provide standard contexts in which social interactions, communication, and other behaviors relevant to ASD are observed.¹¹

The ADOS consists of four modules. Module 4 was designed for adolescents and adults who are verbally fluent (i.e., producing a range of flexible sentence types, providing language beyond the immediate context, and describing logical connections within a sentence). The Module 4 activities focus on social, communicative, and language behaviors important in the diagnosis of ASD. The ADOS Module 4 takes 45 min to 1 h to administer. Notes are taken during administration, and overall ratings are completed immediately after administration. These ratings can then be used to formulate a diagnostic classification through the use of a diagnostic algorithm.¹¹

Versatile and Integrated System for Telerehabilitation

The Rehabilitation Engineering Research Center on Telerehabilitation developed the Versatile and Integrated System for Telerehabilitation (VISYTER). VISYTER is “a platform for building TR [telerehabilitation] applications that takes into account the diverse settings and requirements of various rehabilitation services.”¹⁸ The architecture of VISYTER combines three unique concepts to deliver telerehabilitation: a software application that can be installed easily, a set of off-the-shelf hardware to minimize cost, and a secure server system as the backbone of the service.

VISYTER was designed to comply with standards for privacy and security of personal information (i.e., the Health Insurance Portability and Accountability Act) and uses industry standard security policies, including an authentication system for all users, which also controls the user's access to specific clinic “rooms” or venues, and encryption of all user authentication and the communications between the sites (video and voice) using a symmetric encryption key.¹⁸

At ADOS administrator and client locations, VISYTER runs on desktop computers. The two computer stations are connected by a broadband Internet connection, with a minimum connection of 3 megabits per s/768 kilobits per s (downstream/upstream). This connection is expected to provide optimal performance at an affordable cost. Figure 1 illustrates this connectivity, with one desktop computer station, located at the clinician site (e.g., University of Pittsburgh), one desktop computer station, located at the client site (e.g., rural clinic), and, a server for managing and coordinating all elements of the assessment process.

Fig. 1.

Components of an integrated Autism Diagnostic Observation Schedule (ADOS) Module 4 tele-assessment system.

For this study, the VISYTER application was specifically calibrated to deliver the ADOS remotely (i.e., VISYTER was configured to replicate standard face-to-face administration). The remote ADOS delivery system features videoconferencing, simulated eye contact, layout control, stimuli presentation, electronic scoring system, and session recording/archiving¹⁷:

• • Videoconferencing. VISYTER can handle high-quality full-screen video at 30 frames/s. Based on the speed of the Internet connection, the speed and quality of the video can be adjusted.¹⁸ The videoconferencing feature has the following capabilities:

  • 1. Low-latency (i.e., minimal time delay) and high-resolution audio and video.

  • 2. Two cameras on the client's side. The first camera is a static head-on view that provides face-to-face interaction and simulates eye contact. The second (observational) camera captures the client's hand and finger and other complex mannerisms, gestures, and use of presented objects and materials.

  • 3. Remote camera control of the observational camera. The remote camera control uses the pan-tilt-zoom protocol, allowing the clinician to control the view of the client by panning right and left, tilting up and down, and zooming in and out.¹⁸

  • 4. Image capture, to take snapshots to be included in clinical reports.¹⁷

• Figures 2 and 3 show a view of the videoconference from the clinician's and client's perspectives.

• • Simulated eye contact. Evaluating the quality and quantity of eye contact is an essential part of the ADOS administration. To achieve eye gaze perception at the client side, an inexpensive teleprompter was used on the clinician's site¹⁸ (Fig. 2).

• • Layout control. To minimize distraction and increase comfort on the client side, the display needed to be as simple and clutter-free as possible. However, the clinician needs access to both client video views and the ADOS protocol. A layout control system was developed that allows the clinician to control the screen layout on the client and clinician station (local and remote layout control).¹⁷

• • Stimuli presentation. The ADOS has several visual stimuli that need to be seen by both the administrator and the client. For example, one of the first tasks involves asking a client to tell a story from a picture book. To make the remote administration as close to face-to-face as possible, the visual stimuli are placed below the monitor on a tablet. The administrator has the ability to upload stimuli from the clinician's station and present it on the client's station. Just like with in-person administrations, both the clinician and the client have the ability to “turn the page” (i.e., the client can move forward or backward by pressing large buttons on either side of the tablet) (in Fig. 3, a book is displayed on the tablet).

• • Electronic scoring system. A scoring system was developed that is as close to in-person assessments as possible, except paperless. A Web-based system that is integrated into VISYTER houses the ADOS protocol. During the assessment, the clinician can type notes directly into the ADOS protocol. After the assessment ends, a Web-based version of the scoring form can be accessed, and the clinician assigns scores for each of the 31 evaluation items. The system automatically calculates the final scores using the algorithms prescribed in the ADOS protocol¹⁷ (in Fig. 2, the electronic scoring system is displayed on the right side of the clinician's screen).

• • Session recording/archiving. A secured session archive database was developed to allow clinicians to record entire ADOS administrations in a secure archive database server.¹⁷ Clinicians can use the archived sessions for clinical, educational, and research purposes.

Fig. 2.

Remote Autism Diagnostic Observation Schedule administration from the clinician's perspective.

Fig. 3.

Remote Autism Diagnostic Observation Schedule administration from the client's perspective.

Participants

Participants were recruited from a state-operated vocational training facility for individuals with disabilities. Criteria for inclusion were as follows: a diagnosis of autistic disorder, Asperger's disorder, or pervasive developmental disorder; between the ages of 18 and 40 years; native English speaker; verbally fluent; and Full Scale IQ of ≥70. Criteria for exclusion were as follows: participation in an ADOS administration within the previous 90 days; being unavailable for follow-up (e.g., would graduate and leave the vocational training facility before the second ADOS could be administered); current participation in a social skills group; hearing impairment; visual impairment (other than mild visual difficulties corrected with glasses and/or contacts); nonambulatory; motor problems more severe than very mild cerebral palsy; and having an identifiable syndrome (e.g., Down's syndrome).

Sampling Procedures

Counselors at the vocational facility read a recruitment script to students with an ASD diagnosis. If the student was interested in participating, he or she provided contact information. The primary research coordinator then contacted interested students, read them a screening script, and received verbal consent to be screened. Individuals who consented were then screened. Individuals who after screening were found to meet the inclusion/exclusion criteria then provided consent to participant in the study by reading, discussing, and signing an informed consent form.

Research Design

Institutional Review Board approval was obtained through the University of Pittsburgh (IRB# PR010100117).

The research design was a within-subjects crossover study. A crossover design was used in previous studies involving remote psychometric assessments.^19,20 Benefits of the crossover design include removing between-patient variation and requiring fewer participants.

Participants were randomly assigned to one of two experimental groups: one-half of the participants were initially tested face to face, followed by a remotely administered test; the other half were tested in the opposite order. The two assessments were a minimum of 90 days apart. According to Dr. Lord, author of the ADOS, a 90-day break effectively serves as a washout period to reduce the learning effects on the performance in the second ADOS administration (C. Lord, personal communication, January 4, 2010).

During the remote assessment, the participant was not alone in the evaluation room. A technician was on-site to facilitate administration.

After administering the ADOS activities, the clinician coded the participants' behavior throughout the entire evaluation on 31 items. Possible codes were 0, 1, and 2. A score of 0 indicates no evidence of abnormality related to autism; 1, mildly abnormal or slightly unusual behavior; and 2, definite or markedly abnormal behavior. The ratings are organized according to five main groupings: “A. Language and Communication,” “B. Reciprocal Social Interaction,” “C. Imagination,” “D. Stereotyped Behaviors and Restricted Interests,” and “E. Other Abnormal Behaviors.”¹¹

All assessments were administered and scored by the same ADOS Module 4 research reliable clinician (J.L.S.). Several steps were taken to ensure the clinician's second scoring was not influenced by the first interaction. First, the washout period of 90 days decreased the likelihood of remembering the first administration. Second, all assessments were recorded using VISYTER, and the video recordings of six randomly selected second administrations were scored by outside research reliable clinicians. The reliability of the original clinician as established by the outside clinician was satisfactory: reliability on all items ranged from 81% to 87%, with a mean of 84.5%; reliability on algorithm items ranged from 75% to 88%, with a mean of 82%.

Upon completion of the remotely administered ADOS, participants completed a seven-item Post-ADOS User Satisfaction Questionnaire, consisting of the following statements: (1) I felt comfortable doing this assessment using the computer, (2) The quality and clarity of the video (picture) was acceptable, (3) The quality and clarity of the audio (sound) was acceptable, (4) Being assessed this way provides a true picture of how I typically behave and interact with others, (5) There were things I was unable to do/say because of the computer system that I would have been able to do/say in person, (6) If I had to have assessments or tests in the future, I would be willing to do them over the computer, and (7) If this was your second administration, which administration did you prefer? For statements 1–6, participants responded on a Likert scale from 1 (strongly agree) to 7 (strongly disagree). For the last question, participants have the following choices: greatly prefer face-to-face, slightly prefer face-to-face, no preference, slightly prefer remote system, greatly prefer remote system, or NA—this was my first administration. Participants were also able to write in comments.

Participants were reimbursed for their time in the amount of $10 after completing the first ADOS and $20 after completing the second ADOS.

Statistical Analysis

IBM (Armonk, NY) SPSS Statistics software version 20.0 was used for data analysis.

Sample characteristics

Sample characteristics were determined using frequencies for categorical variables and means and standard deviations (SDs) for continuously measured demographic variables.

Usability

The Post-ADOS User Satisfaction Questionnaire was analyzed using descriptive statistics including mean, median, mode, and frequency distribution.

Reliability of individual items

Statistical procedures used to calculate reliability were derived from the procedures used when reliability for the ADOS was initially established. Lord et al.¹¹ originally used a standard formula for weighted kappa values (ĸ_w). The kappa statistic (ĸ) is a chance-corrected measure of agreement that is the standard measure for psychometric reliability.²¹

Agreement (P_o, the number of exact agreements divided by the number of possible agreements) was also calculated.

Reliability of domain scores and classification

As in the original reliability report, intraclass correlation coefficients (ICCs) were computed across pairs of administrations for domain scores (algorithm subtotals) and classification (algorithm totals).

Alternate form

To determine if participants scored differently when the ADOS was administered remotely than when it was administered face to face, the Wilcoxon Signed-Rank test was used. As the data were skewed (not normally distributed), this was the most appropriate statistical test.

Results

Participant Characteristics

In total, 46 individuals with an ASD diagnosis were interested in participating after being read the recruitment script. After screening for inclusion/exclusion criteria, 26 participants were enrolled in the study. Three dropped out because they were unavailable for the second ADOS (n=23).

Participants were between the ages of 19 and 30 years of age (mean=21.96, SD=2.88). Seventy percent of participants were male (n=16); 30% were female (n=7). Participant Full Scale IQs ranged from 70 to 110 (mean=88.96, SD=10.09).

Usability

Participants responded to six statements on the Post-ADOS Assessment User Satisfaction Questionnaire on a Likert scale, from 1 (strongly agree) to 7 (strongly disagree). The mode was 1 for items 1–4 and 6, indicating most participants “strongly agree” with these statements, which include “I felt comfortable doing this assessment using the computer.” The mode was 5 for item 5, indicating most participants “disagree” with the statement “There were things I was unable to do/say because of the computer system that I would have been able to do/say in person” (Table 1).

Table 1.

Post–Autism Diagnostic Observation Schedule Assessment User Satisfaction Questionnaire Results

ITEM	MEAN	MEDIAN	MODE
1. Felt comfortable using computer	2.48	2.00	1
2. Quality of video was acceptable	1.96	1.00	1
3. Quality of audio was acceptable	2.00	1.00	1
4. True picture of how I typically behave	2.70	2.00	1
5. There were things I was unable to do/say	4.39	5.00	5
6. Willing to do assessments over the computer in future	2.26	1.00	1

Lower numbers indicate greater satisfaction with the system, except on item 5.

The 14 participants who had the remote ADOS administered second were also asked which administration they preferred, face to face or remote. Two participants indicated they “slightly prefer” the remote system. Seven participants had “no preference.” Five participants “slightly prefer” or “greatly prefer” the face-to-face administration.

Reliability of Individual Items

Kappa was calculated for all 31 items: ĸ>0.81 (almost perfect agreement) on 10 items, ĸ=0.61–0.81 (substantial agreement) on 11 items, and ĸ=0.40–0.61 (adequate agreement) on two items. On only three items was ĸ≤0.41.

On five items, at least one of the variables in each two-way table upon which measures of association are computed was constant, so ĸ could not be calculated. P_o was 100% for four of these items; one item had 87% agreement (Table 2).

Table 2.

Autism Diagnostic Observation Schedule Module 4: Percentage Agreement with Kappa Values Indicating Reliability

TOPIC	SUMMARY RATING	PERCENTAGE AGREEMENT	WEIGHTED KAPPA
Language and Communication	A1. Overall Level of Non-Echoed Language	96	0.646
	A2. Speech Abnormalities Associated with Autism	100	1.000
	A3. Immediate Echolalia	100	—
	A4. Stereotyped/Idiosyncratic Use of Words or Phrases	91	0.880
	A5. Offers Information	96	0.777
	A6. Asks for Information	65	0.289
	A7. Reporting of Events	100	—
	A8. Conversation	91	0.777
	A9. Descriptive, Conventional, Instrumental, or Informational Gestures	87	0.642
	A10. Emphatic or Emotional Gestures	87	0.763
Reciprocal Social Interaction	B1. Unusual Eye Contact	100	1.000
	B2. Facial Expressions Directed to Others	78	0.642
	B3. Language Production and Linked Nonverbal Communication	87	0.839
	B4. Shared Enjoyment in Interaction	87	0.697
	B5. Communication of Own Affect	83	0.731
	B6. Empathy/Comments on Others' Emotions	43	0.230
	B7. Insight	87	0.806
	B8. Responsibility	91	0.852
	B9. Quality of Social Overtures	91	0.851
	B10. Quality of Social Response	91	0.846
	B11. Amount of Reciprocal Social Communication	96	0.862
	B12. Overall Quality of Rapport	91	0.826
Imagination	C1. Imagination/Creativity	78	0.566
Stereotyped Behaviors and Restricted Interests	D1. Unusual Sensory Interest in Play Material/Person	91	0.617
	D2. Hand and Finger and Other Complex Mannerisms	78	0.138
	D3. Self-Injurious Behavior	100	—
	D4. Excessive Interest in or References to Unusual or Highly Specific Topics or Objects or Repetitive Behaviors	87	0.517
	D5. Compulsions or Rituals	87	—
Other Abnormal Behaviors	E1. Overactivity/Agitation	96	0.646
	E2. Tantrums, Aggression, Negative or Disruptive Behavior	100	—
	E3. Anxiety	96	0.646

Reliability of Domain Scores and Classification

The ICC was 0.92–0.98 (good) on the “Communication,” “Social Interaction,” and “Communication+Social Interaction Total” scores. The ICC was 0.70 (moderate) on the “Stereotyped Behaviors and Restricted Interest” score. In Table 3, the current test–retest reliability (remote–face-to-face or face-to-face–remote) is detailed along with the test–retest reliability data published in the ADOS manual.

Table 3.

Autism Diagnostic Observation Schedule Module 4: Intraclass Correlations for Test–Retest Reliability

	N	COMMUNICATION	SOCIAL INTERACTION	COMMUNICATION+SOCIAL INTERACTION TOTAL	STEREOTYPED BEHAVIORS AND RESTRICTED INTERESTS
Original ADOS data: test–retesta	27	0.73	0.78	0.82	0.59
Face-to-face versus remote administration	23	0.92	0.98	0.98	0.70

Adapted from Lord et al.,¹¹ p. 115.

^aIntraclass correlations for data pooled across Modules 1–4.

ADOS, Autism Diagnostic Observation Schedule.

The ADOS includes a diagnostic algorithm that allows for the classification of participants into one of three categories: Autism, Autism Spectrum, or Non-Spectrum.¹¹ The ICC was 0.92 on ADOS classification between assessments delivered face-to-face versus assessments delivered remotely.

Alternate Form

On 30 of 31 items, there was no significant difference when comparing face-to-face administration scores with remote administration scores. On one item, “Asks for Information” (a non-algorithm item), participants asked significantly more questions when the ADOS was administered remotely (Z=−2.11, p=0.035); the effect size was medium (r=−0.31).

Discussion

There has been an increase in the prevalence of ASDs and therefore an increase in the need of specialists to assess, diagnose, and treat individuals with ASD, including adolescents and adults. Tele-assessment—the remote administration of systematic procedures for observing and describing behaviors through use of interactive videoconferencing between a client at a local site and a remotely located assessment expert—is an exciting opportunity to ensure that adolescents and adults are able to access gold standard autism assessment services, including the ADOS Module 4.

Results of this study indicate participants had a high degree of satisfaction with the remote administration system. Seventy-five percent of participants indicated they felt comfortable using the system. Seventy-eight percent of participants indicated that being assessed this way provides a true picture of how they typically behave and interact with others. Eighty-three percent of participants indicated that they would be willing to do future tests over the computer. The quality of the video was rated very highly. In addition, this population—adults with ASDs—may be a good fit for telerehabilitation interventions. Individuals with ASD may be comfortable with and open to remote assessment and treatment. For example, Savin et al.^22,p.486 reported a case study in which a 13-year-old male “was able to express himself better during telepsychiatric consultation than he had during his previous face-to-face consultations.”

In addition, the reliability results generally supported the ability to conduct the ADOS via a telemedicine platform. Specifically, item B1, which rates unusual eye contact, had perfect agreement between remote and face-to-face assessments. Facilitating eye contact remotely was a challenge researchers faced and solved with use of a teleprompter. Success was likely due to the fact that ADOS administrators are not evaluating the accuracy and directness of eye contact, but rather how gaze is being used with other communication to initiate, terminate, or regulate social interaction (e.g., does the person “check in” with the evaluator when describing a picture?).

There was low agreement on three out of 31 ADOS items (ĸ≤0.41): “Asks for Information,” “Empathy/Comment on Others' Emotions,” and “Hand and Finger and Other Complex Mannerisms.” There was adequate agreement on two items (ĸ=0.41–0.61): “Imagination/Creativity” and “Excessive Interest in or References to Unusual or Highly Specific Topics or Objects or Repetitive Behaviors.” There are several possible reasons for the low agreement. Several items are highly sensitive, and the participant labeling one additional emotion or asking one fewer question changes the score from a 0 to a 1 or a 1 to a 2. The test–retest study design allows for changes in performance from one ADOS administration to the next. Some participants did behave differently on the two ADOS administration days, likely because of outside factors (e.g., Having a good or bad day?, Looking forward to going home for the weekend?, Just received a bad grade on a test?, etc.).

In addition, despite the washout period, there may have a practice effect on some items. For example, the Imagination/Creativity item weighs heavily on an activity that involves creating a story out of five random items. This is an item that might have had a practice effect, despite the washout period. At least one participant mentioned during the second administration that he had been waiting for the creating a story activity and had been thinking about what he might do with his five items. However, a Wilcoxon Signed-Rank tests showed that there was not a practice effect on this item (Z=−1.342, p=0.180). The only item on which there was a statistically significant difference between first and second administrations was again “Asks for Information,” where participants asked significantly more questions on the second ADOS versus the first ADOS (Z=−2.11, p=0.035, r=−0.31). The median “Asks for Information” rating was 2 for both first and second administrations, indicating the difference may not be clinically significant.

There was also limited agreement on two of the items from the Stereotyped Behaviors and Restricted Interests domain. For these items, the examiner is required to note behaviors that are often of low incidence for this population (e.g., a very brief or rare hand and finger mannerisms or complex mannerisms, or occasional references to unusual or highly specific topics or patterns of interest). Because these are often rare occurrences, it is very possible that the administrator would not observe the behavior during a 1-h-long assessment, but may observe and code it in another assessment.

Limitations

Limitations associated with methodology

The Post-ADOS User Satisfaction Questionnaire was short (seven items) and was only completed after the remotely administered ADOS. The researchers noted that participants were often eager to return to lunch, class, etc., and occasionally seemed to rush through the questionnaire. Few participants left written comments. A longer, more detailed questionnaire or structured interview and questionnaires administered after both the in-person and remote ADOS administrations might have led to more comprehensive feedback about participants' satisfaction regarding remote assessment.

The participants in this study were a homogeneous group; they were all students at a vocational training school, living independently on-campus and taking classes full-time. Therefore, this was a relatively high-functioning group that was more likely to fall into the non-spectrum or autism spectrum classification on the ADOS, as opposed to the autism classification. Having a sample that was more evenly spread across the autism spectrum would lead to more generalizable results. However, the population of adults that is still seeking diagnosis in adulthood is likely to be more high-functioning and have less severe symptoms of autism. In this way, this population was representative of the population who might come into a clinic seeking diagnostic clarity.

Limitations associated with remote administration

It is necessary for the remote ADOS administrator to have a level of comfort and skill with technology to administer the ADOS remotely. As with all technology, with the remote ADOS administration system, there was an occasional need for troubleshooting (e.g., VISYTER settings need adjusted, computer Internet connection secured, video and sound settings manipulated). ADOS administrators currently take steps to receive training and maintain reliability in their specific area of practice, and individuals conducting tele-assessment should perhaps also receive training in factors specific to information technology.

There were differences between face-to-face and remote ADOS administration that may not have been captured by the item and algorithm scores. For example, when planning to administer the ADOS remotely, one optional ADOS activity was excluded from this study because it required the participant to physically hand a puzzle piece to the examiner, and this was impossible to translate to a remote administration. Although this activity was optional and is often excluded when the ADOS is administered face to face, any information that might be gained from this activity is impossible to gather when the assessment is done remotely.

Also, some aspects of social interaction and rapport were difficult or impossible to assess using tele-assessment. Occasionally the ADOS administrator had observations face-to-face that were impossible to make remotely. For example, the administrator noticed a participant's strong odor during a face-to-face ADOS that affected the quality of the rapport. A potential solution to this problem is to include the on-site technician's observations when scoring the ADOS.

Along the same lines, because of the remote administration, another person was brought into the ADOS administration: the on-site technician. Although typically the on-site technician set up the computer and then had no involvement in the ADOS administration, occasionally the participant would engage the on-site technician in conversation during the ADOS. An on-site technician is a requirement in tele-assessment because it is unethical to leave a client alone in a room in case of emergency, but it did alter the standard ADOS administration.

Future Directions and Conclusions

The results of this study demonstrate that an autism assessment designed to be delivered face to face can be administered remotely using an integrated Web-based system and will demonstrate high levels of usability and acceptance by clients. In addition, pilot reliability results indicate feasibility.

There has been limited research on the validity and reliability of assessment instruments via tele-assessment. Given the shortage of specialists and the increasing prevalence of ASD, there is a clear need for further research on remote services for this population. A logical next step for this research is a more comprehensive reliability study and a validity study that estimates the diagnostic accuracy of a remotely administered ADOS Module 4 with adult participants who are diagnostically representative of those who might seek services from an adult autism outpatient clinic.

Another step to developing reliability is to investigate the potential to conduct ADOS Modules 1–3 via tele-assessment. There are additional challenges presented in administering these modules remotely, primarily due to the joint interactive play requirements. A recently published article by Reese et al.¹⁴ addresses this problem by having, in the interactive videoconferencing condition, the ADOS administrator (remotely located) direct a family member (who is in the same room as the client) on ADOS presses. However, there are obvious validity concerns when the administration of the ADOS is modified in this way.

Acknowledgments

This research was supported by grant H133E090002 from the National Institute on Disability and Rehabilitation Research Rehabilitation Engineering Research Center on Telerehabilitation, a grant from the Pennsylvania Department of Public Welfare and Bureau of Autism Services Western Pennsylvania Autism Service, Education, Research, and Training Regional Center, and by grants UL1 RR024153 and UL1TR000005 from the National Institutes of Health. The authors would also like to gratefully acknowledge the cooperation and support of the Pennsylvania Department of Labor and Industry Office of Vocational Rehabilitation's Hiram G. Andrews Center and the Inter Governmental Agency Agreement between the Hiram G. Andrews Center and the Department of Rehabilitation Science and Technology of the University of Pittsburgh.

Disclosure Statement

No competing financial interests exist.