Online Administration of the ADOS for Research with Adolescents and Adults in Response to the Pandemic

Abstract

This study evaluates an online ADOS-2 Module 4 administration. Adolescents and adults with (n = 24; 7 females) and without (n = 13; 5 females) a history of autism spectrum disorder (ASD) completed the ADOS-2 Module 4 via videoconference. Parents or caregivers completed the Parent/Caregiver Form of the Vineland Adaptive Behavior Scales and the Achenbach Adult Behavior Checklist. The ADOS-2 was reviewed and scored by five trained clinicians and supervised by a senior clinician with established research reliability. The autistic group’s scores differed on ADOS total (Calibrated Severity Score, WPS instrument) and domain scores, KSADS domain scores, and Achenbach T-scores. Inter-rater reliability was “moderate” (κ = .732), and percentage itemwise agreement was r = .69. The online ADOS-2 showed significant convergence with parent-reported assessments of ASD-relevant symptoms and characteristics, suggesting it was a valid assessment. While any online assessments must be used with caution, results suggest that the approach described here could have sufficient validity and reliability to fill the urgent need to assess and evaluate ASD symptomatology, as one component of a thorough clinical evaluation of ASD-related behaviors.

Lay summary

In this exploratory study, we asked whether it was possible to do the ADOS-2 in adolescents and adults in a completely online way. Results showed that expert clinicians agreed on 69% of ADOS-2 items; also, participants with autism had higher scores on all parts of the ADOS-2. The online ADOS-2 scores had strong and significant relationships with parents’ reports of friendship and social skills. While we need more research that tests this method, this way of doing the ADOS-2 online may be useful for clinicians and researchers who have an urgent need to evaluate autism during the pandemic.

The diagnosis of autism spectrum disorder (ASD; American Psychiatric Association, 2013) involves multiple components: assessment of primary and ancillary symptoms; standardized assessments of language and communication, nonverbal cognition, and adaptive skills; a psychiatric assessment of comorbid diagnoses; and a caregiver interview, such as the Autism Diagnostic Interview - Revised (ADI-R; Rutter, Le Couteur, & Lord, 2003). Diagnosis should also include an observation that is tailored to age, expressive language level, and cognitive level. The Autism Diagnostic Observation Schedule – Second Edition (ADOS; Lord et al., 2012) is the gold-standard diagnostic tool used in both research and clinical contexts.

Coronavirus-related restrictions on personal interactions present barriers to research and have increased the need for online data collection methods. There are many concerns about online assessments (Lord, May 29, 2020); however, it is critical to develop online methods for use during current and future pandemics, to provide a valid means of enrolling participants in research studies. There is also a growing need for telehealth diagnosis. Individuals in remote locations, and in places with few expert service providers, experience dramatic delays in diagnosis (e.g., Antezana, Scarpa, Valdespino, Albright, & Richey, 2017). Most individuals with ASD are identified in childhood; the median age at diagnosis was five years, seven months, according to a national study (Shattuck et al., 2009). However, a growing number of people seek diagnosis later in life. Adult diagnosis is a growing clinical need (Howlin & Moss, 2012), but can be costly and time-consuming (Autism Speaks, July, 2014).

This study evaluates online administration of the ADOS-2 Module 4. A prior study investigated remote ADOS-2 Module 4 administration (Schutte et al., 2015), providing a direct comparison of in-person versus remote assessment for 23 adults with ASD. Reliability was excellent, with an average κ of .70; item-wise percent agreement (exact agreements divided by possible agreements) was calculated at 87% overall, and at 100% (Kappa_weighted > .81) for six items (see Table 2 for definitions): A2, A3, A7, B1, D3, and E2). The lowest agreement (Kappa_weighted < .41, M agreement = 62%) was evident on three items: A6, B6, and D2; results suggested that these were items for which there may be practice effects or effects of comfort with the ADOS (A6, Asks for Information, B6, Comment on Other’s Emotions) or items which occur relatively rarely (D2, Hand and finger Mannerisms). Intraclass correlation (ICC) scores were 0.92–0.98 on Communication and Social Interaction domains and 0.70 for the Stereotyped Behaviors and Restricted Interests domain.

Another study compared in-person and videoconference Module 1 administrations for 21 children, including 11 with a prior diagnosis of ASD and 10 with a prior diagnosis of developmental delay (Reese et al., 2013); each participant was observed by two in-person clinicians and two video-conference clinicians, each of whom observed and scored the assessment. For the in-person assessment group, parents administered the ADOS-2 after receiving training from the lead clinician. Reliability was good for each pair of raters on all items except “pointing” for which in-person clinicians had significantly greater agreement (86%) than videoconference clinicians (35%). Both studies required in-person clinical staff support at the client site and technical modifications; further, both studies employed small samples of previously diagnosed individuals. To serve the needs mentioned above, there is an urgent need for fully remote ADOS administration. This study describes an online ADOS-2 Module 4 assessment, implemented without specialized equipment or on-site staff support.

Participants.

Adolescents and young adults (n=37) participated as part of an ongoing study. Participants had a history of ASD or of neurotypical (NT) development and were mainstreamed in school. There were 24 autistic individuals (7 females), and 13 NT individuals (5 females), ranging in age from 14 to 39 years; see Table 1. Most participants were White and non-Latinx, with n=11 identifying with another race or ethnicity, as multi-racial, or preferring not to disclose (n=1 Asian American, n=2 Latinx, n=1 multi-racial, n=9 did not disclose); these variables did not differ across diagnostic groups, all p’s > .38). NT participants had no history of ASD symptoms, and had no first-degree relatives with ASD. Many of the participants in both groups (especially those who had participated in prior studies with us) had completed an ADOS previously (minimum of seven years). Participants were recruited from prior research (n=26), community outreach (n=10), or via postings on social media (n=3). There were no group differences by recruitment type, all p’s>.5, so recruitment source was not considered further.

Table 1.

Participant demographics

	ASD; n = 24	NT; n = 13	χ2 /F	p	d
Age (years)	22.2 (5.4); 14–32	19.9 (7.2); 14–39	0.61	0.43	0.36
Male: Female	17:7	7:5	1.06	0.59
VABS ABC	80 (15); 52 – 113	105 (8); 91–119	20.79	<.001	2.06
CBCL Social /ABCL Friends	38 (12); 20–59	49 (10); 31–60	6.88	.02	0.18
ADOS Social Affect	11.0 (4.2); 4–19	1.5 (1.9); 0 – 5	58.69	<.001	2.91
ADOS RRB	3.5 (1.5); 1–7	0.6 (1.0); 0–3	35.58	<.001	2.27
ADOS Total score	14.4 (4.3); 8–22	2.2 (2.7); 0–7	85.76	<.001	3.40
ADOS Calibrated severity score	7.6 (1.8); 4–10	1.5 (.88); 1–3	129.83	<.001	4.31

Notes: Data presented as Mean (SD); range. VABS ABC = Vineland Adaptive Behavior Scales Parent/Caregiver Form, 3^rd Ed. (Sparrow, Cicchetti, & Saulnier, 2016), Adaptive Behavior Composite standard score with M(SD) = 100(15). CBCL/ABCL = Achenbach Child Behavior Checklist / Adult Behavior Checklist (Achenbach, 2009). with M(SD) = 10(3).

All participants gave informed consent via videoconference, providing an opportunity to ask questions, “test-drive” the videoconference tools, and evaluate camera and internet systems. Participants were mailed a package (labeled “Do not open until visit”) containing the Tuesday book, Creating a Story items, and a pre-paid return mailer. Several components of the ADOS-2 materials were digitized to enable remote presentation (the Cartoons and Describe a Picture images). Procedures were approved by the University of [redacted] Institutional Review Board.

The ADOS-2 was administered and scored by one of five graduate clinicians, naïve to referral information, who completed clinical ADOS-2 training and attained reliability with a licensed research-reliable supervising clinician (I.M.E) who reviewed and scored the ADOS-2 from video recordings. The clinicians used standard ADOS-2 Module 4 activities and materials; Cartoons and Picture Description materials were digitized.

Equipment.

All but one of the participants had access to an internet-connected home computer with a camera. One participant had internet access, but no computer, and we sent an inexpensive tablet and rechargeable keyboard case along with the ADOS-2 materials. A USB-connectable camera and mobile hotspot could be sent to individuals without those systems, accommodating a wide range of computing and internet systems.

Videoconference system.

Our team used paid subscriptions to GoToMeeting or Zoom-Pro videoconference software, which met IRB standards for confidentiality and security. The 2019 version of GoToMeeting, and all versions of Zoom, allowed participants to hide their own webcam (to see only the clinician). GoToMeeting required participants to download a desktop application.

Procedures

ADOS-2 administration.

At the start of the appointment, participants were directed to connect a power source, shut down program notifications, and to open the ADOS-2 mailer. The participant’s camera was positioned such that face, hands and arms were visible, to capture gestures, repetitive hand mannerisms, etc. The examiner performed a gaze evaluation, asking participants to visually fixate the corners of the screen and the examiner’s eyes, nose, and mouth, pointing at each target; the purpose was to provide clear information about how effectively the video captured each individual’s eye gaze. In our experience, some participants’ gaze and gaze changes could be readily ascertained from the videos, but the videoconference interface hindered the ability to assess nuanced shifts in gaze. To more closely mimic a face-to-face interaction, and to reduce potential interference/self-monitoring behavior associated with self-view, participants were instructed to hide their own face, and to set the videoconference to Full Screen mode. Thus, for most of the session (except as noted below), participants were presented with a large image of the examiner’s face on-screen. No instructions were given as to where the participants should look during the ADOS-2.

There were several additional modifications:

1. Cartoons: Story cards were scanned for presentation. The clinician shared their own display such that the images filled the screen, with the clinician’s face as an inset. Cards were presented one at a time; the participant was invited to tell the clinician that they were “ready for the next picture” after they had time to inspect each card. After viewing all cards, the examiner clicked a button to return the display to the original configuration, and participants were instructed to stand and position the camera to display the entire torso and head; this process was implemented within 10–30 seconds. After the task, the camera was returned to the original configuration.

2. Description of a Picture: Picture images were scanned for presentation. The clinician displayed the picture (of the island resort) using screen sharing.

3. Telling a Story From a Book: The clinician held up the book while telling the story. During the participant’s turn, the book remained on the tabletop.

4. Demonstration Task: The clinician tilted the camera to display the tabletop, where she outlined the imaginary toothbrush, etc. Participants were asked to enact the demonstration on their own table as if the sink was “right in front of you.”

5. Creating a Story: The clinician held up each of five objects to ensure that they were visible, and told an example story. Again, the clinician tilted their screen so that the tabletop was visible during their turn.

6. The optional Construction activity was omitted (as it requires physical interaction with the puzzle materials).

7. The Break activity was excluded, for two reasons. First, it requires physical interaction with ADOS materials, which would have been cost-prohibitive to send. Furthermore, in pilot testing, participants turned to a cell phone or focused on their computer, and did not engage with the examiner; as such, the procedure did not yield the observations typically expected from this activity (e.g., whether the participant will engage with the clinician).

The appointment included activities unrelated to this study. The ADOS-2 was always the first activity and lasted 45–60 minutes. A caregiver completed the Vineland Adaptive Behavior Scales (VABS-3; Sparrow et al., 2016) Parent/Caregiver Form, which yields Communication, Socialization and Daily Living scores with M(SD) =100(15). Caregivers also completed the Achenbach Child Behavior Checklist (CBCL) with participants 12–18 years or the Adult Behavior Checklist (ABCL, Achenbach, 2009) for participants age 19 and older; these measures yield comparable “Social” or “Friends” T-scores, M(SD)=50(10). The VABS Socialization domain captures parent-reported functioning in social contexts, including interpersonal relationships, play and leisure activities, and coping skills in social situations; VABS Communication captures comprehension, expression, and writing skills; and VABS Daily Living Skills captures the ability to perform every-day, practical, age-appropriate tasks. The Achenbach Friends/Social T-scores yield caregiver impressions of social functioning. Significant correlations with VABS Socialization, VABS Communication, and Friends/Social scores provided evidence of convergent validity.

Statistical analyses.

ADOS-2 administrations were evaluated for overall and item-wise reliability (Cohen’s κ). Convergent validity with the CBCL Social/ABCL Friends T-scores and the VABS Communication and Socialization domain standard scores was assessed using Pearson correlations. Missing data (ABCL or CBCL, n=1 NT, 7 ASD; VABS, n=1 NT, 5 ASD) for parents who were unable to complete them in time were excluded pairwise. Study data are accessible via the National Database for Autism Research (NDAR).

Technical aspects of online ADOS-2 administration.

The sessions went smoothly. Participants were able to use the videoconference software as directed; one participant required assistance from a parent to turn on their camera, and a second participant required staff assistance by phone to download GoToMeeting software. The videoconference system worked well, with three minor exceptions: Only audio was recorded during one appointment due to clinician error; the system failed partway through one appointment, due to clinician error; and the system froze after 27 minutes for a third appointment, due to a wifi problem; in the latter two cases, the session was restarted with no further difficulties. In general, problems (distributed evenly across groups) were infrequent and easily resolved.

Reliability of ADOS-2 total scores.

Percentage item-wise agreement for the five administering clinicians and the supervising clinician was calculated at r = 0.69, where any difference on any item (0 vs. 1 vs. 2 vs. 8, etc.) was counted as a disagreement. Inter-rater reliability was calculated as κ =.732, considered “moderate” (Cicchetti & Sparrow, 1981).

ADOS-2 item analysis.

The items with the lowest reliability were: Overall quality of rapport (B13; 76% agreement), Speech abnormalities associated with autism (A2; 78%), Asks for information (A6l; 78%), Emotional/emphatic gestures (A10; 84%), Amount of social overtures/maintenance of attention (B10; 84%), Empathy/comments on others’ emotions (B6; 84%), Insight into typical social situations and relationships (B7; 84%), and Responsibility (B8; 84%); see Table 2. In only two cases did a disagreement reflect a two-category jump (e.g., disagreement by more than one point). Disagreements were resolved by consensus discussion.

Table 2.

ADOS-2 Module 4 Reliability

Domain	Item	Percent agreement
Language and Communication	A1. Overall level of non-echoed language	100
	A2. Speech Abnormalities associated with Autism	78
	A3. Immediate Echolalia	100
	A4. Stereotyped/Idiosyncratic Words or Phrases	89
	A5. Offers Information	95
	A6. Asks for Information	78
	A7. Reporting of Events	86
	A8. Conversation	95
	A9. Descriptive, Conventional, Instrumental, Informational Gestures	92
	A10. Emotional/Emphatic Gestures	84
Reciprocal Social Interaction	B1. Unusual Eye Contact	100
	B2. Facial Expressions Directed towards Others	92
	B3. Language Production and Linked Nonverbal Communication	89
	B4. Shared enjoyment in interaction	86
	B5. Communication of Own Affect	86
	B6. Empathy/Comments on Others' Emotions	84
	B7. Insight into typical Social Situations and Relationships	84
	B8. Responsibility	84
	B9. Quality of Social Overtures	86
	B10. Amount of Social Overtures/Maintenance of Attention	84
	B11. Quality of Social Response	89
	B12: Amount of reciprocal social communication	86
	B13. Overall Quality of Rapport	76
Imagination	C1. Imagination / Creativity	92
Stereotyped Behaviors and Restricted Interests	D1. Unusual Sensory Interest	100
	D2. Hand/finger/Other Complex Mannerisms	95
	D3. Self-Injurious Behavior	100
	D4. Excessive Interest or References to Highly Specific/Unusual Topics or Objects	97
	D5. Compulsions or Rituals	100
Other Abnormal Behaviors	E1. Overactivity	100
	E2. Tantrums	100
	E3. Anxiety	92

Note. Algorithm items highlighted in bold.

For eight items, there were zero disagreements: Overall level of non-echoed language (A1); Immediate echolalia (A3); Unusual eye contact (B1); Unusual sensory interest (D1); Self-injurious behavior (D3); Compulsions or rituals (D5); Overactivity (E1); and Tantrums (E2). Although we anticipated low reliability on B1 and D2, given prior studies (Schutte et al., 2015), there were few disagreements (0 and 5% disagreement, respectively). The Eye Contact item was scored by monitoring head and postural shifts, as well as gaze shifts, associated with conversational turn-taking, across multiple activities. While direct eye contact was sometimes difficult to assess in the online modality, participant use of eye contact to engage the examiner, direct their attention, and regulate conversation was readily apparent through other physical cues, enabling clinicians to estimate the typicality or atypicality of eye contact. Scoring of D2 (Hand and Finger Movements) was also more reliable than expected; repetitive movements were often made in visible space, and clinicians could score those movements reliably. Of course, movements made while the hands were in spaces not visible to the camera could be scored by an in-person clinician; this would require a direct comparison of in-person and remote assessments, something that awaits future research.

In general, disagreements reflected the relative sophistication and fluent verbal skills of this young adult population. For example, evaluating the typicality with which participants ask for information can be difficult, when participants generally responded in some manner to social bids. The difficulty was largely unrelated to the online modality; scoring would likely have been similarly difficult in person. Similarly, evaluating the manner in which a participant expresses empathy or comments on someone else’s emotions can be challenging, when such comments are present but subtly atypical (as opposed to a more frank absence of such expressions). The reliability of the “Overall Quality of Rapport” item (B13) was low; coding was always discussed during clinical supervision, and the administering clinician’s score was adopted as the final score.

There were significant group differences in ADOS-2 scores. Participants with ASD received significantly higher scores across all domains (Social Affect, Repetitive and Restricted Behaviors, Total Score, Calibrated Severity Score); see Table 1.

Validity.

We probed associations between ADOS-2 total scores, parent report of VABS-3 Communication and Socialization scores, and ABCL/CBCL Friend/ Social T-scores, as a check of convergent validity. There were significant associations between ADOS-2 and VABS-3 scores: Socialization, r(48) = −.689, p = .001 (see Figure 1); Communication, r(48) = −.683, p < .001; and ADOS-2 with ABCL-Friends or CBCL-Social T-scores, r(48) = −.622, p < .001. Results patterned similarly when considering the ADOS-2 domain scores (Social Affect, Repetitive Behaviors); correlations are presented in Table 3. Individuals with higher ADOS-2 scores were rated as having lower parent-reported social abilities. Results were similar with ADOS-2 Calibrated severity scores. These results suggest that this online ADOS-2 effectively captures meaningful social skills information.

Figure 1.

Correlation of ADOS-2 and Vineland Social domain scores

Table 3.

ADOS-2 Calibrated Severity and Domain scores as associated with Vineland and ABCL scores

	ADOS-Total	ADOS-SA	ADOS-RRB	VABS-Comm	VABS-Social	VABS-DLS	VABS-ABC	Friend/ Social T-score
ADOS-SA	.97** (37)	--	.57** (31)	−.80** (31)	−.73** (31)	−.73** (31)	−.82** (31)	−.57** (28)
ADOS-RRB	.74** (31)	.57** (31)	--	−.40* (31)	−.53** (31)	−.40* (31)	−.52** (31)	−.39* (28)
VABS-Comm	−.77** (31)	−.80** (31)	−.399* (31)	--	.66** (31)	.83** (31)	.86** (31)	.48** (28)
VABS-Social	−.74** (31)	−.73*** (31)	−.53** (31)	.66** (31)	--	.75** (31)	.91** (31)	.48** (28)
VABS-DLS	−.70*** (31)	−.73*** (31)	−.40* (31)	.83*** (31)	.75*** (31)	--	.93** (31)	.34‡ (28)
VABS-ABC	−.81*** (31)	−.82*** (31)	−.52* (31)	.86*** (31)	.91*** (31)	.93*** (31)	--	.46* (28)
Friends/Social T-score	−.57*** (28)	−.57*** (28)	−.39* (28)	.48** (28)	.48** (28)	.34‡ (28)	.46* (28)	--

Note. Correlations are presented as r (n).

Discussion

The present study evaluated a remote ADOS-2 Module 4 assessment approach that needed no on-site clinician, technical support, or special equipment. While this study responds to the urgent need of a global pandemic, it is also appropriate for independent adults who cannot otherwise access diagnostic services. Results suggest that, with several low-cost modifications, this approach is an effective means of assessing ASD-related behaviors using the ADOS-2. The current results are preliminary, and do not support the use of this modified online assessment as a replacement for an in-person assessment. For both clinical and research purposes, the diagnostic classification made by the remote ADOS-2 Module 4 should be interpreted within the context of additional data about ASD-related behaviors.

Participants were able to meet the technical demands, possibly reflecting increased general familiarity with videoconference platforms during the pandemic. There are many videoconference platforms (e.g., Microsoft Teams, Doxy.me, Zoom); we encourage users to evaluate and routinely check the security and confidentiality of any system.

The current study utilized only Module 4; if applied to Module 3, more substantial modifications would be required. Inter-rater reliability was substantial; scores were as or more reliable as those reported in prior online ADOS studies (e.g., Reese et al, 2013). Even items that were of low reliability in prior research were highly reliable. We suspect that high reliability in the current study primarily reflects the inclusion here of a young adult group of participants with age-appropriate cognitive abilities, meaning that assessment depends more on language and conversational factors, which can often yield higher agreement, as well as access to videoconference recordings for scoring. We also note that “Overall Quality of Rapport” (Item B13) had the lowest reliability out of all codes in this study. This could reflect the challenges of the videoconference format, which can interfere with conversational turn-taking, and might decrease rapport; in this context, coders need to determine whether to attribute apparent reductions in rapport to the videoconference process, or to the interpersonal social skills of the participant. In the current study, the clinician examiner’s scores were taken as final.

Analyses supported the validity of the online ADOS. There were large and significant differences between scores for individuals with versus without ASD, and correlations between ADOS-2 scores and measures of social and relationship functioning, including Vineland Communication and Socialization, and the ABCL Friends score. The convergence of results with external measures, together with the substantial reliability of scores, suggest that this method is effective for ascertaining ASD behaviors in a verbal adult population.

Limitations.

The current exploratory study did not include an in-person assessment for comparison purposes, given the pandemic; of course, such direct comparisons will be critical for establishing the validity of the online approach, and the current findings must be considered preliminary. We only evaluated Module 4, designed for use with independent verbal adolescents and adults, and heavily reliant on conversational probes; online administration of other modules, which rely more on hands-on activities, is not supported by this study. This study excluded the Break activity from the online assessment; this activity provides important opportunities to observe the examinee’s response to the examiner’s withdrawal from the interaction, and how they engage in unstructured conversation. If feasible, it would be preferable to include the Break, and its absence in this study limits the convergence of the online procedure with more standard in-person administrations. Online administration of the Toddler Module, and Modules 1 and 2, which require extensive object-related interactions, will require additional modifications. Individuals with less technical familiarity might require further assistance from a family member or by research staff, over the phone. Furthermore, there were several missing data points in the correlational analyses, due to failure to complete the ABCL/CBCL or Vineland-3 due to challenges due to the pandemic, mental health concerns, or other difficulties. While correlational analyses demonstrated significant relationships, these should be interpreted within the context of missing data that somewhat limit the sensitivity of the validity analyses.

Conclusion:

Results support the reliability and validity of an on-line ADOS-2 Module 4 administration as one component of a comprehensive diagnostic process, offering an avenue for ascertaining diagnostic classifications of research participants during a pandemic, when in-person administration is impossible. Dr. Lord, developer of the ADOS, noted that online administration could be used as part of a diagnostic evaluation, but discouraged its use as the only index of inclusion in research, or for the provision of a complex diagnosis for clinical purposes in adult populations (Lord, May 29, 2020). We agree that caution is warranted, and that replication with direct comparison of remote and in-person assessment is needed. For both clinical and research purposes, this online ADOS-2 Module 4 assessment should be utilized as one component of a thorough clinical evaluation of ASD-related behaviors. With these limitations, results suggest that the approach described here has sufficient validity and reliability to fill current urgent needs.