Evaluating Interactive Videoconferencing for Assessing Symptoms of Autism

Abstract

Background: Autism affects as many as 1 in 88 children. Best practices recommend early identification and intervention for optimal outcomes. Currently, a gap exists between time of first concern and diagnosis, particularly for families living in rural areas. Telemedicine as a tool for assessment and diagnosis of autism is one way to address this disparity. Emerging evidence suggests telemedicine as a viable option for assessing children with a variety of special needs. Materials and Methods: This study expands upon the current literature by investigating clinicians' ability to assess autism via telemedicine. Using interactive videoconferencing, we simulated autism assessment procedures with families with an existing diagnosis (autism or developmental disability) using current gold-standard assessment tools. We compared diagnostic accuracy, item-by-item reliability on the Autism Diagnostic Observation Schedule (ADOS)—Module 1, and the Autism Diagnostic Interview—Revised (ADI-R) as well as parent satisfaction in an in-person and interactive videoconferencing condition. Ten children (3–5 years old) with developmental delays and 11 children matched on chronological age with a diagnosis of autism were assigned to be assessed and interviewed either in-person or over videoconferencing. Clinicians observed both in-person and through videoconferencing regardless of patient assignment. Results: Results indicated no significant difference in reliability of diagnostic accuracy, ADOS observations, ratings for ADI-R parent report of symptoms, and parent satisfaction between conditions. Results indicate adequate clinician agreement and parent satisfaction regardless of observational condition. Conclusions: Future research should include a larger sample size and assess children without an existing diagnosis.

Introduction

The Centers for Disease Control and Prevention describe autism as a “group of developmental disabilities that can cause significant social, communication and behavioral challenges.”¹ Current prevalence data indicates that autism affects as many as 1 in 88 children.² The American Academy of Pediatrics recommends that all children be screened at 9, 18, and 24 or 30 months as part of medical home surveillance.³ Best practices indicate that those failing a screen should promptly be referred for a comprehensive interdisciplinary evaluation. Currently, no biological or blood test exists to identify autism. Rather, autism evaluation includes structured observation of the child, parent interview, evaluation of cognitive, language, and adaptive functioning, and physical examination.⁴ Gold standard assessment tools include a parent interview such as the Autism Diagnostic Interview—Revised (ADI-R)⁵ and a structured observation of the child using validated instruments such as the Autism Diagnostic Observation Schedule (ADOS).^6,7

To date, no “cure” has been identified for autism, but dramatic changes in the course of development are possible with early identification and intervention.^8,9 However, some reports suggest a lag time of approximately 13 months (sometimes longer in rural areas) between initial suspicion of autism and actual diagnosis.^10,11 More recently, Shattuck et al.¹² found the median age of autism identification to be 5.7 years, with more than 25% diagnosed after 8 years of age. Research indicates that children may reliably be diagnosed with autism by 2 years and that intervention is most effective if implementation begins prior to 5 years of age.¹³ Because of autism's growing prevalence and demand for timely diagnosis and treatment, providers must examine alternative options for providing services to address the lag time between first concern and diagnosis.

A critical barrier for many families is access to diagnostic services. A 2007 study found limited access to autism-related services for families in non-metropolitan areas, families of racial and ethnic minorities, and those living in poverty.¹⁴ Interactive videoconferencing (IVC) technology may be one option that aids families in rural areas experiencing lag time for diagnosis.^15,16 Several studies have shown that using IVC technology improves access to pediatric subspecialty care in rural areas^17,18 and reduces travel expenses, work hours lost, and hospital visits.¹⁹ There is growing evidence for the use of IVC to conduct psychiatric assessments with young children.²⁰ More recently, studies have shown IVC to be effective for diagnosis and treatment for psychological and developmental disorders.²¹

This study compared autism assessment practices through IVC with traditional in-person (InP) models. We compared inter-rater agreement on scores for two gold standard autism measures (the ADOS and ADI-R) based on assessments conducted in person and through IVC. We also evaluated accuracy of diagnosis and parent satisfaction between IVC and InP conditions. The main research questions were as follows:

1. Is there a difference in diagnostic impressions regarding autism when observers are evaluating InP versus IVC?

2. Is there a difference in inter-rater agreement of both observed and reported behaviors on the ADOS and ADI-R when observers are watching InP versus over IVC?

3. Is there a difference in parent satisfaction between InP and IVC?

Materials and Methods

Participants

Participants included 11 children with an existing Diagnostic and Statistical Manual of Mental Disorders, 4th edition (DSM-IV) diagnosis of autistic disorder (autism) and 10 with an existing diagnosis of developmental delay. None of the participants had other primary or co-occurring diagnoses. All 21 participants (3–5 years old; 3 girls, 18 boys; 1 African American, 1 biracial, and 19 white) had received a diagnosis at a university medical center in the Midwest within 2 years prior to participating in this study. In order to confirm the accuracy of diagnosis over IVC, it was necessary to recruit children with existing diagnoses. Additionally, conducting a simulation on-site allowed for greater control of conditions.

Measures

ADOS

The ADOS is a semistructured observation tool used in the diagnosis of autism. The ADOS uses a set of planned activities and “presses” to observe quality of communication and social behaviors in a variety of contexts. The ADOS consists of four modules, with materials and activities designed for use with individuals at particular developmental and language levels. The present study used the presses and the coding/scoring systems from the ADOS—Module 1 for all participants. The first author provided instructions to parents on how to administer the ADOS presses, which is a modification of ADOS standardization. Half of the participants received instructions over IVC, and the other half received instructions in an InP setting according to their random assignment. The ADOS manual reports good psychometric properties with high inter-rater agreement.⁶

ADI-R

The ADI-R is a standardized caregiver interview that gathers information to help inform a diagnosis of autism. Interview questions fall into one of three categories corresponding to the DSM-IV-TR diagnostic criteria for autism: language/communication; reciprocal social interactions; and restricted, repetitive, and stereotyped behaviors and interests. Scores from each category are entered into a diagnostic algorithm to determine the likelihood that a person would have a diagnosis of autism. The ADI-R administration manual reports adequate reliability for individual algorithm items and validity coefficients for domain scores.⁵ For the present study, only algorithm items from the ADI-R were administered.

Satisfaction Survey

Parents completed a 7-point Likert scale survey (ranging from 1=strongly disagree to 7=strongly agree) to evaluate satisfaction with diagnostic procedure and IVC experience. All participants completed seven questions about their satisfaction with the overall experience, with participants in the IVC condition completing an additional six questions about the technical aspects of the equipment.

Procedures

Research clinicians and inter-rater agreement

The research team consisted of five clinicians who were aware that participants had received a previous diagnosis of either developmental delay or autism. However, clinicians were blinded to which diagnosis the participant had actually received. All clinicians received training on the ADOS, the ADI-R, and the DSM-IV-TR diagnostic criteria for autism. The first author completed ADOS and ADI-R research training, and the other clinicians received clinical training on the ADOS. Each of the five clinicians had multiple years of experience working on interdisciplinary autism diagnostic teams that used the ADOS, ADI-R, and DSM-IV-TR criteria. Clinicians attained adequate inter-rater agreement on the ADOS prior to working with study participants. Adequate agreement was defined as 80% agreement for all raters in two of three consecutive sessions.

Setting and materials

This study was conducted at a university medical center in a Midwestern metropolitan city. All assessments were conducted in one room connected through IVC to another observation room in the same building. Conducting simulations on-site allowed for greater control than a community-based setting, which is important for this preliminary study. Both rooms included a table and chairs with two research clinicians present in each room. We used high-definition monitors and cameras to provide the highest quality IVC experience.

Study procedures

Figure 1 illustrates room arrangement and clinician location for each condition. Participants were randomly assigned to either the InP administration condition (autism=6; developmental delay=5) or the IVC administration condition (autism=5; developmental delay=5). To minimize variability in test administration, the first author directed all families on ADOS presses and administered the ADI-R to all participants across all conditions. For each participant, a total of four research clinicians (two IVC, two InP; one of whom was always the first author) observed and scored assessments. Caregivers completed a brief satisfaction survey in all conditions.

Fig. 1.

Location of instructor and observer for interactive videoconferencing (IV) and in-person (InP) conditions. C, child; P, parent; R, rater; R₁, first author instructing parent.

As noted above, all four observers independently scored the ADOS and ADI-R items to determine (a) inter-rater agreement on diagnostic impressions (DSM-IV criteria for autism or developmental delay with no autism), (b) inter-rater agreement on the ADOS, and (c) inter-rater agreement on ADI-R items. For each participant, research clinicians were randomly assigned to an InP or IVC condition to create pairs. We compared clinician agreement on the ADOS and the ADI-R items in two ways: (1) comparison of agreement among clinicians who were in the same and different settings and (2) comparison of agreement among clinicians who were in the InP and IVC conditions. Comparing same and different settings allowed us to evaluate agreement across all clinicians. Comparing InP and IVC conditions allowed us to determine the possible impact of conducting assessment measures over IVC. Finally, a parent participant from each case completed a brief satisfaction survey.

Results

For this study, we conducted three sets of analyses: (1) accuracy of clinicians' diagnostic assignment to the child's previous/existing diagnosis and consistency across clinicians, (2) inter-rater agreement on the ADOS and ADI-R, and (3) parent satisfaction with the IVC experience.

Inter-Rater Agreement on Assignment of Participant Diagnosis

At the conclusion of each observation session, all clinicians assigned a diagnosis of autism or developmental delay to the child. We first evaluated the consistency of the clinicians' diagnostic assignments with one another. Next we compared these predicted diagnoses to the diagnosis given to the child during the initial interdisciplinary evaluation.

Clinicians showed high levels of consistency (nearly 100% of the time) with one another when assigning diagnoses. Clinicians disagreed on diagnosis once, when one of the four raters (in the InP condition) disagreed with the other three clinicians. Next we calculated the percentage agreement of clinician assignment of diagnosis with each child's existing diagnosis. Clinicians made diagnostic assignments consistent with prior diagnoses 83% of the time and 86% of the time in each of the InP and IVC conditions, respectively, resulting in no significant differences across the two conditions.

Inter-Rater Agreement on the ADOS and the ADI-R

To determine reliability on the ADOS and the ADI-R, we first compared the average percentage agreement between clinicians in the same setting followed by average percentage agreement between clinicians in different settings. Each clinician independently coded ADOS and ADI-R items. Items were considered in agreement if the codes on individual items were identical. We calculated percentage agreement on each item (ADOS=29 items; ADI-R=32 items) for each subject (n=21) across pairs of clinicians within the same setting (two pairs of clinicians) as well as between different settings (four pairs of clinicians). The purpose of these calculations was to determine if mode of observation (InP or IVC) and/or setting (same setting or different setting) impacted the inter-rater agreement on the assessment tools.

Results of the item-by-item analysis of percentage agreement are presented in Tables 1 (for the ADOS) and 2 (for the ADI-R). In each table, the first two columns represent the mean percentage agreement on each item for the two clinicians who were in the InP and IVC settings, respectively. Each of these first two columns represents same setting percentage agreement measures (one pair InP, one pair IVC). The third column represents the mean percentage agreement across all same setting ratings (e.g., the mean of the first and second columns) for each item. Finally, the fourth column represents the mean percentage agreement between pairs of clinicians who were in different settings (four clinician pairs).

Table 1.

Mean Percentage Agreement on Autism Diagnostic Observation Schedule Items by Condition

	% AGREEMENT
ITEM	IN-PERSON	IVC	SAME SETTING	DIFFERENT SETTING
ADOSA1	76	80	78	78
ADOSA2	57	80	69	73
ADOSA3	67	65	66	67
ADOSA4	67	70	69	73
ADOSA5	71	75	73	78
ADOSA6	86	80	83	83
ADOSA7	86a	35a	61	69
ADOSA8	48	65	57	43
ADOSB1	86	90	88	88
ADOSB2	71	70	71	71
ADOSB3	62	80	71	71
ADOSB4	67	63	65	65
ADOSB5	52	67	60	66
ADOSB6	76	80	78	71
ADOSB7	62	55	59	70
ADOSB8	67	55	61	65
ADOSB9	67	45	56	53
ADOSB10	71	60	66	48
ADOSB11	91	95	93	91
ADOSB12	71	70	71	66
ADOSC1	62	75	69	62
ADOSC2	67	65	66	66
ADOSD1	71	90	81	67
ADOSD2	81	90	86	82
ADOSD3	100	100	100	100
ADOSD4	67	85	76	67
ADOSE1	71	40	56	56
ADOSE2	76	80	78	81
ADOSE3	91	85	88	91

^aDenotes statistically significant difference between conditions.

ADOS, Autism Diagnostic Schedule; IVC, interactive videoconferencing.

Table 2.

Mean Percentage Agreement on Autism Diagnostic Interview—Revised Items by Condition

	% AGREEMENT
ITEM	IN-PERSON	IVC	SAME SETTING	DIFFERENT SETTING
ADI31	85	91	88	77
ADI33	76	86	81	81
ADI34	76	91	84	74
ADI35	81	76	79	74
ADI36	91	81	86	84
ADI37	76	86	81	67
ADI38	91	86	89	86
ADI39	86	86	86	86
ADI42	76	76	76	81
ADI43	100	90	95	95
ADI44	95	95	95	95
ADI45	91	91	91	86
ADI47	86	86	86	86
ADI48	91	95	93	88
ADI49	91	91	91	87
ADI50	86	91	89	91
ADI51	86	91	89	86
ADI52	86	100	93	93
ADI53	95	91	93	93
ADI54	81	91	86	80
ADI55	100	95	98	89
ADI56	76	95	86	81
ADI57	71	91	81	83
ADI58	86	81	84	85
ADI59	76	95	86	72
ADI61	71	81	76	76
ADI62	81	81	81	84
ADI63	86	86	86	84
ADI64	95	90	93	93
ADI67	91	81	86	73
ADI68	91	86	89	89
ADI69	81	91	86	84
ADI70	95	95	95	95
ADI71	67	95	81	76
ADI77	91	95	93	84
ADI78	86	91	89	84

ADI, Autism Diagnostic Interview; IVC, interactive videoconferencing.

ADOS

To determine if significant differences in percentage agreement existed between InP and IVC conditions, we conducted Z-tests for the two proportions/percentages for each item. ADOS Item A7 (related to pointing) was the only item significantly different between InP and IVC (z=2.6, p=0.009). The two raters in the InP condition agreed more frequently (81% of the time) than the two raters within IVC (35% of the time). We also used Z-tests to determine if the percentage agreement when raters were in the same setting was significantly different from the percentage agreement when raters were in different settings. None of these items was significantly different across settings.

For raters in the InP condition, the average percentage agreement across all items on the ADOS was 71.97% (standard deviation [SD]=11.81%). When raters were in the IVC condition, the average percentage agreement across all items on the ADOS was 72.07% (SD=15.96%). These results suggest no significant differences in percentage agreement between InP and IVC. Next, we compared the average percentage agreement across all items when raters were in the same setting (72.24%; SD=11.55%) with the average percentage agreement across all items when raters were in different settings (71.07%; SD=12.74%). Results indicated no significant difference (t₂₈=0.96, p=0.35) between inter-rater agreement both within and across settings.

Finally, we calculated kappa values to examine the quality of the inter-rater agreement on the ADOS. Average kappa for both IVC and InP was 0.50. Average kappa for raters in the same setting was 0.50, and average kappa for raters in different settings was 0.47. Kappa scores were generally low to moderate regardless of condition.²²

ADI-R

For raters in the InP setting, the average percentage agreement across all items on the ADI-R was 85.28% (SD=8.30%). When raters were in the IVC setting, the average percentage agreement across all items on the ADI-R was 88.89% (SD=5.80%). These results suggest no significant differences in percentage agreement between the InP and IVC settings. However, differences on Item 71 (related to sensory interests) were nearly significant with 95% agreement for the InP and only 67% agreement within IVC (z=1.78, p=0.075). Next, we compared the average percentage agreement across all items when raters were in the same setting (87.25%; SD=5.49%) with the average percentage agreement across all items when raters were in different settings (83.94%; SD=7.02%). Although there was significantly higher agreement when pairs were in the same setting (t₃₅=4.08, p<0.01), results indicated adequate inter-rater agreement both within and across settings.

For the ADI-R items, kappa values were much higher than that of the ADOS. Average kappa across the 36 items when raters were in the same setting was 0.79. Average kappa when raters were in different settings was 0.74. Average kappa within InP was 0.76, and that within IVC was 0.82. Kappa scores were generally moderate to high regardless of condition, indicating adequate agreement among raters.

Parent Satisfaction

Parents completed a survey to compare family satisfaction across the IVC and InP conditions. Using a Likert scale (from 1=strongly disagree to 7=strongly agree) parents provided responses to seven questions about their experiences. We reverse-scored two items so that for all items high ratings would indicate high satisfaction with the evaluation. Table 3 represents means, SDs, t test summary, and effect size (Cohen's d) for each question and for mean satisfaction across all items. There were no significant differences in mean satisfaction between conditions (p≥0.15) on any items. Because of relatively small sample sizes, we examined effect sizes for additional interpretation. Two of the effect sizes were moderate. Item 7, related to overall satisfaction with the evaluation visit, was rated as high as possible by all the IVC families (mean of 7.0). The overall rating was also high for the InP families (mean of 6.82), but the effect size was moderate, suggesting a potentially meaningful difference in satisfaction favoring the IVC condition. Conversely, the effect size was moderate when all seven items were averaged, suggesting that average satisfaction in the InP condition (mean of 6.57) was higher than satisfaction in the IVC condition (mean of 6.23). This discrepancy may indicate no meaningful difference in satisfaction across condition or that with larger sample sizes potential differences would be clarified. Although two of the effect sizes are moderate, the actual difference in the ratings appears clinically insignificant as ratings in both conditions ranged from 6.23 to 7.0, which indicates high satisfaction. A larger study is needed to provide more in-depth examination of satisfaction with evaluation across InP and IVC conditions. Given an effect size of 0.6 as observed in this study, one would need to have a sample of 45 to detect a significant difference between groups at 0.80 power.

Table 3.

Mean Satisfaction Ratings by Condition

CONDITION	N	MEAN	SD	T TEST	EFFECT SIZE (D)
Evaluation 1: comfort with talking to clinicians
InP	11	6.82	0.41	t19=0.89	0.38
IVC	10	6.30	1.89	p=0.39
Evaluation 2: felt clinicians cared
InP	11	6.73	0.47	t19=0.91	0.39
IVC	10	6.20	1.87	p=0.38
Evaluation 3: competent clinicians
InP	11	6.91	0.30	t8.30=0.86	0.41
IVC	9	6.33	2.00	p=0.42
Evaluation 4: felt embarrassed
InP	11	6.00	1.90	t19=0.95	−0.10
IVC	10	6.20	1.93	p=0.36
Evaluation 5: difficulty hearing
InP	11	6.91	0.30	t19=0.70	0.30
IVC	10	6.70	0.95	p=0.50
Evaluation 6: no trouble seeing
InP	11	5.82	2.40	t19=0.81	0.35
IVC	10	4.90	2.77	p=0.43
Evaluation 7: overall satisfaction
InP	11	6.82	0.41	t10=–1.49	–0.62
IVC	10	7.00	0.00	p=0.17
Mean satisfaction Items 1–7
InP	11	6.57	0.49	t19=1.16	0.50
IVC	10	6.23	0.82	p=0.26

Evaluation questions 4 and 5 were reverse-scored to create high scores=high satisfaction.

InP, in-person; IVC, interactive videoconferencing; SD, standard deviation.

Discussion

Given the current prevalence estimates for autism, guidelines for best practices, and disparities in early access to services, there is a need for alternative methods for providing autism-related services, including diagnosis. The current study explored the administration of the ADOS and ADI-R over IVC to help inform an autism diagnosis. Based on clinical judgment, research clinicians assigned a classification of autism or developmental delay using the DSM-IV criteria. Results suggest research clinicians were consistent in identifying children with autism when using IVC. Clinicians were consistent with one another nearly 100% of the time (20 out of 21 cases) when assigning a diagnosis, regardless of the condition and existing diagnosis.

Our results suggest little difference in inter-rater agreement between ratings on autism measures scored InP and over IVC. A significant difference in mean percentage agreement between conditions occurred for Item A7 on the ADOS related to socially directed pointing, whereas no significant differences were found for any items on the ADI-R. Differences in ratings of socially directed pointing could be due to difficulty in observing eye contact and approximations of pointing over IVC between the child and the parent who was eliciting the behavior. However, for ADOS Item A7 the agreement across settings was actually higher than the agreement within settings, suggesting differences may be not be related to difficulty observing via IVC. All other items were adequate in agreement between the pairs of clinicians regardless of condition. These findings indicate clinicians were able to observe behavior and score the ADOS and ADI-R items equally well when viewing the child InP or over IVC. This preliminary evidence suggests these tools can be effectively used over IVC.

Results from the satisfaction survey showed high levels of family satisfaction with no significant differences between conditions. Results show a moderate effect size in favor of the InP setting for satisfaction across all seven items of the survey. However, ratings of 6.57 and 6.23 (corresponding to “satisfied”) out of 7 in the InP and IVC conditions, respectively, suggest families were highly satisfied regardless of condition. A larger sample would help determine if these differences are meaningful or a result of sample size.

Results from this study provide preliminary evidence that autism diagnostic assessments can be conducted equally well over IVC compared with the traditional InP clinic setting. Although we recognize that these tools are only one component of the diagnostic process, our results suggest that clinicians can use IVC to accurately differentiate autism-like behaviors from non–autism-like behaviors. Clinicians scored measures consistently for most items, and families reported high levels of satisfaction in all conditions. With the increasing demand for trained professionals to conduct diagnostic assessments for autism, the use of IVC provides a viable option for families where there may be limited access to specialty care. Families often travel long distances for similar evaluations, resulting in travel expenses as well as lost work time. IVC may provide families with increased access to needed medical care.

Limitations and Future Directions

Although results from this study suggest IVC as a viable option for autism diagnostic evaluations, we recognize several limitations, including communicating diagnosis, issues of assessment (including tools and other evaluation elements), and sample size. In identifying these limitations, we also offer suggestions for future research that may strengthen the current study design.

There may be issues related to communication of diagnosis that could not be evaluated in this study. All of the children in the study had previously received a diagnosis of autism or developmental delay, and their families had some prior education about their child's diagnosis. Future research should explore the use of IVC for communicating information related to a diagnosis.

In this study, we used only Module 1 of the ADOS. Modules 2–4 contain more standardized activities, and families may not have been able to conduct activities from those modules without significantly more guidance. In fact, some families had difficulty with the presses that were required in Module 1 (e.g., families often required multiple explanations to set up a scenario evaluating the presence of joint attention). Future research should investigate the use of other ADOS modules over IVC. Another limitation involves the modified administration of the ADOS. Although parents were not formally trained in the use of ADOS administration procedures, many IVC sites have facilitators who could be trained on the ADOS. We have also found that school district personnel often accompany children and families to IVC appointments. Many of these personnel could also be trained on the ADOS not only to assist in assessment, but also for use in educational planning. Although this study has provided informative results, we recognize that the clinical activities conducted did not represent a full evaluation. Typical evaluations include a physical exam, medical history, cognitive, language, and adaptive testing as well as autism measures. The medical history could be obtained over IVC with the on-site nurse conducting a physical. Cognitive, language, and adaptive levels may also be obtained from school records. Future research should address which children can be seen over IVC and which children would need to come to a clinic site for evaluation.

Given its preliminary nature, the current study used a small sample size. Although very few items were significantly different between raters in the InP and IVC conditions, some differences approached significance. A larger sample size with more power could help identify items on the ADOS and ADI-R that are difficult to score over IVC. Larger samples may also allow for a wider range of possible diagnoses that more broadly represent possible referrals encountered using IVC.

Future studies may include cost evaluation to determine whether and/or how the use of IVC impacts the cost of care to families and providers. We know that early intervention could potentially save costs related to providing services to individuals with autism and developmental delay later in life.^23,24 It is important to understand the cost implications of using technologies such as IVC. The current study suggests IVC as a potential means for conducting evaluations of young children suspected of having autism. The findings described here indicate the need for further study with larger sample sizes, investigating other aspects of comprehensive assessment, use of additional ADOS modules, and communication of results.

Acknowledgments

A Department of Defense Autism Concept grant (number W81XWH-08-1-0233) provided funding for this pilot project. The Center for Telemedicine and Telehealth at the University of Kansas Medical Center provided ongoing technical support during the project with consultation and support from Dave Cook, Eve-Lynn Nelson, and Ryan Spaulding.

Disclosure Statement

No competing financial interests exist.