Implementation of Web-based Autism Screening in an Urban Clinic

Abstract

Screening toddlers for Autism Spectrum Disorder (ASD) with the Modified Checklist for Autism in Toddlers, Revised (M-CHAT-R) has been shown to lower age of diagnosis by two years (Robins et al., 2014). In order to streamline ASD screening, research is exploring the use of web-based screening during well-child check-ups. The current study examined implementation of the web-based M-CHAT-R in an urban pediatric clinic in Atlanta, Georgia. Toddlers (N=2,557; 87% African American) were screened during well-child visits (M_age=22.43 months, SD=3.65). Using the web-based version resulted in a 58.5% increase in the number of cases screened per month. A similar proportion of toddlers in each modality screened positive (p = .43), but significantly fewer children were missing Follow-Up in the web-based administration (p<.001). These results suggest that it is feasible to implement web-based screening in underserved populations. Future research is necessary to understand factors that facilitate successful implementation of web-based ASD screening.

Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by deficits in social communication and presence of restricted and repetitive behaviors.¹ Current estimates in the US are that 1 in 68 children have an ASD.² Given that early intervention has been associated with the best outcomes for children with ASD,^3,4 efforts have been made to implement early universal screening of ASD in pediatric practices to facilitate early detection. Although routine screening has been recommended by The American Academy of Pediatrics,^5,6 racial and socioeconomic disparities in age of identification persist.^7,8 Delayed age of diagnosis in conjunction with cultural perceptions of healthcare and lack of financial resources has also been associated with delayed access to early intervention services,^7,9 which negatively influences prognosis.

One approach to address disparities in the early identification of ASD may be to enhance current screening practices, and increase access to screening for underserved populations. In the recent past, computer based testing has been suggested as a potential means to increase efficiency in developmental screening practices.¹⁰ For example, the Ages & Stages Questionnaires, 2^nd Edition¹¹ has undergone adaptation to a web-based modality from a paper-pencil completion, and has been shown to have revealed support for the general equivalence across modalities.¹² This finding provides promising support for the future use of web-based administration for additional developmental screening measures.

Web-based screening has also been used to enhance the early identification of ASD. Harrington and colleagues¹³ administered the Modified Checklist for Autism in Toddlers (M-CHAT) using an iPad within an urban pediatric setting. Electronic administration of the M-CHAT was rated as more favorable by parents and reduced scoring errors due to the automatic scoring feature.¹⁴ Additionally, the electronic version included the Follow-Up questions for screen positive cases, which streamlined the screening process. These results suggest initial support for using an electronic version of the M-CHAT. Further research is necessary to assess the feasibility and validity of web-based screening measures in diverse clinical settings.

We compared paper and web-based M-CHAT-R screening over a five-year period in an under-resourced urban clinic serving a diverse patient population. Feasibility was measured by comparing the rates of screening across modalities. Preliminary validity was evaluated by examining screening results across modalities.

Method

Participants

Parents and toddlers were enrolled in a larger screening study to validate the use of M-CHAT-R/F. For the purpose of this investigation, only families who received pediatric services from Children’s Healthcare of Atlanta-Hughes Spalding were included in the analyses. Families completed the M-CHAT-R/F at their 18- and/or 24-month well child visit. A total of 2,557 toddlers were screened using either paper or web-based versions of the M-CHAT-R/F. In cases where the same child was screened more than once, only the first screening was included. The sample (N=2,557) was 47% female and 52% male (1% no response), with an average age of 22.43 months (SD=3.65). The sample was predominantly African American (87%), with the average years of maternal education being a high school diploma/GED (M =12.37, SD = 1.32). Individuals who were not fluent in English were excluded from the study. Additionally, children who had already received a diagnosis of ASD prior to being screened were excluded from the study.

Measures

The Modified Checklist for Autism in Toddlers, Revised with Follow-Up (M-CHAT-R/F¹⁴): The M-CHAT-R/F is a 2-stage screener, valid for children 16–30 months old. The paper version of the screener is available online (see www.mchatscreen.com) for free download for clinical, research, and educational purposes, and requires minimal training for the healthcare team. Children whose total score was ≥3 initially and ≥2 after Follow-Up were considered screen positive and were invited for a free diagnostic evaluation. The paper version of the M-CHAT-R/F was validated in Robins et al.;¹⁵ 123 ASD cases were detected (105 based on positive screen, and 18 from alternate strategies to find missed cases) from a sample of 16,115 patients. The psychometric properties from this validation sample¹⁵ were strong: sensitivity= .854, specificity= .993, positive predictive value= .475, negative predictive value= .999, Likelihood⁺ ratio=114.05.

Procedure

Parents completed the M-CHAT, Revised with Follow-Up (M-CHAT-R/F) during an 18- or 24-month well-child visit at Children’s Healthcare of Atlanta- Hughes Spalding. Prior to completion, parents provided informed consent and reported demographic information. In the paper modality, parents were given a packet containing the consent form and M-CHAT-R questionnaire. Completed questionnaires were picked up by study personnel and scored. Parents of children who demonstrated risk on the paper version of the M-CHAT-R with a score of ≥ 3, were contacted by research personnel to complete the Follow-Up as a telephone interview. If children continued to show risk at Follow-Up, with a score of ≥ 2, parents were offered a free diagnostic evaluation at Georgia State University. Pediatricians indicated ASD concerns for the child by checking a box on the paper screen that was marked “office use only.” A small subset of screen-negative participants was also invited to complete a developmental evaluation based on pediatrician concerns for ASD or other indications of ASD concerns.

The web-based screener was completed using a netbook provided to the pediatric office. Parents read a consent form on the screen, and typed in their name in lieu of a signature. Parents reported the same contact information and demographic information as in the paper modality, and then continued on to the M-CHAT-R questions. The initial set of 20 yes/no questions was scored automatically. Children whose scores were 0–2 were considered low risk and screening was complete. Children whose scores were 3–7 were at moderate risk, and the relevant Follow-Up questions were administered automatically; children who continued to demonstrate risk on 2 or more Follow-Up items were considered screen positive. Children whose initial screening scores were 8 and higher were considered screen positive, and the Follow-Up questions were bypassed. Physicians and their staff had access to screening results immediately. Research personnel received the web-based screens through a secure online portal and then contacted families who screened positive to offer diagnostic evaluations at Georgia State University. Pediatricians indicated ASD concerns by adding a note in the electronic portal. A small subset of screen-negative participants was also invited to complete a developmental evaluation based on pediatrician concerns for ASD or other indications of ASD concerns. See Figure 1 for a graphic depiction of procedure.

Figure 1.

General Procedure

Results

Data were analyzed using the Statistical Package for Social Sciences (SPSS 18). The paper version of the M-CHAT-R/F was administered between the months of June 2009 and February 2014 (n=2,042), and theweb-based version of the M-CHAT-R/F was administered between the months of February 2014 to October 2014 (n=515). See Table 1 for demographic data by screening modality. Mothers evaluated using the web-based screening modality had significantly higher maternal education (M = 12.72, SD = 1.53), than mothers who were evaluated with the paper version (M = 12.27, SD = 1.24), (t (2252) = −6.86, p < .001), although the effect was small (eta² = .02).

Table 1.

Demographic Characteristics of the Sample by Modality

	Paper Version (N=2,042)	Web-based Version (N=515)	Total Sample (N=2,557)
Child Age (in months)
Mean	22.41	22.51	22.43
SD	3.66	3.63	3.65
Sex of child
Male	1,057 (51.8%)	272 (52.8%)	1,329 (52%)
Female	956 (46.8%)	239 (46.4%)	1,195 (46.7%)
Parent Race
African American	1,778 (87.1%)	456 (88.5%)	2,234 (87.4%)
Caucasian	34 (1.7%)	9 (1.7%)	43 (1.7%)
Mixed Descent (i.e.,    Bi-racial or Bi-    ethnic)	64 (3.1%)	12 (2.3%)	76 (3%)
Hispanic/Latino	23 (1.1%)	3 (.6%)	26 (1.0%)
Asian	21 (1%)	7 (1.4%)	28 (1.1%)
Native American	--	1 (.2%)	1 (.0%)
Maternal Education
<High School	373 (18.3%)	82 (15.9%)	455 (17.8%)
HS Diploma	888 (43.5%)	208 (40.4%)	1,096 (42.9%)
Some College (no    degree)	359 (17.6%)	102 (19.8%)	461 (18.0%)
Technical    School/Associates    Degree	48 (2.4%)	67 (13%)	115 (4.5%)
Bachelors Degree	54 (2.6%)	33 (6.4%)	87 (3.4%)
Some graduate    School	4 (.2%)	13 (2.5%)	4 (.2%)
Masters	19 (.9%)	13 (2.5%)	32 (1.3%)
Doctoral Degree    (e.g., PhD, MD, JD)	4 (.2%)	--	4 (.2%)

Figures 2 and 3 show the flow of participants through the study, for each modality. No significant differences were observed in screen-positive rate based on modality, χ² (2, N = 2,557) = 1.69, p =.43. Additionally, there were no significant differences in M-CHAT-R total scores based on modality (Paper M = 1.41, SD = 2.05; Web-based M = 1.33, SD = 1.86; t (2,555) = .74, p = .46). As expected, there is a significant association between screening modality and missing data at the Follow-Up stage of the screening process, χ² (1, N = 427) = 32.11, p < .001. All children who screened positive (at-risk for ASD) based on the M-CHAT-R require structured Follow-Up items to determine final risk level. Using the paper version of the M-CHAT-R/F resulted in 35.1% missing Follow-Up, done in the form of phone interviews; the primary causes of missing Follow-Up were parents being unresponsive to calls and phone numbers that were out of service. In contrast, the web-based modality is missing only 3.1% of children with incomplete Follow-Up due to parents not completing the second stage of screening questions administered in the single web-based screening session.

Figure 2.

Screening Procedure for Paper Version of M-CHAT-R

Figure 3.

Screening Process for the Web-based Version M-CHAT-R

In order to compare the rate of screening based on modality, the number of completed screens was computed for each month, and averaged for each modality. Web-based screening (M = 56.78, SD = 15.88) was completed at a significantly higher rate compared to paper screening (M =35.82, SD = 10.75), t (64) = −5.07, p < .001, eta² = .29. When the staff used the web-based screening modality, there was a 58.5% increase in the number of cases screened per month compared to the original paper modality. This increase is much larger than would be accounted for by an 8.5% increase in clinic client volume during the duration of the study.

Additional chi-square analyses were conducted to determine whether parents endorsed at-risk responses for specific M-CHAT-R items more or less often based on modality. Significant differences were observed across four individual items (i.e., items 1, 3, 12, and 17; see Table 2). Three items were answered “at risk” more often in the paper sample, and one item was answered “at risk” more often in the web-based sample.

Table 2.

Significant differences in at-risk responses to specific M-CHAT-R items across modalities

	Paper	Web-Based
M-CHAT-R Item	At- Risk for ASD	At-Risk for ASD	Chi-Square (df=2, N=2,557)	p
1. If you point at something across the room, does your child look at it? (For example, if you point at a toy or an animal, does your child look at the toy or animal?)	70 (3.4%)	8 (1.6%)	χ2 = 6.45	.04
3. Does your child play pretend or make-believe? (For example, pretend to drink from an empty cup, pretend to talk on a phone, or pretend to feed a doll or stuffed animal?)	448 (21.9%)	75 (14.6%)	χ2 = 20.22	< .001
12. Does your child get upset by everyday noises? (For example, does your child scream or cry to noise such as a vacuum cleaner or loud music?)	372 (18.2%)	92 (17.9%)	χ2 = 9.07	.01
17. Does your child try to get you to watch him or her? (For example, does your child look at you for praise, or say “look” or “watch me”?)	215 (10.5%)	72 (14%)	χ2 = 6.80	.03

Discussion

The purpose of the current investigation was to demonstrate feasibility of web-based screening in a busy urban practice, and also to examine the preliminary evidence regarding performance of the screening tool in the web-based modality compared to the validated paper modality. Results of this investigation, commensurate with findings from Harrington et al.,¹³ provide support for the implementation of autism-specific (M-CHAT-R/F) web-based screening to identify risk for ASD. Web-based screening appears to be an efficient and feasible way to screen more toddlers and to seamlessly follow up with those who are identified as at-risk to gather further detail about potential symptoms. Although physicians and office staff initially had concerns about the computer literacy of their patients, web-based screening led to a 58.5% increase in the number of toddlers screened during the first nine months of implementation at an urban low-income pediatric office in Atlanta, Georgia. This suggests that the use of web-based technology was not a barrier to completing screening during well-child pediatric visits. It is important to note that although the authors were not able to determine the precise clinic volume of 18 and 24 month visits specifically, overall the clinic showed only a modest increase in volume (8.5%) which does not account for the large increase in screening rate after transitioning to electronic screening.

Moreover, the significant reduction of incomplete Follow-Up screens when using the web-based modality may be particularly helpful when implementing screening of developmental delays in clinics serving low-income families. As noted in Khowaja, Hazzard, and Robins,¹⁶ although the Follow-Up interview is helpful in reducing a high initial screen positive rate, lower maternal education was associated with higher rates of incomplete Follow-Up; one cause was phone numbers that were out of service, making it difficult to get in touch with parents to complete the second stage of screening by phone. The immediate trigger of relevant Follow-Up questions in a single screening session significantly improved Follow-Up completion rates. This suggests that implementing web-based administration is a potential solution to reduce socio-demographic barriers in screening for developmental delays.

Results also provided preliminary evidence that the performance of the M-CHAT-R is similar across modalities. Most items did not show a differential rate of at-risk responses across modalities. With the exception of item 17 (i.e., Does your child try to get you to watch him or her? (For example, does your child look at you for praise, or say “look” or “watch me”?), there tended to be no difference or a lower at-risk response rate when using the web-based version compared to paper. It is possible that when parents are given a paper version they may quickly circle items to finish all their paper work before seeing the doctor. This rush may have influenced response patterns, although it is only evident in three items. Although significant differences in at-risk response rate were found in four items, this did not significantly influence the overall screen-positive rate or the total scores, suggesting that performance of the M-CHAT-R is similar across modality.

When considering the generalization of these findings it is important to highlight potential limitations of the current study. For example, the focus of the current study examined one pediatric office serving a racially diverse urban population. Replication of these findings in additional (e.g., rural settings, ethnically diverse) pediatric settings is necessary to provide further support of the success of web-based autism-specific screening. Further, we were unable to track the parents who refused to participate in the study, but this was a very small proportion of the number of screened patients. Information regarding parent perceptions and satisfaction in the use of the web-based version was outside the scope of the current investigation. However, this information may be helpful in further comparing the use of paper and web-based screening measures.

Although the results of this investigation provide preliminary support for using web-based universal ASD screening in pediatric offices, this investigation did not explore whether site-specific or individual factors facilitate successful web-based screening implementation. For example, factors such as pediatrician attitudes towards autism-specific screening may play an important role in the successful implementation and potential problem solving involved in transitioning from prior screening practices. Given that the pediatric office used in this study was involved in paper autism-specific screening for approximately four years prior to transitioning to the web-based version, it is possible that these findings may not generalize to other pediatric offices that are unfamiliar with developmental or ASD-specific screening.

Additionally, other pediatric offices may face several practical barriers to implementing web-based screening. As reported by Radecki et al.,¹⁷ approximately half of pediatricians do not routinely use formal screening tools in patients younger than 36 months. This suggests that there continue to be barriers in the widespread utilization of screening for developmental delays in toddlers. Sheldrick & Perrin¹⁸ discussed potential challenges to web-based screening including lack of familiarity with new technology and costs of equipment. In response to these concerns, we point out that most young parents are exposed to technology during school, 90% of US adults have cell phones (www.pewinternet.org), and many options for accessing web-based screening are fairly low cost, such as the netbooks used in the current study, which were purchased for less than $200. Additionally, many public locations (e.g., coffee shops, libraries) offer free Internet. Therefore, it appears that barriers such as lack of familiarity with technology may be less of a concern and new equipment costs can be addressed. Other factors such as experience with autism-specific screening and pediatrician attitudes towards screening should be explored in order to better understand how to successfully establish and maintain web-based screening in other pediatric offices. It is important to note that the current study did not randomly assign participants to web-based administration or paper conditions across the same time period. Although this is a limitation of the current study, this is consistent with the methodological approach from similar investigations (e.g., Harrington et al.¹³).

This investigation provides encouraging findings in support of using web-based autism-specific screening as an effort in reducing disparities in the age of identification of autism in an urban setting. Continued efforts by pediatric office staff are necessary to promote successful integration of web-based screening within pediatric practices. Given the efficiency and opportunity for reducing age of identification of ASD, widespread implementation of web-based screening at pediatric offices is a feasible and worthwhile endeavor. There is also the potential for direct integration of web-based screening into the electronic health record, which further increases efficiency and the likelihood that pediatricians incorporate routine screening into their practices. The improved rate of overall screening, coupled with the minimization of data loss at the Follow-Up stage, suggests that web-based screening is an improvement over paper administration, and will facilitate universal ASD screening.