Abstract
Awareness and understanding of autism are increasing, sometimes in small incremental steps and sometimes through paradigm shifts. Much of this is driven by autistic advocates, whose voices are increasingly recognized as experts in their own lived experiences. The growing presence of autistic people in autism research—as participants, advisors, and (co)researchers—has brought about changes in the quality and relevance of autism research. We are seeing more studies by autistic autism researchers, more studies on issues of importance to autistic people, and more studies utilizing methodologies that resonate with autistic people. However, there remains a reliance on measures that are technically “valid” and “reliable” rather than relevant. As long as this is the case, we will continue to see studies published that “explain autism” in ways that do not reflect autistic experiences. Autism research is still making incremental changes and it is time for a paradigm shift.
Keywords: research, measurement, methodology
Community brief
“Why is this topic important?”
Research into the experiences of autistic people is used to inform policy, practice, and community understanding. It is important that autism research respects and reflects autistic voices. Current research often uses scientific instruments that are not consistent with our lived experience of autism.
“What is the purpose of this article?”
This article explains why the current approach to research—encouraging the use of existing measures and incremental change—is harmful to autism research and to autistic people. It provides examples of commonly used tools and explains why they are problematic. The article aims at encouraging a discussion into why the findings of autism research are often inconsistent with the experiences of autistic people.
“What personal or professional perspectives do the authors bring to this topic?”
I am an autistic adult, the mother of two autistic adults and an autism advocate. I am also an autism researcher with a background in public health and communication.
“What is already known about this topic?”
Academic journals aim at publishing papers that present the best and most accurate research projects. Researchers are encouraged to use tools that have been developed and tested in previous studies, because these are considered to be “reliable” (consistent) and “valid” (accurate). Researchers are discouraged from developing new measures or making major changes to existing ones. So, it can be hard to publish community-driven research that uses tools developed by autistic people.
“What do the authors recommend?”
I recommend that autism researchers ensure that the tools they use to study autism actually measure the experiences of autistic people. I also recommend that we stop relying on existing measures that do not make sense to autistic people and work with autistic people to develop better tools.
“How will these recommendations help autistic adults now or in the future?”
These recommendations will help autistic adults by making autism research more relevant and useful. Research that reflects our actual experiences will improve community understanding of autism. It will also lead to better policies and practices that meet the needs and aspirations of autistic people.
Measuring the Wrong Thing the Right Way? Time to Rethink Autism Research Tools
Any researcher who has tried to publish their findings in a scientific journal will know the importance of using measures that have been shown to be “reliable” and “valid.” Unfortunately, autism researchers who have utilized measures developed by autistic people to measure aspects of their experience in a way that is meaningful and respectful will have had their hopes dashed. Reviewers typically conclude that the study is not publishable, because the measures used were created for the study and the authors should have used measures that have been used in previous studies.
Why am I—as an autistic autism researcher—calling for a new approach? Why am I so opposed to using the standard measures used in autism research? Why do I think it is harmful to future autism research for reviewers to gatekeep in this manner?
In this article, I present two examples of existing measures as a way of demonstrating why I would rather see research with untested measures that actually measure the things that matter to our lives. I present these examples not as a criticism of the individual studies or their authors, but rather as a reflection on a system that rewards incremental change and sameness over innovation and insight.
I present them in the context of previous research, which has demonstrated that many existing measures present the relevance and accessibility issues for autistic people.1 I also present them through the lens of my personal experience and expertise: as an autistic person, the parent of two autistic people, an autism advocate, and an academic with more than 20 years of experience in qualitative and quantitative research.
Case Study 1: Societal Attitudes to Autism
A 2019 article commenced with the query, from an anonymous autistic person: “Why are neurotypicals so pig-ignorant about autism?” and concluded with:
The question “Why are neurotypicals so pig-ignorant about autism?” could perhaps best be responded [to] by the results demonstrating that [the] clear majority in this sample were actually not ignorant about autism. In addition, 99.9% had positive or strongly positive attitudes towards autism.2
As an autistic person and an autistic researcher, the finding that “81.3% of the overall population reported a strong positive attitude towards autism” and all but one of the remainder a positive attitude was so divergent from my (and others) lived experience that I thought I must have misread it.
Data from research with autistic people and their families consistently show that this cohort experiences significant exclusion and discrimination. For example, autistic people experience systemic barriers to education and employment3,4 and autistic adults report feeling isolated and excluded from the community.5
The study utilized an existing measure that has been widely used in other research—the Social Attitudes Toward Autism (SATA)6 scale; specifically, the social attitudes subscale. This 16-item instrument has been shown to have a “good internal consistency and construct validity” and, thus, has been recommended as a measure of societal attitudes. Since that time, the SATA has been used in several published studies as a measure of attitudes toward autistic people.7–9
The 16-item SATA scale has a possible range of 16–64. In the original study, the “experts” had a mean score of 54.38 and the undergraduates a mean score of 28.14.6 The 2019 study (with 15 items) had a possible range of 15–60 and a mean score of 54. It would be easy to conclude that the remarkable finding from this study was a result of bias in the sampling. The authors note that the sampling techniques were “purposive, convenience and snowball sampling,” including personal contact and media stories.
It is likely that those who volunteered to participate had both some awareness of autism and a degree of interest in the topic; this possibility (not noted in the paper) is evident in the uncanny similarity in the scores of the 2019 respondents and the original “experts.”
However, to attribute it solely to this recruitment strategy would be to ignore the bigger question: How are attitudes toward autism measured?
In the original study, reliability was determined by the Cronbach alpha score for internal consistency (16 items = 0.86), although the authors note that the relationship between this and the other subscales was inconsistent.* So, we know that there is consistency in people's responses across the items in the scale.
In the original study, validity was determined by comparing the results both across participant samples and across measures. In the former, the responses of 475 undergraduate students were compared with the responses of 22 service providers who work with autistic children, demonstrating that the “experts” had more positive attitudes than the university students.
In the latter, the SATA scores showed a “medium positive association” with a seven-item Preference for Persons with Autism Statement; a subset of the participants completed Attitudes toward Disabled Persons, finding a “medium positive association” with their SATA scores; and a smaller subset completed an implicit association test, indicating a “small positive correlation” with their SATA scores. Interestingly, however, there was no significant relationship between the scores and responses to a question about their willingness to volunteer to work with “people with disabilities.” A recent paper in this journal similarly found a nonsignificant correlation between an Implicit Association Test and the SATA.10
The SATA scale items are shown in Table 1. As an autistic person and advocate, I am somewhat concerned by the low bar set by these items. To be a person with a “positive attitude” in the 2019 study, I needed to score 31 out of 54 (item 16 was not included in this study).
Table. 1. Social Attitudes Toward Autism Scale Items
| People with autism should not engage in romantic relationships. |
| People with autism should have the opportunity to go to university.a |
| People with autism should not have children. |
| People with autism should be institutionalized for their safety and others. |
| If a facility to treat people with autism opened in my community, I would consider moving out. |
| Individuals with autism are incapable of living on their own. |
| I would be afraid to be around a person with autism. |
| A person with autism is an emotional burden to his/her family. |
| I would be comfortable sitting next to a person with autism in the same class.a |
| A person with autism is a financial burden to his/her family. |
| People with autism should be encouraged to marry a person with autism.b |
| People with autism are incapable of forming relationships and expressing affection. |
| Children with autism should be fully integrated into mainstream classes.a |
| I would be uncomfortable hugging a person with autism. |
| People with autism cannot understand other people's feelings. |
| Students with autism who are mainstreamed into regular classrooms are a distraction to students without autism in that classroom. |
Reverse scored.
I am unsure how this is meant to be interpreted. Strongly disagreeing with the statement would be scored as a positive attitude (but the ambiguity of the statement means that I could score the same whether I interpret it to mean that autistic people should marry autistic or non-autistic people OR to mean that autistic people should not marry anyone, autistic or not).
Let us consider some hypothetical people who could have completed the survey.
Hypothetical Person (HP) #1 thinks that autistic people are fairly capable; that they can go to university, form relationships, and live independently—however, HP #1 would not be comfortable sitting next to an autistic person or hugging an autistic person and is a little bit afraid of autistic people (SATA score 36, positive attitude).
HP #2 thinks that autistic people are cute and quirky and would happily hang out with an autistic person—however, HP #2 does not think that autistic people are capable of living independently of their family or achieving successful careers so they are probably a burden on their family (SATA score 41, positive attitude).
HP #3 thinks that autistic people are competent, capable members of society and welcomes them to the community—however, HP #3 “knows” that autistic people are incapable of feeling or expressing emotions so they should not get married or have children (SATA score 42, positive attitude).
Sadly, people such as these are very real. As research shows, and autistic people can attest, the general population has a limited understanding of what it means to be autistic and we experience systemic exclusion and discrimination.4,11,12 A study of special education teachers in Malaysia shows the risk of focusing on mean scores to quantify attitudes; the teachers reported overall positive SATA scores, but the authors note that their lowest score was for the item “Students with ASD who are mainstreamed into regular classrooms are a distraction to students without ASD in that classroom.”8
Case Study 2: Repetitive Behaviors
Restricted and repetitive behaviors are one of the key characteristics of autism, as reflected in the diagnostic criteria13 and the accounts of autistic people and their families. Consistent with the historical focus of autism research on children, rather than adults, the majority of existing instruments are designed to measure childhood behaviors.
In a welcome step, recent research has begun to explore the assessment of restricted and repetitive behaviors in autistic adults. The Adult Repetitive Behaviour Questionnaire-214 has recently been attracting attention in the field, with the instrument used in a range of studies, including research into cognitive processes of autistic adults.15
As an autistic person (and researcher), I was delighted to see mention of this measure appearing in the literature; but I was disappointed when I saw the actual report. The modifications from the (Child) Repetitive Behaviour Questionnaire to the Adult Repetitive Behaviour Questionnaire are minimal, primarily grammatical and the addition (or removal) of a few words. See Table 2 for a comparison of the two instruments.
Table. 2.
Comparison of the Child and Adult Repetitive Behavior Checklists
| [Does your child/Do you] |
|---|
| Like to arrange [toys or other things/items] in rows or patterns? |
| Repetitively fiddle with [toys or other things/items?] (e.g., spin, twiddle, bang, tap, twist, or flick anything repeatedly?) |
| Spin yourself around and around? |
| Rock backward and forward, or side to side, either when sitting or when standing? |
| Pace or move around repetitively? (e.g., walk to and fro across a room, or around the same path [outside/in the garden?]) |
| Make repetitive hand and/or finger movements? (e.g., flap, wave, or flick hands or fingers [repeatedly/repetitively]?) |
| Have a fascination with specific objects? (e.g., trains, road signs, or other things?) |
| Like to look at objects from particular or unusual angles? |
| Have a special interest in the smell of people or objects? |
| Have a special interest in the feel of different surfaces? |
| Have any special objects you like to carry around? (e.g., a teddy, a blanket, a book, or a stick?) Please describe the object: |
| Collect or hoard items of any sort? Please describe what your child collects: |
| Insist on things at home remaining the same? (e.g., furniture staying in the same place, things being kept in certain places or arranged in certain ways?) |
| Get upset about minor changes to objects? (e.g., [bits/flecks] of dirt on clothes, [small/minor] scratches on [toys/objects]?) |
| Insist that aspects of daily routine must remain the same? |
| Insist on doing things in a certain way or redoing things until they are “just right”? |
| Play the same music, game, or video, or [look at/read] the same book repeatedly? |
| Insist on wearing the same clothes or refuse to wear new clothes? |
| Insist on eating the same foods, or a very small range of foods, at every meal? |
| What sort of activity will you choose if you are left to occupy yourself? |
| (a) A range of different and flexible self-chosen activities |
| (b) Some varied [play/and flexible] interests but commonly choose the same activities |
| (c) Almost always choose from a restricted range of repetitive activities |
Bold indicates only in child version; Italics indicates only in adult version; Grammatical changes—and reference person indicators—are not shown.
As the authors note, “As previous research into RRBs has generally focused on children, little is known about the expression and trends of RRBs across autistic adulthood, as well as how RRBs differ according to age and gender in autistic adults….”14 Thus, a reasonable starting point would be to do exploratory research into restrictive and repetitive behavior (RRBs), ideally by talking to autistic adults. Instead, they started with the child measure, made very minor modifications, and finally tested the instrument with a sample of autistic adults.
The instrument was found to be statistically reliable and valid (based on Cronbach's alpha and correlation analyses), and the authors conclude that the research shows “preliminary evidence of the usefulness of the RBQ-2A as a reliable and valid measure of RRBs in self-reporting autistic adults.”
When the results were compared with those for children, the scores were found to be significantly lower, particularly for Repetitive Sensory and Motor Behaviours (and even more so for older adults). Thus, the authors conclude: “RSMBs do not only occur less often in autistic adults compared to IS [Insistence on Sameness] behaviours, but this difference is particularly pronounced in older adults.” As an autistic person, I found that conclusion somewhat surprising!
Assuming it is correct that as we age we engage in less RRBs, the important question is why? Perhaps one clue comes from the differences in responses across items. The lowest levels of repetitive sensory and motor behaviors (RSMB)-item endorsement were for spin (1.29) and rock (1.87), and the highest level of endorsement was for fiddle (2.51); so perhaps as we age, we engage in less RSMBs that are visible to others and socially stigmatizing while continuing ones that are easier to hide or to undertake in a socially acceptable way. If this is the case, then reduced RRB scores may well be associated with increased masking, which, in turn, is associated with reduced well-being.16,17
It is also important to consider the way that virtually identical questions will be answered by two very different populations: parents reporting on their child's behavior and adults reporting on their own behavior. First, as adults we have more autonomy over our behaviors and more control over our environment. If I am soothed by the feeling of rocking, I can purchase a rocking chair; if I enjoy spinning, I can take up dancing (and then I can honestly reply no to the questions on “spin” and “rock”). Second, as an adult, I have insights into my behavior, which enables me to interpret my behavior as functional and/or modify it to be functional. If I like repetitive finger movements, I can take up knitting; if I like fiddling, I can become a jigsaw puzzle aficionado (and reply no to the questions on “fiddle” and “hand/finger”).
Another important consideration is how I as an adult interpret the questions and make sense of them in the context of my lived experience. Questions that may seem black and white to the parent of a small child can seem much more nuanced to an adult. For example, do I “Have a fascination with specific objects (e.g., trains, road signs or other things)?” It depends on how you define “fascination,” and even how you define “objects.” Are you asking whether I pick up bird feathers on my walk to the grocery store or are you asking whether I have an encyclopedic knowledge of native birds?
Recommendations for Getting Autism Research Right
Despite numerous calls for the inclusion of autistic people as collaborators and co-researchers, we are still measuring important aspects of autistic lives by using instruments developed by non-autistic people. It is not enough to make minor modifications to existing instruments so that we continue to “reliably” and “validly” measure the wrong things.
We have seen a number of paradigm shifts in the way we understand autism. We now know that autism is not a childhood condition, is not caused by refrigerator mothers, is neurodiversity not disease, and is prevalent (but under-diagnosed) in females.
It is time for a paradigm shift in the way we research autism. Rather than continuing to conduct research by making incremental changes to existing measures that misunderstand and misrepresent autistic people, we need to critically explore the applicability of existing measures and develop new measures that accurately reflect autistic experiences.
If we are to utilize measures developed with neurotypical populations to assess autistic populations, we need to first conduct research to determine that they measure the phenomenon of interest in this population; for example, recent research that explored the validity of the Beck Depression Inventory for measuring depression in autistic adults.18 Where these instruments are found lacking in specificity, we need to undertake further research combining insight from autistic people with scientific rigor; for example, the research on the existing Suicidal Behaviors Questionnaire-Revised (developed for the general population) that utilized qualitative and quantitative methods to produce and validate the Suicidal Behaviours Questionnaire—Autism Spectrum Conditions.19
The refinement and development of these measures needs to be undertaken in genuine collaboration with autistic people—as project advisors, pilot testers, co-researchers, and acknowledged experts in their own experiences. First, we need to understand the concept we are aiming at measuring. This understanding can only come from incorporating deep and meaningful consultation with autistic people. How do we observe and experience societal attitudes? How do we achieve our need for repetitive behaviors within the confines of a predominantly neurotypical society?
For too long, autism research has been limited by the rigid adherence to criterion-related validity—concurrent validity (does the measure produce similar results to other existing measures?) and predictive† validity (does the measure predict performance on an outcome variable?)—and particularly the former. We need to shift the balance to (at least) an equal focus on content validity: Does the test measure what it aims at testing (face validity), and does it relate to underlying theoretical concepts (construct validity)? In simple terms, does the measure resonate with actual autistic experiences?
This is not to suggest that we should abandon scientific rigor. These new measures will need to have content validity (measures a representative sample of the subject/behavior) and criterion validity (correlates with an established standard of comparison); they need to be carefully developed and tested to ensure they are reliable and valid measures of the underlying phenomena. In some cases, this will require researchers to collect more data—using both these new measures and existing measures—to facilitate this process of instrument validation, acknowledging the additional burden that this places on study participants.
This will take bravery on the part of researchers, to start from scratch and develop measures that accurately assess the topic of interest, developing appropriate tools in collaboration with autistic people. It will take bravery from authors to submit their research papers with their data based on new and novel instruments. However, mostly it will take bravery from journal reviewers and editors to accept that for the field to move forward we need to encourage a greater focus on content-related validity rather than criterion-related validity. Instead of focusing on whether we are measuring autistic experiences in the same way as our predecessors, let us be sure we understand what it is we are trying to measure.
Author Disclosure Statement
No competing financial interests exist.
Funding Information
The author did not receive any funding for this work.
The initial instrument consisted of three subscales: Factor 1 (the final SATA); Factor 2 (knowledge), which was determined, needed further development; and Factor 3 (personal distance), which was rejected.
These aspects are also important to retain alongside content validity; for example, predictive validity assures us that the measure predicts what it is supposed to predict.
References
- 1. Nicolaidis C, Raymaker DM, McDonald KE, et al. Creating accessible survey instruments for use with autistic adults and people with intellectual disability: Lessons learned and recommendations. Autism Adulthood. 2020;1;2(1):61–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Kuzminski R, Netto J, Wilson J, et al. Linking knowledge and attitudes: Determining neurotypical knowledge about and attitudes towards autism. PLoS One. 2019;14(7):e0220197. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Sarrett J. Interviews, disclosures, and misperceptions: autistic adults' perspectives on employment related challenges. Disability Stud Q. 2017;37(2). 10.18061/dsq.v37i2.5524. [DOI] [Google Scholar]
- 4. Cooper R, Kennady C. Autistic voices from the workplace. Adv Autism. 2021;7(1):73–85. [Google Scholar]
- 5. Jones SC, Gordon CS, Akram M, et al. Inclusion, exclusion and isolation of autistic people: Community attitudes and autistic people's experiences. J Autism Dev Disord. 2021. 10.1007/s10803-021-04998-7. [DOI] [PubMed]
- 6. Flood LN, Bulgrin A, Morgan BL. Piecing together the puzzle: Development of the societal attitudes towards autism (SATA) scale. J Res Spec Educ Needs. 2012;13(2):121–128. [Google Scholar]
- 7. Haney JL, Cullen JA. An exploratory investigation of social workers' knowledge and attitudes about autism. Social Work Mental Health. 2018;16(2):201–222. [Google Scholar]
- 8. Low HM, Lee LW, Ahmad AC. Knowledge and attitudes of special education teachers towards the inclusion of students with autism spectrum disorder. Int J Disability Dev Educ. 2019;67(5):497–514. [Google Scholar]
- 9. Thorsteinsson EB, Draper A, Lykins AD To Vaccinate or not: The relative impact of attitudes toward autism spectrum disorders and the ability to interpret scientific information on vaccination decisions. Int J Environ Res Public Health. 2020;17(7):2542. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Dickter CL, Burk JA, Zeman JL, et al. Implicit and explicit attitudes towards autistic adults. Autism Adulthood. 2020;2(2):144–151. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Aubé B, Follenfant A, Goudeau S, et al. Public stigma of autism spectrum disorder at school: Implicit attitudes matter. J Autism Dev Disord. 2021;51(5):1584–1597. [DOI] [PubMed] [Google Scholar]
- 12. Botha M, Dibb B, Frost DM. “Autism is me”: An investigation of how autistic individuals make sense of autism and stigma. Disability Soc. 2020. 10.1080/09687599.2020.1822782. [DOI]
- 13. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (5th ed.). 2013. 10.1176/appi.books.9780890425596. [DOI]
- 14. Barrett SL, Uljarević M, Jones CRG, et al. Assessing subtypes of restricted and repetitive behaviour using the Adult Repetitive Behaviour Questionnaire-2 in autistic adults. Mol Autism. 2018;9:58. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Cooper K, Russell A, Calley S, et al. Cognitive processes in autism: Repetitive thinking in autistic versus non-autistic adults. Autism. 2021. 10.1177/13623613211034380. [DOI] [PMC free article] [PubMed]
- 16. Beck JS, Lundwall RA, Gabrielsen T, et al. Looking good but feeling bad: “Camouflaging” behaviors and mental health in women with autistic traits. Autism. 2020;24(4):809–821. [DOI] [PubMed] [Google Scholar]
- 17. Bradley L, Shaw R, Baron-Cohen S, et al. Autistic adults' experiences of camouflaging and its perceived impact on mental health. Autism Adulthood. 2020. 10.1089/aut.2020.0071. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Williams ZJ, Everaert J, Gotham KO. Measuring depression in autistic adults: Psychometric validation of the beck depression inventory–II. Assessment. 2021;28(3):858–876. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Cassidy SA, Bradley L, Cogger-Ward H, et al. Development and validation of the suicidal behaviours questionnaire—autism spectrum conditions in a community sample of autistic, possibly autistic and non-autistic adults. Mol Autism. 2021;12(46). 10.1186/s13229-021-00449-3. [DOI] [PMC free article] [PubMed] [Google Scholar]