“Simultaneously Vague and Oddly Specific”: Understanding Autistic People's Experiences of Decision Making and Research Questionnaires

Abstract

Background:

Autism researchers often use questionnaires to gather the views and experiences of autistic people. However, questionnaires may not always be designed in accessible ways. In addition, answering questions within a questionnaire involves decision making, which some autistic people have reported finding difficult. Therefore, this exploratory study aimed to enhance our understanding of autistic people's experiences of decision making, and to analyze their feedback on questionnaire measures to further understand decision making within the research context.

Methods:

One hundred seventeen participants completed an online questionnaire. In the questionnaire, they answered an open question about what affected their ability to make decisions. They then completed four questionnaire measures and after each one provided feedback. We used content analysis to categorize participants' qualitative answers.

Results:

Participants discussed how their internal state, other people, the quality and quantity of information, pressure on choosing a response, external distractions, and lack of time all affected their decision making. Feedback on the questionnaires highlighted how questions needed context, often questions themselves were unclear and difficult to understand, that there were issues with Likert scales, and how measures could have questionable validity for autistic people.

Conclusions:

Autism researchers need to consider how they can make their research as accessible as possible for autistic people. Our study highlights how decision making is not a straight-forward process, and researchers have a role in ensuring they give their participants clear and contextualized information. Involving autistic people in the design of research is a potential way of improving the quality of research.

Community brief

Why is this an important issue?

Autism researchers often use questionnaires but might not always use well-designed ones. Its important that autistic people feel they can complete questionnaires and provide accurate information about what is important to them. Also, when answering a questionnaire, someone usually must make decisions, for example, about which option best fits their own experiences. But researchers might not consider that making decisions can be difficult for some autistic people.

What was the purpose of this study?

We wanted to find out what autistic people thought about questionnaires and what things affected their decision making.

What did the researchers do?

We created an online survey, which 117 autistic people completed. We had an open question asking participants what affected their ability to make decisions. Participants then completed four questionnaire measures that other people had created before, which had statements followed by tick boxes on different scales (e.g., strongly agree to strongly disagree). After each of these measures, we asked participants to give open feedback. We then identified patterns in the participants' responses.

What were the results of the study?

Participants' decision making was affected by things such as their mood and energy level, having to consider how their decision affects others, the quality and quantity of information provided, pressure on choosing a response, external distractions (like noise), and having to decide quickly. Participants' feedback about the questionnaire measures identified potential improvements. There were some positive comments, but participants often said the measures needed more context to explain them or needed to consider the current context (like how a global pandemic might be affecting their answers). They also said some questions were not clear and difficult to understand, the response options needed improving, or the questions were not relevant for autistic people.

What do these findings add to what was known?

Our findings show how decision making can be challenging for some autistic people, and this could influence their responses when doing research. Participants' responses give important information for autism researchers to consider when using questionnaires. Our findings are further evidence that questionnaires should be adapted or new ones created specifically for autistic people. The findings imply that it is important to involve non-academic autistic people in research.

What are potential weaknesses?

Most participants were female, White, and from Western countries, which limits how our findings might apply more widely. We asked generally about decision making rather than specifically about the research context, and more in-depth interviews on this topic would be useful.

How will these findings help autistic adults now or in the future?

The things we have learned from our study can be used by autism researchers to improve their research design. If they make improvements, this means taking part in research will be more enjoyable and straightforward for autistic people. Autism research can potentially improve autistic people's lives, but we need to make the experience of taking part in research better.

Background

It is common for autism researchers to use pre-existing questionnaire measures that have fixed responses when trying to assess or measure different concepts or experiences. For example, researchers might be interested in quantifying mental well-being,¹ sensory experiences,² or masking behaviors.³ Sometimes questionnaire measures are designed and validated with non-autistic people and then utilized with autistic people.⁴ Using standardized questionnaire measures can, theoretically, have advantages. They allow researchers to collect data quickly, which they can compare with previous data using the same measure or track a concept over time. Results from fixed responses are quick to analyze. Thus, autism researchers may often use questionnaires as they are convenient.

However, there are concerns about the use of self-reported questionnaire measures.^5,6 Vermeulen commented how terminology could be abstract, ambiguous, and lack concrete examples.⁶ Jones discusses how focusing on validity and reliability can mean researchers use measures that do not meaningfully apply to autistic people's lives.⁷ Autistic people often report that questionnaires are difficult to complete: Nicolaidis et al analyzed records from participatory academic-community partnerships about views on adapting questionnaires.⁴ They noted that autistic people experienced confusion, frustration, anxiety, and anger when completing questionnaires, which resulted in them believing the information they provided was unreliable, and often prevented them from completing the research. Other studies have noted how autistic people can be put off taking part in research if study information and design is unclear.⁸ The information gathered by Nicolaidis et al⁴ came from autistic people involved in community partnerships, and Haas et al⁸ looked generally at factors influencing research participation—we know less about the experiences of autistic people who are not involved in such partnerships, or their specific views on questionnaire measures, as we explored in this study.

Researchers have looked at specific measures and their validity for autistic people. For example, systematic reviews on questionnaires for assessing suicidality⁹ and depression¹⁰—with the tools originally developed for non-autistic people—indicated that these tools lacked validity for autistic people, and they noted the need to involve autistic people meaningfully in development or adaptation. A study of suicidality measures suggested that measurement properties were different in autistic compared with non-autistic people.¹¹ Autistic participants interpreted items differently and preferred concrete language.

Accordingly, some researchers^12–14 have worked with autistic adults to adapt questionnaire tools, although these efforts are few and far between. Adaptations include ensuring clear wording of items and instructions, adding items related to autistic experiences, visual aids, and more appropriate response options. Although this work is ongoing, it demonstrates that questionnaire measures originally for non-autistic people may need adaptation and researchers cannot assume such measures are suitable for autistic people.

When a participant chooses a response to a questionnaire item, they make a decision; therefore, this study was also interested in understanding more about autistic people's experiences of decision making. Decision making has been defined as the selection of one course of action from different possible alternatives.¹⁵ Integrating cognitive and emotional processes is required to reach a decision.¹⁶ Some prior research has suggested that some autistic people have difficulties with decision making.¹⁷ However, experimental research highlights how autistic people may be more steadfast in their choices and make more advantageous decisions.^18,19 We could view this as more thoughtful decision making, with autistic people gathering more information before making a decision than non-autistic people; thus, the decision may take longer but is made with a higher degree of certainty.^19,20

Self-reported research into autistic people's decision making suggests that some autistic participants find it difficult to engage in decision making or be fearful of negative judgements from others.²⁰ Autistic participants have reported experiencing decision-making difficulties at a higher level than non-autistic participants, although both report experiencing “frequent changes of mind” the same amount.²⁰ In addition, autistic participants report finding decisions that have to be made quickly, those requiring a change of routine, or requiring talking to another person, most problematic.²⁰ Almost 90% of autistic participants felt anxious when making decisions, and participants with higher anxiety felt that being autistic interfered with decision making.²¹

If decision making is challenging or anxiety-provoking for autistic people, then completing questionnaires could be challenging because it requires making decisions. However, to the best of our knowledge, researchers have not thoroughly explored decision making in relation to its role in research. Investigating views and experiences of research may facilitate the development of more accessible future research. Therefore, this exploratory qualitative study using an anonymous online questionnaire aimed at (1) investigating why autistic people may find it difficult to make decisions and (2) analyzing their feedback on questionnaire measures to further understand decision making within a research context. We anticipated that autistic participants would identify a range of cognitive and emotional factors that influenced decision making, and that participants would identify several issues affecting their ability to answer questionnaire items accurately.

Methods

Participants

Overall, 117 participants took part. Table 1 shows demographic information regarding gender, ethnicity, and country, showing that most participants were female, white, and living in the United Kingdom. Participants' ages ranged from 18 to 78 years (M = 43.47, standard deviation = 15.61). Ninety-two (78.63%) participants reported formal autism diagnoses, and 25 (21.32%) self-identified as autistic. Further, 63 participants (53.85%) reported experiencing mental health conditions. Of these, only 18 listed 1 mental health condition. The most frequently reported conditions were anxiety (n = 48) and depression (n = 38). Half (n = 59, 50.43%) also reported physical health conditions.

Table 1.

Participant Demographic Information for Gender, Ethnicity, and Country

	n	%
Gender
Female	64	54.7
Male	41	35.0
Non-binary	5	4.27
Queer	1	0.85
Ungendered	1	0.85
Genderfluid	1	0.85
Prefer not to say	4	3.42
Ethnicity
White British or white	97	82.9
European	15	12.8
Australian	2	1.71
Creole Caribbean	1	0.85
Jewish	2	1.71
Country
United Kingdom	65	55.6
North America	25	21.3
Australia and New Zealand	12	10.3
Other European countries	10	8.54
Prefer not to say	5	4.27

Note: For these questions, participants could self-define their own preferred terms in open textboxes.

We recruited via Facebook and Twitter. We obtained ethical approval via the University of Stirling General Ethics Panel Delegated Authority. All participants gave informed consent before commencing the survey. Data collection took place in April and May 2020, during COVID-19 pandemic restrictions.

Materials and procedure

We used an online survey, presented with the software “Qualtrics.” An autistic and non-autistic researcher designed the questionnaire together. Due to funding limitations, we were not able to fund community involvement. In the survey, participants first completed demographic questions, including age, gender, ethnicity, and country. Participants self-reported whether they had a formal autism diagnosis or identified as autistic, and then mental and physical health conditions.

Next, we asked participants whether they had taken part in autism research before (yes, no, unsure, or prefer not to say), and approximately how many autism research studies they had previously taken part in (0; 1–5; 6–10; 11–15; 16–20; 20+; other or prefer not to say). We also asked them to input how many autism research questionnaires they had completed in the past 12 months. We then asked an open question: “What affects your ability to make decisions?” Next, participants completed four questionnaire measures. To the best of our knowledge, these measures are all predominantly designed by non-autistic people and/or have come from the non-autistic psychological literature, as outlined next:

Decision making

We used a measure of decision making developed by Luke et al.²⁰ This measure consisted of 12-items related to why decision making could be difficult, for example, “I find decision-making exhausting.” Participants rated statements on a 4-point Likert scale (“this is never a problem,” “this is rarely a problem,” “this is sometimes a problem,” “this is often a problem”), with higher scores indicating more difficulty with decision making and a range of possible scores from 12 to 48. Internal reliability was very good (Cronbach's α = 0.89).

Indecisiveness scale

Indecisiveness has been defined as finding decision making discomforting in all areas.²² Within non-autistic samples, indecisiveness has been linked to factors influencing decision making, such as worrying about making mistakes and feelings of panic when needing to make inevitable quick decisions.^22,23 These issues have also been identified in autistic adults.^5,20 We used Frost and Shows²³ measure of indecisiveness, which consisted of 15 items rated on a 5-point agreement Likert scale (from “strongly agree” to “strongly disagree”). Example items included “It seems that deciding on the most trivial thing takes me a long time.” A higher score equals greater indecisiveness, with a possible range from 15 to 75. Internal reliability was poor (α = 0.52).

Autistic characteristics

We used the Ritvo Autism and Asperger Diagnostic Scale (RAADS-14)²⁴ as a measure of autistic experiences and traits. This measure includes 14 items rated on a 4-point Likert scale (“true now and when I was young,” “true only now,” “true only when I was younger,” “never true”). Example items included “I take things too literally, so I often miss what people are trying to say.” A higher score presumes more autistic traits, with a possible range from 0 to 42. Internal reliability was very good (α = 0.82).

Anxiety

We used the Generalised Anxiety Disorder Assessment (GAD-7)²⁵ to measure anxiety. This measure consists of seven items rated on a 4-point Likert scale (“not at all,” “several days,” “more than half the days,” “nearly every day”), with participants asked to think about how each item had affected them over the past 2 weeks. Example items included “worrying too much about different things.” Higher scores indicate greater anxiety experienced, with a range from 0 to 21. Internal reliability was very good (α = 0.88).

We structured the survey so respondents could provide qualitative feedback in a textbox after completing each questionnaire measure; as such, respondents could provide feedback on four occasions. For each measure, the open question said: “I am interested in your opinions on the questions you just answered about [topic, e.g., decision-making]. There are no correct or incorrect responses, and all feedback is valuable. If you would like assistance, there are prompts below to help you consider your views, but please do not limit your feedback to these prompts.” We then provided a small number of prompts to facilitate feedback, such as: “what do you think of the language?,” “how would you ask the question?”

Design and data analysis

Our aim was to investigate why autistic people may find it difficult to make decisions, and to analyze qualitative feedback to understand factors influencing decision making within a research context. First, we analyzed qualitative responses to the question “What affects your ability to make decisions?,” and second, due to thematic similarities in comments about the four measures, we analyzed all feedback on the questionnaire measures together (rather than measure-by-measure), as during data familiarization we noted similarities reported for each measure. We analyzed the qualitative responses using conventional content analysis due to limited research in this area and the method allowing for inductive data interpretations, whereby we built conclusions from the data up.²⁶

Content analysis is a method that derives meaning from qualitative data, and it is suitable for large volumes of text-based data where the researcher wishes to interpret and seek realistic conclusions from data, while also producing systematic quantification of responses.²⁷ Content analysis is not linked to any specific philosophical concept or epistemology, and we deemed it a suitable method for this exploratory study.²⁷ Our analysis followed the recommendations of Erlingsson and Brysiewicz.²⁸ The first author became immersed in the data, coding and categorizing feedback. We then merged categories into overarching categories to organize the data. The first author coded all the data into the coding framework, and comments could be coded into more than one category.

Two independent coders initially cross-checked 15% of coding to assess the validity of the coding framework. The lead author then refined the coding framework further and re-coded the data based on the independent coder feedback and discussions with the second author. A third independent coder then coded 25% of the data into the final coding framework. Inter-rater agreement ranged from 81% to 96% for the decision-making question and from 71% to 90% for the questionnaire feedback. The first author reviewed all disagreements and finalized the coding.

Researcher positionality

The first author is autistic, has found completing questionnaires challenging and after diagnosis heard other autistic people had similar experiences. This will have had some impact on the shape of the research, with some reasons for this challenge such as needing time to produce accurate, genuine responses, facilitating an inductive interpretation of the data. The second author is a non-autistic autism researcher who attempts to engage in participatory research, and regularly uses questionnaire methods in her research. She is keen to improve approaches within autism research. The first author completed this research as part of their Masters degree under the supervision of the second author.

Results

Forty-nine participants (41.90%) reported no prior autism research participation, and 6 (5.13%) were unsure. Forty-six participants (39.32%) reported participating in autism research studies on one to five occasions in the past decade (Table 2). Regarding questionnaires, 50 (42.74%) had not done any in the past 12 months, 19 participants had completed one, 20 participants had completed two to five questionnaires, 20 preferred not to say, and 8 had done six or more.

Table 2.

Frequency of Participation in Autism Research

	N	%
Have you ever participated in research about autism?
Yes	62	53.00
No	49	41.90
Unsure	6	5.13
Approximately how many autism research questionnaires have you completed in the past 12 months?
0	50	42.74
1	19	16.24
2–5	20	17.09
6–10	5	4.27
11–14	0	0.00
15+	3	2.56
Prefer not to say	20	17.09
How many autism research studies have you participated in within the past decade?
0	49	41.90
1–5	46	39.32
6–10	7	5.98
11–15	6	5.13
16–20	3	2.56
20+	3	2.56
Other (participant unsure)	1	0.85
Prefer not to say	1	0.85
No response	1	0.85

To provide additional context regarding the questionnaire measures, Figure 1 shows violin plots of the data for each. These plots show there was diversity but on average participants reported high levels of difficulty in decision making, mid-range levels of indecisiveness, high autistic characteristics, and moderate anxiety levels.

FIG. 1.

Violin plots showing participants' data for the decision-making, indecisiveness, RAADS-14 (autistic characteristics) and GAD-7 (anxiety) measures. The plots show the spread of data as well as means (white dotted line) and underlying box plots (solid white lines). GAD-7, Generalised Anxiety Disorder Assessment; RAADS-14, Ritvo Autism and Asperger Diagnostic Scale.

What affects decision making?

One hundred twelve participants provided comments for this question. We identified six categories, with sub-categories, as listed in Table 3.

Table 3.

Categories Identified for Feedback on the Factors Affecting Decision Making, Including the Number of Participants Who Endorsed the Category and the Total Number of Individual Comments per Category

Category	Number of participants endorsing category	Sub-category	Total number of comments
My internal state	57	Affect (emotion, mood, alertness)	31
		Fear of making the wrong choice	23
		Being overwhelmed or overloaded	13
Pressure on choosing the response	53	Personal limitations	19
		Type of decision	18
		Need to be prepared or thorough	11
		Lack of certainty	12
Other people	48	Influence of others	32
		Reactions of others	13
		Consequences on others	10
Quality and quantity of the information provided	39	Too many choices	19
		Clarity of information	11
		Not enough information	14
		Lack of difference between options or no “correct” answer	10
External distractions	26	Sensory information	21
		Distractions	6
Lack of time	23	Time restraints	14
		Making a quick decision	10

Note that responses could be coded into multiple categories.

My internal state

Most frequently, participants mentioned “My Internal State.” Overall, this category captured the view that emotions and affect impacted the decision-making process. Within this, participants most often commented specifically on affect, which included anxiety, procrastination, inertia, impulsivity, stress, fatigue, and self-confidence. For example, “The anxiety can cause me to feel ‘foggy headed’ which makes thinking, remembering and decision-making harder.” A fear of making the wrong choice was frequently mentioned, for example: “the fear of making the wrong decision and being responsible for [a] negative outcome.” Some participants also commented that being overwhelmed or overloaded was a factor: “Being stressed/overloaded/tired—lately I am all of those things and I find it hard to make simple choices like ‘this film or that film.’”

Pressure on choosing a response

The next most frequently mentioned category focused on “Pressure on Choosing a Response,” which reflected internal, cognitive factors that affected decision-making ability. In the sub-category personal limitations, some participants reported that their own skills, often executive functions, affected their ability to decide: “Executive function problems like poor working memory, which make it hard for me to juggle all the different factors playing into a decision in my head.”

A few participants reported that their ability to make decisions was impacted by overthinking: “I often overthink the consequences or possible outcomes of a decision, and analyse them in detail, which make it more difficult to eventually come to a decision. This applies to larger ‘life decisions' but also to more small ones.” We coded another sub-category as type of decision, which regarded if it was an important decision, small, work-related, or personal. Within this sub-category, participants commented on being good decision makers for “facts” and “specialisms” but finding it harder for decisions related to emotions: “I am really good at making decisions in my job … decisions are based on facts. Where I struggle to make decisions is around feelings.” Further, some reported that the importance of the response affected their ability and made decision making harder: “How important the outcome of the decision will be … how it may or may not affect myself or others.”

Another sub-category reflected the comments participants made about needing to be prepared or thorough to make a decision: “[I] need all the information relating to the decision, need to understand/plan for all outcomes.” A few participants reported a lack of certainty: “I cannot predict for certain what will happen either way.” This pertained to the possible consequences of decisions on self and others, particularly if they made the wrong decision.

Other people

Another frequently mentioned category concerned how “Other People” affected decision making. Within this, participants frequently commented on the influence of others on the decision-making process, with most voicing that other people made decision making harder, often because they were distracting: “people talking to me when I'm trying to think,” or because they created additional processing: “Making decisions becomes more difficult the more people are involved. For example, if I have to take into account the feelings and opinions of other people, I may think in circles and never come to a conclusion.”

Participants also commented on expectations, implied social rules, and anxiety within the subcategory of reaction of others. Here, participants reported a concern about others disliking their decision: “choosing something that might not be what everyone else wants even if choice is left up to me.” We also identified Consequences on others as a sub-category, for example, “concern about the impact of outcomes on others.”

Quality and quantity of information provided

Participants also mentioned the “Quality and Quantity of Information Provided,” where they commented on issues within the information made available to them when they were deciding. Here, participants specifically commented on there being too many choices affecting decision-making, for example, “too many choices and I want to be sure I make the right one.” Other participants reported that the clarity of information provided to them was also an influencing factor: “lack of clarity in phrasing or layout of questions.” Participants also mentioned not enough information as a challenge when making decisions, such as there being “Insufficient detail or information to facilitate a choice.” Finally, we identified a lack of difference between options or no correct answer as a sub-category mentioned by a few participants: “When there is no possible way to reason out a decision because the options are equally good/useful/liked/equal in price.”

External distractions

We coded a smaller number of comments into External Distractions, where some participants commented on how distractions made decision making harder. Often participants identified sensory information as a specific distraction: “too much interference (noise, bright or flickering lights or moving banners around the screen when on PC and people interrupting).” Five participants used the term distractions without specifying what kind of distraction, for example, “distractions that make it difficult to process my thoughts.”

Lack of time

Finally, we coded some comments under “lack of time” as they commented on how time negatively affected decision making. Specifically, some advised that time restraints were an issue: “too little time to make the decision” and some advised that having to make a quick decision was particularly pressurizing: “if I'm in a heightened state of anxiety, which is often the case when I'm put under pressure to make a decision in a short space of time.”

Questionnaire measures feedback

In answer to the open questions following each measure, 99 participants provided comments across the four measures. We identified five main categories, listed in Table 4, which also includes sub-categories and coding per measure.

Table 4.

Categories Identified for Feedback on Questionnaire Measures, Including Information on Number of Participants Endorsing the Overall Category, Total Number of Comments per Sub-Category, and Then Percentage of Comments Pertaining to Each of the Measures

Category	Number of participants endorsing category	Sub-category	Total number of comments	Percentage of comments per measure
				Decision-making	Indecisiveness	RAADS-14	GAD-7
Questions need a context	67	Items lack context	63	39.7	28.6	17.5	14.3
		Responses influenced by COVID-19 pandemic	30	0	3.33	0	96.7
		Influence of internal context	20	75	15	5	5
		Social influence	16	25	12.5	31.3	31.3
		Developmental changes to responses	12	8.33	8.33	50	33.3
		Different responses for objective and subjective decisions	13	46.2	53.8	0	0
Positive comments	60	Understandable language and questionnaire	70	42.9	24.3	17.1	15.7
		Could fit answers onto the scale	33	54.5	15.2	21.2	9.09
		Positive, general feedback	37	43.2	29.7	13.5	13.5
Questionable questions	52	Difficult, or cannot, answer questions	44	20.5	29.5	40.9	9.09
		Responses will be inaccurate and lack meaning	52	23.1	34.6	26.9	15.4
		Not understanding concepts, words, questions	10	20	20	20	40
Problem with Likert scales	41	Scale makes it difficult or not possible to respond	45	22.2	17.8	53.3	6.67
		Want to provide more detailed answer	29	41.4	17.2	31.0	10.3
Questionable validity for autistic people	37	Question misses important information	48	27.1	20.8	29.2	22.9
		Questionnaire is not applicable to autistic people	19	31.6	26.3	15.8	26.3
		Impact and intensity of measured concept	14	42.9	21.4	21.4	14.3

Note that responses could be coded into multiple categories.

GAD-7, Generalised Anxiety Disorder Assessment; RAADS-14, Ritvo Autism and Asperger Diagnostic Scale.

Questions need a context

The most frequently mentioned category was “Questions Need a Context.” This category captured the view that providing context was crucial—context refers to circumstances, settings, and provides information for understanding. Most often, these comments pertained to the decision-making and anxiety measures. There were six sub-categories. First, most comments focused on items lacking context, with participants' feedback suggesting that context was “crucial” and without any, the answer to questions was “it depends.” For example, participants viewed some concepts (particularly decision making itself) as “vague,” and the feedback suggested that responses depended on a variety of factors including role and situation: “When I am asked about how I manage a generic situation there are a hundred different possibilities.”

The next most mentioned subcategory was around the fact that responses were influenced by the COVID-19 pandemic. We collected data during the spring of 2020 and all but one of these comments was after the anxiety measure. Participants believed the pandemic to have altered answers on this measure considerably, with respondents reporting increased anxiety. A respondent stated their answers were: “Extremely atypical compared to what they would usually be.”

Another subcategory was influence of internal context. Here, respondents reported how psychological aspects, such as emotional processing, state of mind, confidence, and level of concentration, could influence their responses: “A lot of these, if not all, would depend on my state of mind at the time, which fluctuates a lot sometimes, others not. I answered with an average mood in mind.” Some mentioned the subcategory of social influence as affecting their responses, for example, the presence of others and the effect of masking on behavior: “As an adult I have learnt to hide most of what I am feeling to enable me to carry out tasks and roles. This wasn't taken into account by the questionnaire.”

We coded a few participants' responses into the subcategory of developmental changes to responses, whereby they fed back that their answers changed over time, particularly in relation to getting older: “I am currently retired and don't feel nearly the same degree of anxiety I used to when I was working. My responses would have been very different if I had responded to the questionnaire ten years ago.”

Finally, we coded a small number of responses into the subcategory different responses for objective and subjective decisions. Here, feedback indicated that responses could vary depending on whether they were thinking about something more objective or subjective. For example, some reported difficulties with questions requiring an answer that was based on their feelings (this being more subjective), whereas they viewed work-related decisions as easier than personal decisions (seeing this as more objective). These contextual differences made it difficult for respondents to respond to some items: “The questions were ambiguous in that sense as I am one way for personal decisions and the other [way] for others.”

Positive comments

The next most reported overall category was “Positive Comments,” where participants provided good feedback on the measures: “This questionnaire was clearly worded and easy to understand and was fast for me to fill out.” Around 23% of comments overall were positive, and most often in response to the decision-making measure. Specifically, within this we coded responses into subcategories of understandable language and questionnaire, where the participants thought the language used for the items, scale, and overall measure had been “clear,” “appropriate,” “specific” and the questionnaire overall was “easy to follow” or “These seem like good questions and they're clear and unambiguous.”

Some participants also noted that they could fit answers onto the scale, suggesting the scales were valid for them. Finally, we coded some responses into the subcategory of positive, general feedback, where participants had often used short statements or words such as “fine,” “useful,” “relevant.”

Questionable questions

The next most frequently cited category, however, focused more on negatives, particularly “Questionable Questions.” Within this category, respondents had questions and criticism about the wording and intent of items, resulting in difficulties providing a response, and mentioned this regarding all measures but particularly the indecisiveness and RAADS-14 measures. We categorized many responses under difficult, or cannot answer, questions. Here, respondents talked about not remembering enough examples from their lives to answer the question, questions being too generic, and multi-part questions with the response differing for each part: “How to make friends is no problem, how to keep them is a totally different story! I would differentiate/nuance that question.” Participants reflected on the assumptions within items, such as assuming respondents have a problem with a concept (e.g., impulsivity), or the starting point for a decision: “The question about the menu implies that all options are possible for me. Life as an autist is NOT like that. When I go to a restaurant, I know that I will have a burger.”

Participants also mentioned the subcategory of how responses will be inaccurate and lack meaning. Here, participants considered some words or phrases too vague when they needed precise definitions, so they knew what the researcher was referring to. This appeared particularly true for adverbs: “what is meant by extremely? I have written never true, because I do not feel that I am extremely upset by change, but although I do not feel upset, my body will show symptoms of stress and I may also experience hyperactivity. So whilst I am not extremely upset (by my definition), I am affected.”

Finally, within Questionable Questions we categorized some comments as participants reporting not understanding concepts, words or questions. Here, respondents were unclear about the specific definition of words (e.g., decision), and their criteria (e.g., what ages represent “young”?). Participants considered certain words vague, with questions worded unhelpfully and being “difficult to process.” There were comments about how to interpret the questions: “The wording of the questions … both simultaneously vague and oddly specific at times which confuses me. I don't know whether I should be reading them pedantically or if I should be trying to read something else into them as is so often expected by neurotypical people.”

Problems with Likert scales

Another overall category centered specifically on “Problems with Likert Scales,” and participants most often mentioned this in relation to the RAADS-14. Within this, most often participants talked about the subcategory of scale makes it difficult or not possible to respond. The feedback here reflected a struggle to answer because the scale did not provide a response that reflected the participant's reality. In particular, the feedback challenged the RAADS-14 scale, with respondents wanting a “sometimes” response: “My main problem with the scale was that to some of the questions I would reply that it only affected me some of the time, not all of the time, however this was not a response I could give.” Respondents commented on how scales required a response that was too definite: “[it asks for] very absolute answers which do not necessarily feel 100% accurate.” They also commented on imprecise language in scales: “no clear boundaries between categories.” The language could result in participants doubting whether their answers were accurate, for example: “[its] hard for me to decide where my answers fit, I could easily get bogged down in trying to define each part of the scale (how strong is ‘strongly’ vs ‘somewhat’?).”

Participants also mentioned the subcategory of wanting to provide a more detailed answer. They commented on how scales were “restrictive,” “limited” and there were not enough response options: “the scale felt too small—not enough choices of response.” Space to explain quantitative answers or provide a qualitative response instead was desired: “I felt that some of the options were too black and white. I would have liked to have been able to add more nuance and background explanation to some of the questions in a comments or additional feedback section.”

Questionable validity for autistic people

The final category we named “Questionable Validity for Autistic People” because feedback suggested the measures were not valid for measuring the concepts in autistic people, with these comments made about all measures. Most often we categorized participants' answers under the subcategory question misses important information. Comments suggested the questionnaires were not sensitive to change, nuance, and the environment: “It is widely known that autistic sensitivities can fluctuate day to day or depending on circumstances and stress level. Therefore, though I answered the questions as best I could, there will be a certain level of inaccuracy.”

Some talked about how the questionnaire is not applicable to autistic people, for example: “I get cross … the researcher doesn't seem to understand autism enough to generate questions and frameworks that work.” Finally, a few talked about the impact and intensity of measured concept. For example, when talking about the answers to the anxiety measure, one participant noted, “The symptoms described only occur a minority of days … but they are utterly disabling when they do occur.”

Discussion

We investigated why autistic people might find decision making difficult and their opinions when taking part in questionnaire research. By investigating their views, it may be possible to improve autism research and increase the appeal of participation among the autistic community. This study provides valuable information about what affects autistic people's decision making and highlights useful feedback on the experience of completing questionnaires. Doing questionnaires involves decision making, and therefore it follows that information about what impacts decision making will have implications for the design of questionnaires for research purposes.

What affects decision making?

Our first aim was to investigate why autistic people may find it difficult to make decisions. Our content analysis supported prior research suggesting that some autistic people could find decision making challenging,^17–20 and indeed scores on the quantitative measure suggested high levels of decision making difficulty. The most reported reason for decision-making difficulty was “my internal state,” where participants commented on how anxiety and being overwhelmed could affect decision making. This finding supports literature showing how anxiety and burnout are high for autistic people.^29–31

Participants also mentioned how feeling pressurized to choose a response negatively impacted decision making. They specified how they felt some of this pressure was from other people. This finding aligns with Luke et al's²⁰ study, who suggested that the involvement of others makes choosing a correct answer more complex, because it requires a decision about how much to consider the reactions and consequences on others. Arguably, this consideration of other people counters claims that autistic people are not socially interested or concerned about what others think³² or lack a “theory of mind”³³ and instead shows acute awareness of other people when it comes to decision making.

Participants also mentioned how the quality and quantity of information affected their decision making—this could relate to autistic styles of processing information. A processing style focused on local details (rather than the wider picture) and enhanced perceptual processing³⁴ may mean it could become overwhelming when there are lots of details to process during decision making. Participants mentioned how external, and particularly sensory, distractions negatively impacted decision making. Sensory aspects of the environment are known to affect the everyday lives of autistic people,³⁵ thus it makes sense that this could affect decision making, and where feasible (e.g., for in-person research), researchers should consider how they can provide a distraction-free environment for participants.

We can perhaps tentatively apply these findings concerning general decision making to the research context and more broadly to contexts where clinicians are using screening or clinical tools that are questionnaire-based. Researchers should design questionnaires in a way that provides clear and specific information, and not present too many questions, thus addressing our participants' point regarding how the quality and quantity of information provided can affect decision making. Nicolaidis et al⁴ make specific suggestions such as adding short explanations before questions to explain the content, using simple sentence structure and plain language. Researchers may have to let go of feeling that they must stick with “valid” and “reliable” pre-existing questionnaire measures and be open to adaptations and developing new tools, although this is often constrained by funding bodies.⁷

Questionnaire feedback

Our second aim was to analyze participants' feedback on questionnaire measures to further understand decision making within a research context. We identified core issues in terms of questions needing a context, questionable questions, problems with Likert Scales, and questionable validity for autistic people. Notably, there were many positive comments (23% of comments), and interestingly most often in reference to the decision-making measure, even though participants critiqued this measure in other areas. This suggests that researchers should still consider various improvements when using questionnaires, as even though a measure may be suitable for some people, it should be suitable for as many as possible within a well-defined target population.

The most commented on issue was how questions need a context, and this particularly related to the decision-making measure. This does not support experimental studies on decision making, which concluded that autistic people have reduced context-sensitivity compared with neurotypicals.^19,36,37 In contrast, our participants considered context crucial. Two of the questionnaires regarded aspects of decision making and our participants thought this term was vague, and they were uncertain what kind of decisions the questionnaire wanted them to consider. As suggested by Nicolaidis et al,⁴ providing a brief context could help, for example, by defining key terms before presenting questionnaire items.

More broadly, the COVID-19 pandemic was a change in a wider context that had a considerable impact on participants' responses to the GAD-7. The responses matched comments elsewhere about the experiences of autistic people during COVID-19.³⁸ Our participants advised that their responses reflected increased levels of anxiety due to the pandemic. One participant suggested that it was more “normal” to be afraid of something awful happening, rather than a response related to anxiety. Researchers, thus, need to be cognizant of external contexts influencing participants, perhaps by including open textboxes for participants to note whether they feel anything may have affected their responses.

Participants also frequently discussed “questionable questions,” which further supports research commenting on the importance of using concrete and clear language in questionnaires^10,12 as well as our findings from our general question about decision making. Feedback suggested that participants paid careful attention to the meaning of words, and at times were unable to respond accurately and in a meaningful way, and these comments tended to apply across all measures.

This is similar to Vermeulen's⁶ example of how an autistic person might strongly disagree with the statement “I am intensely interested in other people” because they are not intensely interested in all people. If researchers are unsure of the wording of their questionnaire, piloting their questionnaire with autistic people could help ensure accessibility. Indeed, our participants talked about how all the measures could have questionable validity for autistic people. It should be noted that this category had the lowest inter-rater reliability (71%), and thus should be interpreted with caution. However, it is still important that questionnaires are not only accessible but also valid for autistic people. Jones⁷ points out a need for autism researchers to think carefully about whether they are measuring what they think they are measuring. By using participatory methods, gathering autistic feedback on a questionnaire's design could help enhance its quality, relevance, and validity.³⁹

Finally, participants mentioned problems with Likert scales. Likert scales are a common feature of questionnaires, but our participants felt these scales did not fit with their experiences, or they wanted to provide more information. Particularly these comments related to the RAADS-14, but participants mentioned this for all measures. The RAADS-14 scale may be problematic, as it uses a time-based developmental scale that adds an additional layer of consideration and context.

This finding further calls into question the validity of questionnaires, if participants feel they are not able to respond appropriately. In addition, the indecisiveness scale quantitatively had poor internal consistency, which suggests that it would also benefit from careful adaptation (perhaps using our findings to influence how it is adapted). Using visual/picture-based scales alongside the standard Likert scale, providing a context and open textboxes for scale feedback could all help here.

The study did have limitations. Most participants were female, white, and from Western countries. Lack of representation of Black autistic people in autism research is deeply concerning—a scoping review of 77 years of autism research about autistic women and girls found that only three papers focused on Black autistic women and girls.⁴⁰ Jones and Mandell⁴¹ discuss the need to support Black scientists and to listen to the Black autistic community, and this could increase the participation of Black autistic people within research. Research with Black families of autistic children highlighted barriers to participation such as distrust of researchers, lack of time, and inaccessible materials.⁴² Other limitations include the fact that the responses may only apply to the measures used in this study; however, the participants did repeat similar concerns across the measures and some issues (such as needing context and disliking Likert-scales) would be widely applicable to other measures.

We also broadly asked about decision making rather than specifying within the research context, which may mean participants would have different responses if we asked more directly about research-based decision making. Finally, we collected data via a questionnaire with open textboxes, which limits a more in-depth approach—thus, further interview-based research would be useful to follow up our findings.

Despite these limitations, we believe research on research is important. Our participants' responses corroborate previous work and additionally link in challenges associated with decision making to questionnaire research. Gathering and listening to autistic people's feedback on research should serve to improve autism research and enhance trust within the autistic community toward autism research. Autism research can and should meaningfully contribute to autistic people's lives, but if research design is inaccessible this limits its potential. We hope that our research will stimulate researchers to think carefully—and listen thoughtfully—when they do their next questionnaire study.

Authorship Confirmation Statement

R.S. conceived of the idea for the study as part of their Master's dissertation at the University of Stirling, which was supervised by E.C. R.S. designed the survey with support from E.C. R.S. recruited participants, analyzed the data, and wrote the findings up as their dissertation. E.C. significantly edited the dissertation into the manuscript for submission. All authors contributed to the final manuscript and approved the final version.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

The authors received a small financial contribution from the Department of Psychology, University of Stirling, for the study prize draw.