Inclusive approaches to developing content valid patient-reported outcome measure response scales for youth with intellectual/developmental disabilities

Abstract

Background:

Patient-reported outcome measures (PROMs) are increasingly used in health care to evaluate service quality and client progress. Response scales are a critical component of PROM content validity and must be designed to be relevant and comprehensible by users.

Methods:

In collaboration with eight youth co-researchers with intellectual/developmental disabilities ages 14–21, we used an iterative, three-stage approach to develop and select a response scale for the PEDI-PRO. Stages: 1) inclusive development of response scale options; 2) Collecting data about response scale options during focus groups with youth with intellectual/ developmental disabilities (n = 62); and 3) Analysing data to refine response options.

Results:

Through two cycles of the three-stage process, the inclusive research approach led to the development of a content valid response scale that describes functional performance of everyday activities (“very easy,” “a little easy,” “a little hard”).

Conclusion:

An inclusive research approach can support the development of content valid PROM scales. We identified four broad strategies that supported youth co-researchers to engage in this response scale development process: universal design for learning, use of lived experiences, breaking down tasks, and peer support. Researchers may adopt and/or adapt the accessible inclusive research approaches described in this manuscript for measurement development and other research projects.

Accessible Summary

• People with disabilities have the right to use patient-reported outcome measures (PROMs) to tell their doctors and therapists what they think and feel. Therefore, PROMs need to be accessible for people with disabilities. This means that the questions and the response choices for the questions need to be easy to understand. Response choices can be words like, “yes” and “no,” or “agree” and “disagree,” or “easy” and “hard.”

• To make sure that the response choices are easy to understand, it is a good idea to collaborate with people with disabilities when designing PROM response choices, or response scales.

• We describe how we worked with 8 young people with disabilities to make an easy-to-use response scale for a PROM.

• After our data collection and work together, we decided the best response scale was “very easy,” “a little easy,” “a little hard.”

• This research is important, because other researchers and people making PROMs can use our process to help make their response scales easy to use.

1 |. INTRODUCTION

Healthcare systems increasingly use patient-reported outcome measures (PROMs) to provide users with opportunities to contribute to healthcare decision-making and evaluate outcomes of care. Yet, youth with intellectual and/or developmental disabilities have been underrepresented in PROM development. Disability studies scholars describe how the historical exclusion of youth with intellectual/developmental disabilities in research is based on the assumption that they cannot provide trustworthy information due to cognitive impairments (Walmsley & Johnson, 2003). However, policies and frameworks such as the UN Convention on the Rights of Persons with Disabilities, the Americans with Disabilities Act, and the social model of disability have increased the awareness of the rights of youth with intellectual/developmental disabilities to exercise self-determination in healthcare settings. While there are several examples of involving youth with intellectual/developmental disabilities in the development of PROM items, there has been less involvement of youth with intellectual/developmental disabilities in the development of response scales.

Response scales play a central role in ensuring the content validity of a PROM. Response scales use Likert categories or other ordered series of words, images or numbers (Terwee et al., 2018). The first component of content validity, relevance, requires that response options align with the way respondents conceptualise the construct or outcome of interest. For example, if youth conceptualise their responsibility for everyday tasks in terms of their ability to complete the task, rather than the support they receive, then the response scale should reflect degree of task completion (Kramer & Schwartz, 2017a). The second component of content validity, comprehensibility, ensures the response options are interpreted in the intended manner and includes two criteria (Terwee et al., 2018). First, the PROM response options must be understood as intended. The process of interpreting and matching a response category to one’s self-evaluation poses multiple cognitive demands that can be challenging for youth with intellectual/developmental disabilities with cognitive impairments (Kramer & Schwartz, 2017a). When youth are not able to meet these demands and misinterpret the response scale, the validity of the PROM is threatened. Second, the response options must grammatically and conceptually match all items in a PROM in a logical manner. For example, in a quality of life measure, the response options “I can” and “I can’t” match items related to performance of meaningful activities (e.g. “Walk between rooms at home,” “Talk with friends”), but would not match items about symptoms (e.g. “Pain prevents me from doing activities I enjoy”).

This paper illustrates how we used an inclusive research process to design content valid PROM response scales for youth with intellectual/developmental disabilities. Inclusive research is characterised by the involvement of persons with intellectual/developmental disabilities in all phases of research including the design of the research materials, data collection and data analysis (Walmsley & Johnson, 2003). Thus, this approach is responsive to the historical exclusion of people with disabilities in research; in addition, the unique contributions of co-researchers with intellectual/developmental disabilities, can result in PROMs that align with the unique lived experiences and ways of knowing of youth with intellectual/developmental disabilities (Stack & McDonald, 2014; Mirra et al., 2016), thus elevating their voice in healthcare settings. We describe our 3-year collaboration (Kramer & Schwartz) with eight youth co-researchers with intellectual/developmental disabilities to develop a new PROM of functional performance of everyday activities, the Pediatric Evaluation of Disability Inventory-Patient Reported Outcome (PEDI-PRO). We have described our overall inclusive research approach to developing the measurement conceptual framework and ensuring the accessibility of items in other manuscripts (Kramer & Schwartz 2017b, 2018). The PEDI-PRO assesses the functional performance of discrete functional tasks in the context of everyday life situations in three domains: Daily Activities, Social/Cognitive, and Mobility. This manuscript uniquely describes the process of designing PROM scales. We also discuss four theory-driven strategies that supported youth with intellectual/developmental disabilities to engage in PROM development, in alignment with the disability rights movement’s motto, “nothing about us without us.”

1.1 |. Inclusive research process for PROM response scale development

We used an iterative approach to develop and evaluate the content validity of the PEDI-PRO response scales (Figure 1) using methods aligned with COSMIN quality standards (Terwee et al., 2018). The full inclusive research team engaged in three stages of response scale design: develop potential response options, collect data about options in focus groups, and analyse data to refine response options. This process was repeated twice (referred to as Cycle 1 and Cycle 2 in this manuscript). We describe each stage and the shared and distinct roles of the co-researchers and academic researchers. Following the development process described in this manuscript, the content validity of the resulting response scale was evaluated in a cognitive interview study, reported elsewhere (Kramer & Schwartz 2017b).

FIGURE 1.

Iterative, inclusive process to design and evaluate the PROM response scales and alignment with COSMIN content validity criteria

1.1.1 |. Academic researchers’ values and assumptions

Researchers’ values and assumptions drive their methodological choices and are necessary to reflect upon, due to the power imbalances between researchers with disabilities and those with academic training (Zarb, 1992). The academic researchers (first and second authors) both have an educational background in disability studies and occupational therapy, and their desire to include youth with intellectual/developmental disabilities in PROM development was driven by values regarding individuals’ rights and lived expertise. As espoused by the disability rights movement and the United Nations Convention of the Rights of Persons with Disabilities, these academic researchers believe youth with intellectual/developmental disabilities have the right to be involved in all matters concerning them, including development of PROMs designed to measure youth with intellectual/developmental disabilities’ healthcare outcomes.

The academic researchers’ assumptions about the origin of disability also informed the strategies they used to partner with youth with intellectual/developmental disabilities in PROM development. A dominant assumption about disability is that bodily differences lead to impairments in communication, cognition and motor skills that make it difficult for youth with disabilities to complete complex tasks. Alternatively, the social model of disability posits that incompatibilities between an individual and their environment lead to difficulties completing complex tasks. The social model of disability suggests the exclusion of individuals with impairments from research is due to physical, social and policy environments that fail to provide access to the research process, rather than any inherent incompetency (Oliver, 1996). Thus, aligned with the social model of disability, our team’s strategies focus on adapting various aspects of the research environment to be compatible with the abilities of youth with intellectual/developmental disabilities.

2 |. METHODS

2.1 |. Research team

The research team included two academic researchers, including the study PI (second author), a co-investigator (first author, a graduate student at the time of the study), and eight youth co-researchers with intellectual/developmental disabilities (see Kramer & Schwartz, 2018 for additional details; Table 1)¹. This team was formed specifically to develop the PEDI-PRO in 2013. The full team began meeting immediately after the PI was awarded funding, and as of writing, several team members have continued to remain involved across multiple grants to complete this work. The full team met 12 times in a 33-month development period. We also met individually or in small groups with co-researchers; combined, the eight co-researchers contributed 246 hr of work to the development of the PEDI-PRO as reported here and in Kramer and Schwartz (2018).

TABLE 1.

Co-researcher characteristics

	% (n)
Average age at recruitment (years)	17.8 (14.8–21.6)
Female	50.0% (4)
White	62.5% (5)
Non-white	37.5% (3)
Primary disability
Autism	12.5% (1)
Cerebral palsy	12.5% (1)
Down syndrome	25.0% (2)
Intellectual disability	25.0% (2)
Spina bifida	12.5% (1)
Other developmental disability	12.5% (1)
Duration of involvement
>2 years	75.0% (6)
<2 years	25.0% (2)

Walmsley and Johnson (2003) describe five criteria for inclusive research. This project met some of these criteria, in that the work was aligned with the priorities of people with disabilities to have a more active voice in their health care, it was collaborative, and the research question, processes and reports (conference presentations, rather than the present manuscript) were accessible to people with disabilities. We note that the research problem was owned by the academic researchers. As academic researchers and co-researchers shared decision-making power, the way our team worked together may be classified as a “collaborative group” under the typology proposed by Bigby and colleagues (2014b). This manuscript may be described as primarily the academic researchers’ reflection, but with quotations from co-researchers (Strnadová & Walmsley, 2018). To contribute to this manuscript, co-researchers engaged in a reflective phone interview about their experience with rating scale development (and are quoted, below). In addition, their intellectual contributions greatly influenced the work and subsequent conclusions reported in this article. Although all co-researchers were given the opportunity, only two co-researchers elected to review the final manuscript, with support from the first author.

2.2 |. Ethical considerations

All methods received IRB (Institutional Review Board) approval. During the consent/assent process, the academic researchers explained the study and reviewed a plain language consent/assent form that included images. When applicable, parents/guardians provided permission for study participation.

2.3 |. Stage 1: Developing response scale options

The objective of this stage was to ensure the relevance of the PROM response scales by ensuring they align with youth with intellectual/developmental disabilities’ understanding and experiences of functional performance of everyday activities. The co-researchers completed a multi-step process to design response scales that were grounded in their lived experiences of performing everyday tasks (the construct the PEDI-PRO was designed to measure). First, co-researchers individually reflected on their performance during “field trips” in which they engaged in self-identified meaningful activities. For example, when eating at a restaurant, they identified functional tasks, such as using a touch screen at an ATM, asking a store employee for help, and cutting food with a knife and fork. After completing their identified meaningful activity, co-researchers reflected on the prompt, “If your parent or teacher asked you, ‘how did it go’, what would you say?”

Second, co-researchers learned how to construct response scales with ordered categories. The co-researchers manipulated 3D cubes of different sizes and matched the cubes with words such as “small,” “medium,” and “large.” Through this activity, co-researchers became familiar with how ordered categories are used to represent magnitude of a construct.

Third, each co-researcher designed their own response scale(s) describing functional performance. Academic researchers wrote the words and phrases generated in the experiential reflection activity (described above) on individual notecards. The academic researchers generated additional words and phrases related to performance (e.g. good, bad), difficulty (e.g. hard, easy) and magnitude (e.g. very, a lot). Co-researchers selected from these choices and arranged notecards into ordered scales. They started by selecting “anchors” for the response scale by identifying “opposites” (e.g. “hard” and “easy”) and then added additional words to develop 3-, 4- and 5-point response scales. In total, the co-researchers designed 11 potential response scales to describe the quality or experience of performing functional tasks on the PEDI-PRO.

We used a collaborative approach to select five of the 11 potential response scales for future evaluation. The academic researchers drew upon their experience and knowledge of measurement quality to identify response scales they thought had the most potential to produce acceptable psychometric properties. For example, they initially avoided scales with an odd number of response categories, as research suggests such scales are less likely to detect differences within and across respondents (Kulas et al., 2008). From the remaining scales, the co-researchers voted for the five responses scales they wanted to further explore in the first focus groups (Cycle 1, Stage 2).

2.4 |. Stage 2: Collecting data about response scale options

The objective of this stage was to determine if response scales were easy to understand (comprehensibility) and could be used with youth with intellectual/developmental disabilities to report their functional performance of everyday activities (relevance). To achieve this objective, academic researchers and co-researchers co-conducted focus groups to identify the optimal response options.

2.4.1 |. Focus group participants

We held focus groups with 62 additional youth with intellectual/developmental disabilities over two iterative development cycles: the first cycle involved three focus groups (n = 19) in which youth provided feedback on five potential response scales. The second cycle included six focus groups (n = 43; additional details below).² This manuscript refers to youth who engaged in focus groups as “focus group participants.”

2.4.2 |. Procedures

The academic researchers drew upon their expertise in qualitative methods with youth with intellectual/developmental disabilities and PROM development to propose focus group procedures to the co-researchers. The co-researchers then trialled and revised the procedures (see Table S1) for use with focus group participants as described here.

To evaluate response scales, all focus group participants answered five items about common functional tasks (e.g. “Tie my shoes,” “Use a GPS or smart phone to get to the restaurant”) using the different response scales; all five items were answered using one response scale at one computer station, and then, the participant moved to a second computer station to complete the same five items using another response scale. This continued until each focus group participant trialed all response scales. Using a computer-based approach increased accessibility for participants, as it allowed us to incorporate large font and text to speech. Presentation of scales was counterbalanced across focus group participants to reduce order effects (Gaito, 1961).

Second, each focus group participant completed a worksheet to provide open-ended information about their preferences for each response scale. Participants identified their most and least favourite response scale and why specific scales were easy or hard to use. Participants received support from academic researchers and co-researchers to complete the worksheet. Participants then shared their perspectives about each response scale with the group during a facilitated discussion.

Third, to reduce the impact of social influence on reported preferences, focus group participants used a set of private ballots, with a picture of each response scale, to vote for the preferred response scale. Based on the co-researchers’ concern that it was difficult to select one favourite scale, participants voted for up to two favourite scales and voted for one least favourite scale. We scaffolded decision-making by supporting focus group participants to sort, stack or discard ballots based on their preferences.

Across both cycles of focus groups, co-researchers co-facilitated seven focus groups (Kramer & Schwartz 2018). The co-researchers’ experiences trialling and refining the focus group activities supported them to implement the activities. Because the co-researchers had shared experiences as young people with disabilities, they were able to provide unique social support to focus group participants (Mirra et al., 2016). Co-researchers helped individual participants use the computer, read and respond to the questions on the worksheet, place votes in the ballot box, and provided encouragement. Aligned with the reflections of Bigby and colleagues (2014a), some co-researchers commented on how their relationships with participants in focus groups may have increased the comfort of participants, “they may [feel], ‘oh yeah, [co-researcher’s name], I know this guy”; “I liked the fact…I was with people I used to go to school with, so that made it more comfortable and easier during the process and explaining things to them.” The academic researchers took primary responsibility for facilitating the discussion in the focus groups and supporting co-researchers to perform their role in data collection. Due to scheduling conflicts, academic researchers had to facilitate two focus groups without co-researchers.

2.5 |. Stage 3: Analysing data to refine response options

The objective of this stage was to identify focus group participants’ preferred response scale, with the assumption that the most preferred response scale would be the most relevant and easy to use by a wide range of youth with intellectual/developmental disabilities. The co-researchers engaged in a multi-step process to analyse focus group participants’ votes and qualitative feedback about the response scales. Co-researchers valued analysing data, stating, “the data helps in research…because we could know what the problem was…see how we can fix, how we can change the research”; “so that we can know…how many people liked the self-report or how many people didn’t like the self-report.”

First, three co-researchers met individually with the academic researchers to prepare the initial code list. These co-researchers viewed each qualitative response from focus group participants in a table format; for each response, they identified a one word “main idea.” Next, the co-researchers highlighted similar main ideas in the same colour to develop codes. They identified the codes: words, images and choices (Figure 2). Using co-researcher-identified codes ensured the academic researchers were not imposing their analytical schemes on the analysis process.

FIGURE 2.

Use of colour coding to support youth to content code qualitative feedback of the rating scales

Second, the full team of co-researchers used these codes to analyse qualitative data about each response scale. The process of analysis was broken into multiple steps to reduce working memory demands. First, pairs of co-researchers received notecards with the qualitative responses (i.e. quotes) about one response scale. The co-researchers sorted each notecard into the codes “pictures,” “words,” or “choices.” The three co-researchers who had engaged in initial qualitative coding were able to support their peers to organise the data into these codes. Second, co-researchers further analysed data within each individual code using a structured worksheet to identify “positive” and “negative” aspects of the response scale (Figure 3a). This process was repeated for all codes and synthesised on a large poster (Figure 3b).

FIGURE 3.

(a) Example analysis of data about the response scale “words.” (b) Example of completed qualitative analysis “poster” for one response scale. (c) Co-researchers’ integration of qualitative data and voting results from focus groups

Third, co-researchers integrated the qualitative analysis for each scale with voting data. They counted and recorded the number of favourite and least favourite votes for each scale. Co-researchers wrote the main ideas from the qualitative analysis next to the voting results, so they could view both results simultaneously (Figure 3c). Co-researchers then voted for the response scales to carry forward to cognitive interviewing, and discussed their rationale for their votes.

2.6 |. Iterative development: Repeating stages 1–3

As depicted in Figure 1, we completed the development process twice. In Cycle 2, Stage 1, we worked together to revise the response scales. The academic researchers drew upon their measurement knowledge and other emerging results (Kramer & Schwartz 2017b) to suggest using response scales with three response categories (see Brooks et al. (2013) for another example in which an inclusive research process led to the reduction of a response scale from 5 to 3 points). The academic researchers returned to the 11 initial response scales designed by the co-researchers and the results of the Cycle 1, Stage 3 to generate potential 3-point response scales. Then, the academic researchers and co-researchers worked together to refine these scales and select four, 3-point response scales to evaluate in Cycle 2, Stage 2, data collection, with an additional six focus groups to ensure we gathered a broad range of perspective on these final response scale candidates (n = 43; see (Kramer & Schwartz 2018 for participant details). These focus groups followed the same procedures as described in Cycle 1, above. The Cycle 2, Stage 3 analysis of the focus group results followed the same procedures as described in Cycle 1, above. The final scale selected for use in cognitive interviews was, “very easy,” “a little easy”, “a little hard.”

A large body of research has explored how individuals with intellectual/developmental disabilities respond to self-reports, including their use of responses scales (e.g. Hartley & MacLean, 2006). Some of this research suggests individuals with intellectual/developmental disabilities have stereotypic response patterns, including selecting the highest response option for all items (Finlay & Lyons, 2002). Scholars have suggested this is due to the cognitive demands of using the response scale and the social demands of self-reporting (Finlay & Lyons, 2002; Fujiura, 2012). Our inclusive process helped us deeply explore how youth interpret and respond to response scales, in addition to how they feel about discussing activities that are hard for them with assessment administrators. For example, although youth comprehended response scale options, such as “hard,” and “I can’t,” they reported feeling negatively about using these words to describe their performance. They also shared their negative associations with images such as “thumbs down,” sad faces and colours (e.g. “red” indicating something negative). Co-researchers reflected on and incorporated this feedback to develop response scales that are weighted towards “positive” response options, such as, “Very easy,” “a little easy,” and “a little hard.”

2.7 |. Evaluating content validity of response scales

After identifying the final response scale, academic researchers and co-researchers co-conducted³ cognitive interviews using the response scale, “very easy,” “a little easy”, “a little hard” with 37 youth with intellectual/developmental disabilities, ages 8–22 (mean = 19). Results are reported elsewhere (Kramer & Schwartz 2018). After cognitive interviews, academic researchers evaluated the comprehensibility of the response scale by calculating the percent of response scale choices that were aligned with qualitative descriptions of functional performance. As reported, youth selected “very easy” 94% of the time they described positive performance and selected “a little hard” 73% of the time they described negative performance. This pattern suggested youth were able to interpret the response scale as intended to describe their performance. The academic researchers shared these results with co-researchers. The co-researchers contributed to the refinement of the visual aesthetics of the final response scale prior to large-scale testing and psychometric evaluation (Kramer et al., 2021).

3 |. DISCUSSION

Through an iterative process, we used an inclusive research approach to develop and evaluate response scales for the PEDI-PRO. Additional research suggests that the scale developed and selected as an outcome of this process has strong content validity for youth with intellectual/developmental disabilities and acceptable measurement properties (Kramer et al., 2021). To support other PROM developers to replicate this inclusive approach, we discuss four broad theory-driven strategies that guided our process: universal design for learning, use of lived experiences, breaking down tasks, and peer support.

3.1 |. Universal design for learning

Universal design for learning (UDL) anticipates user variability in the design of materials to ensure all individuals, regardless of specific abilities and needs (Rose et al., 2006), can engage in the learning process. UDL posits that when youth are provided with multiple modalities for learning and communication, they are more likely to be able to demonstrate their strengths and knowledge. In our study, all activities and tasks were designed with UDL in mind. When using written materials, we supported co-researchers with a range of literacy skills by using familiar words, supplementing text with concrete images, and using colour to convey information (e.g. green for “favorite” and red for “least favorite”). For expressive tasks, such as developing response scale options, co-researchers could share their ideas in multiple ways, including writing, typing, speaking, drawing or selecting images that represented their thoughts or feelings. As described, we supported co-researchers’ understanding of new concepts (e.g. response scales) and their ability to compare and contrast ideas (e.g. analysing qualitative data about response scales) by using manipulative representations of data and key ideas. Additionally, UDL recognises the centrality of motivation when learning new tasks (Rose et al., 2006); we supported co-researchers’ abilities to engage in novel research tasks by explicitly linking the research to their goals and interests, including engagement in meaningful activities. The co-researcher’s positive response to UDL strategies during the research process informed the design of the response options, and the final selected scale included images to optimise comprehensibility. Our use of multiple modalities is aligned with that of other inclusive research teams and a recent consensus statement (e.g. Frankena et al., 2019; O’Brien et al., 2014).

3.2 |. Use of lived experiences

Experiential, constructivist and ecological theories of learning all suggest experiences “are the basis for observations and reflections” that form a foundation for comprehension of abstract concepts (Kolb et al., 2001, p. 228). Experiences can also help learners form connections between new information/tasks and their existing knowledge and prior experiences. When learners actively interact with information or experience tasks in a real-life situation, they may be more likely to internalise the concepts and understand the real-life relevance of the concepts. We incorporated experiential learning in several ways. First, to support the relevance of the response scale options, co-researchers developed response scales after engaging in meaningful activities and reflecting on their experiences. Second, co-researchers and focus group participants provided feedback on response scales after trialling different response options. This helped them identify which scales were easiest to understand and use. Third, prior to conducting focus groups, co-researchers learned how to run the focus groups by completing the activities themselves; experiencing the focus groups as “participants” supported their ability to reflect on the procedures and provide critical feedback (Table S1). Finally, because the co-researchers co-conducted focus groups, they could draw upon their experiences while analysing data. During data analysis, we reminded co-researchers of specific participants with whom they interacted during data collection, prompting them to compare and contrast these experiences with the group-level data. This supported them to link their individual experiences to the data set. Their ability to draw upon their experiences improved the quality of the inferences made about the data.

3.3 |. Breaking down tasks

Cognitive load theory calls attention to the executive function and working memory demands involved in learning (Clark et al., 2011). A related postulate is that learning becomes more difficult when tasks contain high “interactivity.” Interactivity refers to how different elements of a task relate to each other and must be simultaneously processed and managed. With increasing interactivity, the learner must manage more new information in working memory. Because many research tasks and activities require the manipulation and synthesis of large amounts of information, they may be described as tasks with “high” interactivity. Previous studies have suggested that complex tasks with high interactivity may be most easily learned when interactivity is reduced by breaking down a complex task into multiple simpler tasks and explicitly teaching the relationship between the tasks (Clark et al., 2011). Thus, it follows that youth with intellectual/developmental disabilities may be more able to engage in research when task interactivity is reduced.

We used activity analysis to reduce task interactivity and develop a series of discrete tasks, each with demands commensurate with co-researchers’ abilities. For example, the complex processes of developing and evaluating response scale options (Figure 3a–c) were broken into several discrete tasks to reduce abstract thinking and working memory demands. Of this process, one co-researcher said, “[I] liked getting…each individual thought…and sorting the thoughts…it helped me…learn.” Each discrete task drew upon co-researchers’ unique strengths, such as recognising patterns, counting and reflecting on their experiences. Each discrete task was also made accessible by incorporating other strategies, such as universal design (e.g. colour-coding; Figure 2, use of images, etc.) and use of lived experiences (e.g. reflecting on how response scales related to one’s own life). Breaking down tasks benefitted all members of the team, including the academic researchers, as it required us to clarify our ideas and supported a shared understanding of our goals. While reviews of inclusive research, consensus statements and practice guidelines (e.g. Cooperative Research Center for Living with Autism, 2016; Frankena et al., 2019; Stack & McDonald, 2014) have highlighted the importance of accessibility, our use of cognitive load theory and task analysis provides a way to organise researchers’ thinking around how to ensure access.

3.4 |. Peer support

Peer support increased co-researchers’ self-efficacy and ability to perform research tasks and mitigated power differences between the co-researchers and the academic researchers. The central tenet of social cognitive theory is that individuals learn from each other via modelling and observation (Bandura, 1989). Learning from and with peers with similar life experiences can also facilitate increased self-efficacy and self-esteem (Bandura, 1989). In this study, working in groups enabled co-researchers to draw upon their complimentary and collective strengths and skills to support each other and accomplish and learn complex research tasks.

An additional benefit of peer support is its potential to mitigate power differences between co-researchers and academic researchers. The inherent power dynamics between academic researchers without disabilities and co-researchers with intellectual/developmental disabilities may lead to co-researchers being hesitant to share their perspectives (Schwartz & Durkin, 2020). Alternatively, the presence of peers with shared identities may increase individuals’ comfort and the group’s collective efficacy to speak up and challenge academic researchers without disabilities (Northway et al., 2014; Schwartz et al., 2020; Stack & McDonald, 2014). Throughout the design process, co-researchers usually outnumbered academic researchers. Additionally, by working individually or in pairs to complete activities, co-researchers took on an expert role by presenting their ideas to the larger group. Finally, we ensured each meeting had time for unstructured socialisation to facilitate relationships between team members. Doing so helped foster a group identity, familiarity and trust among team members (Schwartz et al., 2020; The Learning Difficulties Research Team, 2006).

3.5 |. Reflections on the inclusive research process for response scale development

There has been a call for increased transparency about inclusive research processes and greater attention to the specific processes, strategies and theories underlying inclusive research with individuals with intellectual/developmental disabilities (e.g. Frankena et al., 2019; O’Brien et al., 2014; Schwartz et al., 2020). Others have noted the need to clarify the “added value” of inclusive research (Walmsley et al., 2018). We responded to this call by providing specific details about the activities co-researchers undertook and describing how our inclusive process was informed by four broad strategies. Combined with long-term engagement, these strategies supported co-researchers to critically engage in the research, and thereby take ownership over specific aspects of the research process (e.g. developing response scales, developing data collection procedures), thus shaping the end product: a content valid PEDI-PRO response scale (i.e. “adding value”). However, some of these strategies may have decreased co-researchers’ ownership and power. For example, it is possible that when we broke down tasks, we imposed our beliefs about the way in which these larger tasks should be conducted. Co-researchers may have had a unique way to conduct these tasks that may have led to different results (Janes, 2016; Milner & Frawley, 2018). Nonetheless, explication of these four strategies is an important step towards increasing transparency of inclusive research processes. While this lengthy inclusive research process precluded large-scale psychometric analysis as early as we had hoped, the close collaboration with youth with intellectual/developmental disabilities led to the development of a response scale youth could use in a consistent manner (Kramer & Schwartz 2017b; Kramer et al., 2021) and that will undergo future psychometric evaluation.

3.6 |. Limitations and future research

While we retroactively identified four broadly applicable strategies that supported our inclusive research process, we did not systematically study their efficacy. Additionally, we did not evaluate how these strategies may be optimised or which strategies were most applicable to and efficacious for different phases of the research process. Future research may address these topics. While these strategies are supported by the literature (e.g. Frankena et al., 2019; O’Brien et al., 2014; Stack & McDonald, 2014) and the authors have used these strategies in other studies (Schwartz et al., 2020; Kramer et al., 2011), additional research is needed to evaluate their generalisability. We described how the accessible inclusive research approach supported co-researchers to collaborate in the design of PROM scales that were easy to understand and use by other youth with intellectual/developmental disabilities; future research is needed to determine the relationship between inclusive research approaches and accessible research products. Finally, we primarily discussed strategies that supported the youth co-researchers to perform research tasks and processes. Future research may address strategies and processes that directly address power sharing and decision-making.

4 |. CONCLUSIONS

The inclusive research process resulted in the development of response scales with strong content validity that youth with intellectual/developmental disabilities can use to report their functional performance of everyday activities. We identified four strategies that supported co-researchers to drive this process: application of UDL approaches, breaking down tasks, use of lived experiences, and peer support. Future research may explore how to optimise these strategies in inclusive design of PROM response scales and study their efficacy across diverse inclusive teams and projects.

DATA AVAILABILITY STATEMENT

This study did not draw upon a specific data set. Please contact the authors for additional information about and/or access to the data discussed.