Assessing culturally inclusive instructional design in online learning

Abstract

This study assessed the psychometric properties of the culturally inclusive instructional design (CIID) scale with 31 items on a 7-point Likert scale. The data were collected from the training (N = 55) and validating samples (N = 80) of K-20 educators. Data analysis employed exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). EFA results revealed a clear five-factor structure, and the CFA results indicated good factor loadings. The reliability indices were .95 and .94 for the training and validation samples, respectively. The significant correlations among the factors indicated the five subscales measuring on the same CIID construct. In contrast, a non-perfect correlation presented a discriminating power for each subscale measuring the unique dimension of the construct. The study results established the validity and reliability of the instrument to measure culturally inclusive instructional design with implications for the design and development of online learning for cultural inclusivity.

Introduction

Global accessibility to online learning has opened up access to education without temporal or spatial constraints (Allen & Seaman, 2017). Quality education is based on the principles of equity and inclusion. Inclusion is “a process that helps to overcome barriers limiting the presence, participation, and achievement of learners” (UNESCO, 2017, p. 7). Creating an inclusive, equitable socio-constructivist learning environment is essential in meeting the needs of all learners (McLaughlin & Oliver, 2000). When a common background is assumed in learners, marginalized identities may face substantial barriers due to access and equity issues (Rogers-Shaw et al., 2018). Equity ensures that, "there is a concern with fairness such that the education of all learners is seen as being of equal importance." (UNESCO, 2017, p. 7). Cultural inclusivity ensures that diverse learners thrive and that their linguistic and contextual differences are leveraged for maximum participation and achievement.

Education is inherently sociocultural (Thomas et al., 2002) as knowledge is situated in a sociocultural context. Cultural inclusivity is the foundation of student-centered learning (Chen et al., 1999). "Culture is so much part of the construction of knowledge that it must underpin not only the analysis phase by all phases of the design process" (Thomas et al., 2002, p. 41). Asino et al. (2017) further posited that, “the “next’” focus of design should be an inclusion of culture into design practices; an inclusion that is merged into every stage” (p. S875). Culture is at the heart of meaning-making (Vygotsky, 1978). Culture provides meaning and order to our lives through shared social values, cognitive codes, behavioral standards, world views, and beliefs (Delgato-Gaitan & Trueba, 1991). Gunawardena et al. (2019) defined culture as “a collection of the shared perception of the world and our place in it. We always share aspects of cultural identities with others, but their aggregate is ours alone. In this way, culture contributes both to our connection with others and our uniqueness as individuals” (p. 3). Culture also shapes our thinking and learning.

By attending to the cultural aspect of inclusivity in online learning, course designers ensure that the needs and expectations of diverse learners are being met for an equitable learning or training experience. The goal of inclusivity is “establishing partnerships and conducting needs assessments to maximize the capacity that already exists within a given context, and addressing cultural factors that impact online learning—developing a learning community, negotiating identity, power, and authority, generating social presence, supporting collaboration, engaging in authentic inquiry-based learning, navigating interactions in a second language, and developing co-mentoring relationships to support learning” (Gunawardena, 2020, p. 5).

Despite the need for cultural inclusivity, online courses are generally designed for dominant cultural norms in engagement and content (Neto et al., 2014) and “may not provide the need for cultural contextuality” (McLaughlin & Oliver, 2000, p. 59). Gender, race, ethnicity, and culture have often been overlooked in the systematic design of instruction (Asino et al., 2017; Knupfer, 1997; Powell, 1997a; Thomas et al., 2002; Romero-Hall, 2022). Instructional design is about creating a cultural identity; therefore, it cannot be culturally neutral (Henderson, 1996). Making a single course for global audience results in efficient but culturally bankrupt courses (Carr-Chellman, 2005). Lack of cultural inclusivity impacts learners’ sense of belonging in the learning community; thereby, impacting learner engagement and motivation, and subsequently, their performance in the course.

The U.S. is changing in significant ways from technology use to demographics. COVID-19 has transformed the way we live and work; social trends of hybrid and online learning, remote work, and skill-based learning continues to shape the future of education (Pelletier et al., 2022). As higher education makes strategic investments in future, equity and inclusion in distance learning remains central to student retention and success. Currently, half of K-12 students in public schools are from culturally diverse backgrounds (Schaffer, 2019). The U.S. Census Bureau projects that more than half of the U.S. population will be from diverse ethnicities by 2050 (Parker et al., 2019). Therefore, instructors should be prepared to design for cultural inclusivity in all modalities to address the needs of diverse learners across the U.S. and abroad.

The United States exports online educational content globally through Massive Open Online Courses (MOOCs). The amount of exported online educational content in MOOCs continues to grow and favor the more privileged (Pouezevara & Horn, 2016). Additionally, it may not meet learner expectations and needs due to instructional designers’ faulty assumptions and “cultural blinders” (Rogers et al., 2007, p. 198). Thus, learners may not be given equitable educational access due to incorrect assumptions. Consequently, the lack of support and access can result in high attrition rates for underserved, marginalized learners (Dziuban et al., 2018).

This study aimed at addressing the problems of lack of engagement, diminished performance, and attrition of diverse, global learners due to the lack of cultural inclusivity in instructional design. We propose a solution by designing a valid and reliable instrument to assess culturally inclusive instructional design in online learning. The purpose of this research was to study the psychometric properties of the measurement of the educators' perception of the culturally inclusive instructional design (CIID). Specifically, through this study, we studied: (1) the dimensionality of the CIID and (2) whether the instrument is reliable and valid for measuring CIID. The following section presents the theoretical framework informing our study.

Theoretical framework: Gunawardena’s wisdom communities

The theoretical framework for this research is based on Gunawardena's Wisdom Communities (Gunawardena et al., 2019). The concept of Wisdom Communities or Wis-Com is grounded in the sociocultural theory of Vygotsky (1978). Sociocultural theory (Vygotsky, 1978) posits that learning occurs within the sociocultural framework of our lives, where direct communication and indirect cultural influences, play a role in making sense of the world. Social and individual processes of learning are interdependent. The social processes scaffold zone of proximal development, and individual processes help internalize the information. Learning in this framework is distributed, occurring in multiple places and spaces; it is interactive, where the learner is actively engaging with the content, other learners, and the instructor; and it is contextual, indicating that the sociocultural, situational context matters in learning.

Gunawardena et al.’s (2019) Wis-Com focuses on transformative learning by viewing wisdom as a personal quality of humility, inclusiveness, kindness, generosity, and reflectiveness that honors the cultural gifts of individuals. The wisdom community fosters collaboration and reflection among its members. It supports them in their pursuit of knowledge and personal growth through communication, community, inclusive adaptable technologies, distributed co-mentoring, and learner support to attain transformational learning. “An effective learner support system in Wis-Com reflects the diversity of learners’ experiences, including differences in age, gender, cultural backgrounds, education, language, socio-economic status, family and employment commitments, goals, objectives, needs, desires, and access to technology” (Gunawardena et al., 2019, p. 9).

Wisdom communities are designed to enhance interpersonal skills, teamwork, empathy, communication, global perspectives, and emotional intelligence to approach cross-cultural interactions with humility and a desire to learn more. Collaborative inquiry is an interactive, experiential process to find interdisciplinary, cross-cultural solutions (Lombardi, 2007). The Wisdom community promotes learning across cultures and inclusion. "Inclusion fosters a sense of belonging to a community— feeling appreciated for one's unique characteristics, perspectives, contributions, and ways of thinking. Participants feel comfortable sharing their ideas, identities, cultures, languages, and ways of seeing the world" (Gunawardena, 2020, p. 7). The following section reviews and discusses the literature to identify the dimensionality of cultural inclusivity based on prior empirical and theoretical research (Frechette et al., 2014; Gunawardena, 2020; Gunawardena et al., 2019; McLaughlin, 2001; McLaughlin & Oliver, 2000; Rogers et al., 2007; Thomas et al., 2002) and online learning (Arbaugh et al., 2008; Garrison et al., 2000; Moore, 1989).

Dimensions of cultural inclusivity

Based on Gunawardena's Wisdom Communities framework (2019), we examined the dimensions of cultural inclusivity that aligned with the principles of Universal Design for Learning (UDL) to lower anticipated barriers for a more inclusive learning experience. The dimensionality of cultural inclusivity was initially proposed by Thomas et al. (2002) in their Third Dimension of ADDIE: The Cultural Embrace conceptualization with introspection, intention, and interaction. The interaction component in this research was further categorized into learner-learner interaction, learner-instructor interaction, and learner-content interaction based on the community of inquiry (Garrison et al., 2000) for online learning.

Introspection

The first dimension, introspection, entails course designers and instructors reflecting on their values, beliefs, privileges, biases, and assumptions. "The designer's worldview cannot be divorced from his societal context; therefore, it becomes critically important that the designer becomes introspective in their approach when designing instruction. We must consider our own thoughts, beliefs, attitudes, desires, and feelings toward these cultures" (Thomas et al., 2002, p. 44). Jonassen et al. (1999) noted that reflection or introspection is essential to meaningful learning. Unpacking privileges and implicit cultural bias (Gunawardena, 2020) is the first step toward examining social, cultural, and linguistic assumptions. A cross-disciplinary perspective (Vasquez et al., 2013) espouses for unpacking privileges and biases through reflexivity. Reflexivity involves thinking about our values, beliefs, attitudes, and assumptions, and how they shape our instructional design and pedagogical decisions.

Rogers et al. (2007) posited that challenges arise when instructional designers encounter foreign cultures with different expectations and abilities than otherwise assumed. Incorrect assumptions lead instructional designers to re-examine "the influence of their cultural blinders" (p. 198) on their practice. Most assumptions are implicit, and awareness of cultural differences does not necessarily lead to understanding them (Rogers et al., 2007). Therefore, cultural inclusivity requires that we analyze our social, cultural, and language assumptions (Frechette et al., 2014) with intentionality and understanding. Social assumptions are related to the social context; cultural assumptions are preconceived ideas about culture; and linguistic assumptions are predetermined notions about meaning-making and communication practices within a cultural group. For instance, linguistic variety, cultural discourse, and communication styles can be assumed to align with the dominant cultural norms of the course designer.

Having an awareness of overlapping cultural identities (Gunawardena et al., 2019) prevents stereotypical assumptions being made solely based on one’s ethnicity or nationality. Overlapping cultural identities manifest when an individual identifies with more than one cultural group based on their background, affiliations, and life experiences. Hofstede’s (1980) identification of the cultural differences at the national level in terms of individualism and collectivism, and Hall’s (1976) bipolar dimensions of high context-low context, implying direct and indirect communication, have evolved with globalization and cross-cultural pollination of ideas. An individual may not think in terms of bipolar dimensions but in multiple dimensions being shaped by evolving cultural identities with a contextualized adaptive thought process. As Gunawardena et al. (2019) aptly put it that we all have a "unique cultural DNA" (p. 3). An individual is constantly evolving and growing based on intercultural and cross-cultural interactions and experiences. Therefore, a static conception of cultural identity does not apply in a globalized, interconnected world.

The disconnect between assumptions and reality may happen within the same cultural heritage for people living in different cultural contexts. Therefore, it is wise to approach online learners with cultural humility (Gunawardena, 2020; Tervalon & Murray-Garcia, 1998) to know more about their needs, cultural context, and expectations through systemic, ongoing needs assessments throughout the design process, and regular check-ins with local stakeholders. Analyzing social, cultural, and linguistic assumptions helps in seeking solutions to cultural dissonance as it bridges the disconnect between assumptions and reality to meet online learners' expectations and needs. Designers must reflect on how their positionality impacts the course design. The next dimension is the intention, as explained below.

Intention

Instructional designers should be intentional in course design to embed diverse cultural perspectives (Milheim, 2018; Thomas et al., 2002). Resta et al. (2018) observed that digital equity is more than access to hardware, software, and connectivity but also relevant culturally inclusive content and instruction. Capitalizing on the diversity of online learners requires an intentional effort in course design and development. Honoring learners begins by embedding learners’ cultural perspectives to leverage their cultural capital (Gunawardena et al., 2019; MacLaughlin, 2001; McLaughlin & Oliver, 2000) while centering marginalized perspectives. Furthermore, providing choice and flexibility in engaging with content and demonstrating learning based on Universal Design for Learning (UDL) principles ensures inclusivity.

Designing the course to help learners navigate and negotiate power differentials should be planned as power relations are deeply ingrained in the fabric of diverse cultural contexts. Gunawardena (2020) observed that hierarchy and power distance play a role in educational transactions in some cultural contexts. Due to power differentials, learners may feel uncomfortable questioning authority, asking questions, or seeking help. In some cultural contexts that are rooted in patriarchy and reverence for elders and teachers, learners may not be equipped with strategies to assert their instructional needs. Removing any anticipated barriers by being intentional in course design can ensure inclusivity. A suggested course of action is "to help learners to negotiate power dynamics and develop an inclusive learning environment; we need to support instructors to transition to the role of a facilitator and mentor, valuing and utilizing students' prior experiences and skills" (Gunawardena, 2020, p. 19).

Cultures persist through interaction, yet interaction may not ensure a lack of marginalization. Gunawardena (2020) noted that "while interaction can generate different ways of seeing the world, it does not guarantee that these views will be heard, appreciated, and valued" (p. 7). Therefore, it is crucial to ensure that course design leverages cultural strengths and centers marginalized perspectives for cross-cultural understanding. For example, learners may explore authentic global issues through a local contextual lens to enrich cross-cultural perspectives. Additionally, choosing instructional media, materials, and resources to authentically represent the target audience helps learners in connecting their background knowledge to the new information. Gunawardena (2020) suggests, “Images and photographs must relate to the cultural context” (p. 19). Utilizing and modifying Open Educational Resources (OERs) to align with the local cultural context could provide relevance to design authentic online learning.

Being immersed in the culture and studying its hidden cultural nuances is beneficial in assessing the contextualized needs to design relevant learning or training experiences to enhance performance. Authentic representation in course design offers the additional benefit of representing diverse cultural perspectives; this representation contributes toward a sense of belonging to a learning community. “An inclusive view is about valuing diversity and ensuring the traditions and beliefs of learners are recognized and given expression in the learning environment” (Branch, 1997, as cited by McLaughlin & Oliver, 2000, p. 230). Culturally inclusive online learning aims to ensure that pedagogy and curriculum are flexible, adaptable, and relevant to students from various cultural and language backgrounds (McLaughlin, 2001). The following three dimensions indicate the social and cognitive processes of online learning as reflected in the community of inquiry (Garrison et al., 2000) and transactional distance theory (Moore, 1989). The Learner-Content Interaction discussed below is the first dimension crucial for engaging in the cognitive processes of online learning through the collaborative construction of meaning.

Learner-Content Interaction (LCI)

Interaction is viewed as the "defining characteristic of education" (Moore, 1989, p. 1). With its three forms of interaction, the community of inquiry (Garrison et al., 2000) provides a collaborative-constructivist perspective to support discourse and reflection in online learning. In addition, interaction treatments or instructional conditions that facilitate various types of interaction have been shown to impact student achievement positively (Bernard et al., 2009).

Learner-Content Interaction (LCI) is intellectually interacting with the content, resulting in changes in the cognitive structures (Moore, 1989). Gunawardena et al. (2019) posited that "Interaction prioritizes learner involvement via different forms of communication and by welcoming divergent viewpoints (p. 5). Learner-Content Interaction or cognitive presence is "the extent to which learners can construct and confirm meaning through sustained reflection and discourse in a critical community of inquiry" (Garrison et al., 2001, p. 11).

Knowledge-building processes differ in diverse cultural contexts (Gunawardena, 2020). Learners explore meaning situated in their cultural contexts as they build upon their background knowledge to engage with the content. In specific cultural contexts, consensus building (Lopez-Islas, 2001) is more suitable than an argument or debate. During collaborative inquiry, learners leverage their cultural resources as they negotiate meaning and synthesize knowledge through critical reflection. The goal of collaborative inquiry is to preserve learners’ collective knowledge (Gunawardena et al., 2019).

Cognitive presence indicates collaborative knowledge construction (Arbaugh et al., 2008). Linguistic backgrounds influence the articulation of the thought process; multilingual learners’ linguistic structure is influenced by their native language. Access to a foreign language may be influenced by learners' familiarity with accents and connotative meaning in a cultural context. Idioms, situated in specific cultural contexts, may be challenging to discern and apply for global learners. Providing visuals, graphic organizers, and accessible media scaffold meaning making to promote equitable learning for diverse learners. The interaction between learner and instructor is also a significant aspect of the culturally inclusive instructional design. The following section discusses the relevance of Learner-Instructor Interaction in the design and facilitation of online learning.

Learner-Instructor Interaction (LII)

In Learner-Instructor Interaction, course design and facilitation strategies are planned to leverage cultural strengths with choice and flexibility. For example, Branch (1997) believed that Gagne's nine events of instruction (1985) could promote the exchange of knowledge from cultural perspectives with intentional design decisions. The Learner-Instructor Interaction (LII) or teaching presence is a crucial component of student satisfaction and the success of a formal educational community (Garrison & Arbaugh, 2007). Teaching presence or Learner-Instructor Interaction (LII) is a multidimensional concept that includes the design of the educational environment, facilitation of discourse, and direct instruction (Arbaugh et al., 2008) by integrating social and cognitive processes for a meaningful educational experience in online learning. Anderson et al. (2001) stated that "Teaching presence is the design, facilitation, and direction of social and cognitive processes to realize personally meaningful and educationally worthwhile outcomes" (p. 5).

Creating a community of inquiry requires a thoughtful teaching presence (Shea et al., 2010). Teaching presence, in the form of course design and facilitation, significantly affects learners' sense of belonging (Borup et al., 2012). Caskurlu et al. (2021) highlighted the importance of teaching presence in supporting social interaction in online learning. Meaningful technology integration through intentional course design and facilitation can also enhance group interaction (Pinsk et al., 2014). Teaching presence is distributed in Gunawardena’s wisdom communities as distributed co-mentoring (Gunawardena et al., 2019) that leverages the expertise of community members, peers, and mentors in supporting learners. This form of distributed co-mentoring is highly relevant in supporting online learners situated in varied cultural contexts as it leverages local resources and community.

Teaching presence or the Learner-Instructor Interaction supports learners’ interest and motivates them for self-directed learning. Motivation entails a willingness to exert considerable mental effort and seek solutions to problems. Among the determinants of motivation are value and relatedness (Blumenfeld et al., 2005). Course design, that leverages cultural perspectives, conveys value and relatedness to motivate diverse learners and creates a sense of belonging in the course community (Stevens et al., 2016).

Demonstrations, presentations, and modeling of cognitive and social processes are part of Learner-Instructor Interaction. Learner-Instructor Interaction facilitates the application of skills and evaluation of student progress to personalize learning. The instructors counsel, support, and encourage learners, and communicate course expectations for learner autonomy while providing timely, constructive feedback (Arbaugh et al., 2008). They guide learners in navigating cultural norms and help-seeking behavior. Instructors design access to inclusive, accessible technology to scaffold learning. Gunawardena (2020) stated that since “cultural biases often shape the design of digital tools, carefully reflecting on how to implement technology tools to find, filter, record, and dialogue with one another will result in more inclusive learning experiences" (p. 24).

Facilitating learner autonomy with choice and flexibility addresses diverse student needs; creating opportunities for learners to choose learning activities that offer different ways to communicate, process information, and interact encourages inclusivity (Gunawardena, 2020). Also, facilitating reflection (metacognition) supports self-regulated learning as learners set personalized goals for themselves (Schon, 1984); learners plan, monitor, evaluate progress, and make necessary adjustments (Blumenfeld et al., 2005). To summarize, facilitating reflection supports self-regulation in learners. The section below presents the last dimension, Learner-Learner Interaction, that is essential for creating a trusting community in online learning.

Learner-Learner Interaction (LLI)

Learner-Learner Interaction (LLI) occurs in real-time or asynchronously in pairs, small groups, and as a whole group. Moore (1989) noted that Learner-Learner Interaction (LLI) is essential for effective group functioning. The community of inquiry framework (Garrison et al., 2000) referred to Learner-Learner Interaction as social presence in an online course. Social presence contributes toward a trusting community by building interpersonal relationships in a positive social space to facilitate academic discourse. Social presence has been defined as the "ability of participants to identify with the community (e.g., course of study), communicate purposefully in a trusting environment, and develop affective and interpersonal relationships progressively as a way of projecting their individual personalities” (Garrison, 2011, p. 34).

In online communities, negotiating identity and expressing it through self-disclosure and storytelling builds trust (Gunawardena, 2020; Gurjar, 2020). Open communication, group cohesion, and affective expression were included in the original conception of social presence to operationalize the concept (Arbaugh et al., 2008). Later, social comfort (Carlon et al., 2012) and social space (Kreijns et al., 2014) were added to reflect the multidimensional nature of this construct. Social comfort and social presence can be achieved through spontaneous informal interactions (Dunlap & Lowenthal, 2009), humor, and fun (Gunawardena, 2020).

Learner-Learner Interaction (LLI) or social presence plays a central role in supporting academic discourse as it is a "major lever for engagement, sense-making, and peer support." (Armellini & De Stefani, 2016, p. 1202). When learners come from diverse national, ethnic, cultural, and linguistic backgrounds, creating a comfortable social space to share diverse perspectives is vital. Cultivating a comfortable social space supports collaboration for a shared purpose or vision. Also, honoring learners' cultural gifts and talents creates social comfort. The contribution of learners' cultural resources cultivates a sense of belonging (Gunawardena, 2020; McLaughlin & Oliver, 2000) in an online learning community: “Inclusive learning environments must foster a sense of belonging to and participation in a learning community” (Gunawardena, 2020, p. 11). Furthermore, the learning community leverages its wisdom for the social good (Gunawardena et al., 2019).

The purpose of the study was to examine: (1) the dimensionality of the construct of Culturally Inclusive Instructional Design (CIID) and (2) whether the instrument is reliable and valid for measuring the CIID construct. The following section reports the method with participants, procedures, and data analysis results.

Method

Participants

The participants were a total of N = 135 K-20 educators, primarily from higher education settings, with a training sample of 55 participants and a validation sample of 80 participants. As shown in Table 1, the sample consisted of a majority of Caucasians and participants from other ethnic groups. There were more female participants than males. The mean age of the participants was 48 years. Most of the participants came from the instructional design and technology field, with some participants from other fields, such as curriculum and instruction and educational leadership. About 70% of the participants had one to ten years of online course design experience; over 90% of the participants had one to twenty years of online teaching experience. (See Table 1).

Table 1.

Demographic data of the participants

Demographics	Categories	Percentage (%)
Ethnicity	Caucasian	69.57
	African American	10.87
	Asian	6.52
	Hispanic/ Latino/a	2.17
	Mixed	3.26
	Prefer not to answer	7.61
Age	Below 40	19.27
	40–49	40.36
	50–59	24.00
	60–69	18.00
Gender	Male	31.52
	Female	66.30
	Prefer not to answer	2.17
Specialty	Instructional Design and Technology	31.82
	Curriculum and Instruction	18.18
	Educational Leadership	11.36
	Educational Measurement and Evaluation	2.27
	Educational Psychology	3.03
	Special Education/Disabilities Studies	3.03
	Language Arts/Literacy/Reading Education	5.30
	Writing and Rhetoric	4.55
	Others	20.46
Years of Online Course Design	None	10.34
	1–3 years	33.33
	4–5 years	18.39
	6–8 years	10.34
	9–10 years	6.90
	More than ten years	20.69
Online Teaching Experience	None	4.35
	1–3 years	27.17
	4–5 years	19.57
	6–8 years	9.78
	8–10 years	11.96
	10–15 years	11.96
	16–20 years	9.78
	20–25 years	2.17
	25–30 years	2.17
	Over 30 years	1.09

Procedures

Data were collected after obtaining IRB approval from the researcher's university. This survey research employed random sampling of educators from higher education and K12. The data sources consisted of questionnaire responses on a Likert scale from 1 to 7 ranging from strongly agree (7), agree (6), somewhat agree (5), neither agree nor disagree (4), somewhat disagree (3), disagree (2), and strongly disagree (1). The questionnaire was created on Qualtrics and distributed through email, online faculty listserv, professional organization (AECT) website, Facebook groups, and Twitter.

Data analyses results

The study was conducted through three stages described in the following sections. First, in stage 1, we focused on item development by identifying the domain based on previous literature (Garrison et al., 2000; Gunawardena et al., 2019; Moore, 1989; Thomas et al., 2002). Then, we generated the items and examined face validity with five expert reviewers for clarity and feedback. The criteria for inclusion of experts were: They had to be instructional design and technology educators at a 4-year public or private university; and engaged in online teaching and learning research. Next, in stage 2, we conducted a validity study on the training sample through Exploratory Factor Analysis (EFA) after checking the face validity. The items that did not fit the factor structure were removed to design the survey instrument iteratively. Finally, in stage 3, a validity study was conducted on the validation sample with Confirmatory Factor Analysis (CFA). The stages of data analysis are presented more in depth below.

Stage 1: Item development

Domain identification

Based on the culturally inclusive instructional design (CIID) affect approaches regarding psychometric properties assessed through self-reporting data, the current study employed an iterative process using literature reviews to confirm the dimensions of the CIID construct specifically. Through reviewing the existing studies (e.g., Arbaugh et al., 2008; Bernard et al., 2009; Garrison et al., 2000; Gunawardena et al., 2019; McLaughlin, 2001; McLaughlin & Oliver, 2000; Moore, 1989; Rogers et al., 2007; Thomas et al., 2002), we identified five dimensions of the CIID construct. First, introspection, intention, and interaction were identified through culturally inclusive instructional design framework (Gunawardena et al., 2019; Thomas et al., 2002). Thomas et al. (2002) initially conceptualized Introspection, Intention, and Interaction in their cultural dimension of ADDIE. Gunawardena et al. (2019) further posited that these dimensions align with Universal Design for Learning (UDL) to create equitable learning. Finally, the interaction was further categorized into three types pertinent to social and cognitive processes of online learning based on the community of inquiry framework (Arbaugh et al., 2008; Garrison et al., 2000) and Transactional Distance theory (Moore, 1989).

Item generation

We developed an item pool to generate the items based on the domain identified and delineated through the theory grounded in previous research. We applied a deductive method. First, we used the descriptions of the relevant domain and the identification of appropriate questions through a deductive method to do the logical partitioning. After generation of an item pool with 51 questions, we retained 37, 7-point-Likert scale items, as the manifest variables for the constructs of CIID with five dimensions: Introspection with example items as Humanizing learners begins by examining one’s privileges, and My identity (values and beliefs) influences the way I teach; Intention with example items as Embed divergent cultural perspectives into the course, and Empower learners by leveraging their native cultural heritage; Learner-Content Interaction with example items as Learners explore meaning situated in their unique cultural context and Learners synthesize knowledge through critical reflection; Learner-Instructor Interaction with example item as Clarity of communicating course expectations (e.g. cultural norms and help-seeking) enhances learning., and Facilitating learner autonomy with choice and flexibility addresses diverse learner needs; Learner-Learner Interaction with example item as Contribution of learners' cultural resources creates a sense of belonging and Honoring learners' cultural gifts/talents creates social comfort. In the process of generating items, we cross-checked the wording of each item to avoid double-barreled questions and offensive or potential biases in social identities. We also examined the consistency in understanding of the questions to receive consistent communication of what constitutes an adequate answer for each item.

To achieve face validity, we sent the review version of CIID to five educators who specialized in instructional technology and online learning to review the items in CIID. The purpose of examining the face validity was to confirm the meanings of the items in each subscale to measure the affect under each dimension. The inter-rater reliability revealed a substantial agreement with Cohen's Kappa statistics of .701 (McHugh, 2012). Thus, the content validity of the CIID was confirmed in the first stage by checking the consistency of the content expressed by the items against the theoretical dimensionality of the construct, and the judgments expressed by the experts. Next, we conducted the study on the training sample as presented below.

Stage 2: Study the training sample

The original 37 questions were administered to the K-20 educator participants, and 55 responses were received. This data was used as a training sample. An exploratory factor analysis (EFA) was used to assess the factor structure or dimensionality of the construct. The principal component analysis identified six items that did not fit the factor structure. After removing the bad (non-fitting) items, a clear 5-factor structure was revealed using the principal axis factoring method to identify latent dimensions with a total of 31 items retained (See Table 2). Principal components analysis was used and found 65.02% total variance explained. IBM SPSS 26 was used for data analysis at this stage.

Table 2.

Rotated factor matrix of the factor structure of the CIID scale

	Factor
	1	2	3	4	5
1_6	.911	.025	.181	.063	.142
1_2	.880	.296	− .014	− .024	.082
1_7	.849	.035	− .025	.041	.013
1_5	.820	− .093	.240	.200	.240
1_4	.716	− .038	.017	.194	.085
1_3	.694	− .051	.162	− .107	− .191
1_1	.559	.364	.079	.271	− .147
Q4_8	.193	.848	.118	.136	.280
Q4_4	.041	.748	.168	.267	.146
Q4_7	.001	.722	.217	.219	.316
Q4_3	− .066	.648	.273	.106	.153
Q4_5	.004	.618	.328	.241	.411
Q3_6	.203	.184	.834	.150	.223
Q3_5	.075	.091	.793	.109	.344
Q3_4	.053	.333	.666	.321	.064
Q3_1	.042	.228	.658	.221	.303
Q3_2	.164	.340	.503	.140	.112
Q3_7	.287	.441	.483	.228	.035
Q2_8	.111	.116	.083	.836	.201
Q2_9	.108	− 0.020	.340	.775	.278
Q2_7	.079	.281	.423	.762	.118
Q2_2	.009	.338	.073	.694	− 0.017
Q2_5	.259	.405	.180	.628	.067
Q2_12	.109	.391	.474	.530	.210
Q5_5	.028	.104	.056	.110	.760
Q5_6	.047	.425	.228	.171	.743
Q5_7	.066	.347	.367	.129	.671
Q5_1	.098	− 0.006	.419	.157	.647
Q5_4	.017	.140	.072	.008	.576
Q5_9	− 0.001	.350	.333	.194	.495

Extraction Method: Principal Axis Factoring

Rotation Method: Varimax with Kaiser Normalization

^aRotation converged in six iterations

As we can see from Table 2, the factor loading scores in bold are all greater than 0.4, that reflects consistency in measuring the same sub-construct as specified in each factor from 1–5 (Guadagnoli & Velicer, 1988). The factor loading scores in bold indicate that the items measure the same sub-construct. The test reliability of the training sample indicated a high reliability with Cronbach’s alpha value of α = .95, indicating good internal consistency measuring five-affect in the culturally inclusive online instructional design construct. The internal consistency for the Introspection subscale was α = .90; the Intention subscale was α = .90; the Learner-Content Interaction (LCI) subscale was α = .89, the Learner-Instructor Interaction (LII) subscale was α = .91, the Learner-Learner Interaction (LLI) subscale is α = .87.

In sum, the internal consistency of all the five subscales ranged from .87 to .90, which is reliable based on the criterion as reliable for Cronbach’s alpha (.84–.90), and α = .95 for total item internal reliability is excellent based on Taber (2018). To further study the psychometric prosperities of the instrument, we used a different sample to cross-validate the instrument. In the following section, we report the cross-validation results.

Stage 3: Study the validation sample

The validity and reliability of CIID were further assessed in this stage. An independent sample of 80 was used as a validation sample to confirm the psychometric properties of CIID further. First, confirmatory analyses were conducted by comparing the models to verify the scale's construct validity. Then, the adequacy of the five-factor measurement model fit was tested using AMOS 26 based on the theoretical structures for five constructs which were also identified as the five components by the EFA procedures through the training sample. The five-factor model produced a chi-square of 738.77.32 (df = 424, p < .05), root mean square error of approximation of .09, a smaller value of Akaike information criterion .1, and the other fit indices, including comparative fit index (CFI) is .77 and TLI = .75. The fit indices revealed that the model fit was not ideal; however, the factor loadings were from .37 to .71 which are all significant (See Table 3; Fig. 1). Therefore, with the currently available sample, the model fit is acceptable.

Table 3.

Factor loadings of the CIID scale

Latent Variables:
	Estimate	Std.Err	z-value	P( >|z|)	Std.lv	Std.all
f1 =~
Q1_1	0.781	0.148	5.263	< .001	0.781	0.600
Q1_2	0.498	0.105	4.744	< .001	0.498	0.543
Q1_3	0.368	0.094	3.915	< .001	0.368	0.459
Q1_4	0.497	0.112	4.432	< .001	0.497	0.526
Q1_5	0.389	0.055	7.140	< .001	0.389	0.758
Q1_6	0.366	0.123	2.979	0.003	0.366	0.370
Q1_7	0.389	0.070	5.549	< .001	0.389	0.636
f2 =  ~
Q2_1	0.597	0.103	5.792	< .001	0.597	0.623
Q2_2	0.578	0.101	5.736	< .001	0.578	0.618
Q2_3	0.642	0.078	8.270	< .001	0.642	0.800
Q2_4	0.603	0.073	8.276	< .001	0.603	0.798
Q2_5	0.667	0.108	6.148	< .001	0.667	0.643
Q2_6	0.709	0.081	8.711	< .001	0.709	0.833
f3 =  ~
Q3_1	0.557	0.096	5.793	< .001	0.557	0.616
Q3_2	0.745	0.116	6.405	< .001	0.745	0.670
Q3_3	0.849	0.113	7.498	< .001	0.849	0.749
Q3_4	0.740	0.087	8.539	< .001	0.740	0.818
Q3_5	0.900	0.098	9.166	< .001	0.900	0.855
Q3_6	0.884	0.109	8.098	< .001	0.884	0.788
f4 =  ~
Q4_1	0.573	0.148	3.864	< .001	0.573	0.439
Q4_2	0.572	0.101	5.678	< .001	0.572	0.611
Q4_3	0.585	0.082	7.163	< .001	0.585	0.727
Q4_4	0.535	0.066	8.109	< .001	0.535	0.793
Q4_5	0.539	0.070	7.643	< .001	0.539	0.758
Q4_6	0.712	0.112	6.385	< .001	0.712	0.669
f5 =  ~
Q5_1	0.643	0.079	8.104	< .001	0.643	0.791
Q5_2	0.448	0.087	5.173	< .001	0.448	0.562
Q5_3	0.496	0.071	7.013	< .001	0.496	0.715
Q5_4	0.565	0.069	8.156	< .001	0.565	0.794
Q5_5	0.644	0.070	9.243	< .001	0.644	0.862
Q5_6	0.569	0.081	7.052	< .001	0.569	0.718

Fig. 1.

CFA structure of five constructs of culturally inclusive instructional design in online learning

To summarize the confirmatory factor analysis results, Table 3 shows that the loadings for all factors were significant, with the loadings from .55 as good and .63 as very good or .71 as excellent and no loading were below .32 as poor (Comrey & Lee, 1992). Therefore, in this case, and at this point, we stopped the model modification to improve the model fit.

Test reliability of the validation sample indicated a high reliability with Cronbach’s alpha value of α = .94, indicating good internal consistency of CIID measuring five-affect in the culturally inclusive online instructional design construct. The internal consistency for the introspection was α = .73; the intention was α = .86, the LCI was α = .88, the LII was α = .81, and the LLI was α = .89.

In general, the internal consistency of all the five subscales ranged from .73 to .89. This indicated high reliability based on the criterion outlined in Cronbach’s alpha (.73–.95). Further, total item internal reliability of α = .94 was excellent based on Taber (2018). The reliability results from the validation sample were also consistent with the study results from the training sample.

The correlations between the factors were from .44 to .74, with the p-values that were all less than .001. The findings suggested that the five factors measured the same construct. Furthermore, the inter-factor correlations were all less than .75, which suggested that the factors were not redundant; that is, the five factors measure one construct's unique affects (See Table 4). Finally, we present the findings, implications, limitations of our research and provide suggestions for future research in the following discussion.

Table 4.

Factor correlations of the CIID scale

	Introspection	Intention	LCI	LII	LLI
Introspection
Pearson Correlation	1	.593**	.439**	.513**	.460**
p (2-tailed)		< .001	< .001	< .001	< .001
Intention
Pearson Correlation	.593**	1	.642**	.632**	.529**
p (2-tailed)	< .001		< .001	< .001	< .001
Learner-Content interaction (LCI)
Pearson Correlation	.439**	.642**	1	.725**	.571**
p (2-tailed)	< .001	< .001		< .001	< .001
Learner-Instructor interaction (LII)
Pearson Correlation	.513**	.632**	.725**	1	.740**
p (2-tailed)	< .001	< .001	< .001		< .001
Learner-Learner interaction (LLI)
Pearson Correlation	.460**	.529**	.571**	.740**	1
p (2-tailed)	< .001	< .001	< .001	< .001

^**Correlation is significant at the 0.01 level (2-tailed)

b. Listwise N = 66

Discussion

Findings

The results from this study strongly support the argument that the culturally inclusive instructional design in online learning can be reliably and validly measured by CIID, a scale with 31 items on a 7-point Likert scale. Findings from the two independent samples of educators, primarily from higher education settings, reflected the constructs presented in the theoretical framework based on Gunawardena’s Wisdom Communities (Gunawardena et al, 2019). The measure showed high internal consistency and reliability, for both the training sample (Cronbach’s α = .95) and the validation sample (α = .94). The internal consistency of all the five subscales ranged from .87 to .90 for the training sample and from .73 to .89 for the validation sample, which indicates that the measures for the consistency of each subscale is reliable to measure the same construct. The clear factor structure from exploratory factor analysis provided preliminary evidence for the validity of the CIID scale, while the CFA further confirmed the validity of the measure with high factor loadings. The inter item correlation between factors also revealed that each subscale measures each of the five different dimensions of the construct with a good discriminating power while the significant correlations (p < .001) between factors suggested that all 31 items of CIID consistently measured the same construct.

Implications

This research has implications for how online learning is designed to support global learners. First, there is an apparent need to address problems such as lack of learner engagement due to social inequities, oppression, and systemic injustice by recognizing the inseparability of culture, learning, and technology in instructional design to improve educational opportunities for all (Dickson-Deane et al., 2018; Reeves & Lin, 2020). Second, there is a need for centering social justice and excavating assumptions to address the needs of all learners in their unique cultural contexts (Zamora et al., 2021). Third, as student attrition in online learning is higher for students of color and marginalized populations (Dziuban et al., 2018), this research has the potential to support and retain global learners who seek representation, community, acceptance, and validation of their contributions and intersectional identities to stay engaged and motivated. This empirical study contributes toward a theoretical understanding of culturally inclusive instructional design construct and addresses the need for equitable learning to improve educational opportunities for online learners globally.

A valid and reliable instrument will make instructional designers more cognizant of their assumptions, values, and beliefs. Introspection is necessary for online course designers and instructors to critically examine their values, beliefs, privileges, and subsequent implicit bias that may result in educators making cultural, social, and linguistic assumptions (Gunawardena, 2020; Rogers et al., 2007). These assumptions should be verified through reliable information and outreach efforts with multiple stakeholders, such as community members in the local context, instructional designers, subject matter experts, external evaluators, and learners invested in the design and development of MOOCS or discipline-specific online courses that cater to the global population.

Based on the identified learner characteristics and needs analysis, the intention to embed divergent cultural perspectives (Thomas et al., 2002) must be incorporated throughout the systemic design process. Learner-content interactions, learner-instructor interactions, and learner-learner interactions need to be intentionally designed to build a learning community that leverages cultural gifts (Gunawardena et al., 2019) and is responsive (Morong & DesBiens, 2016) to diverse cultural contexts. Designing for and assessing cultural inclusivity in online learning is crucial for a global audience, and a valid and reliable Culturally Inclusive Instructional Design (CIID) scale contributes toward that effort. As the world becomes a global village, designing for diverse cultural contexts and the global population becomes essential to support learner success in improving performance and retention. This research contributes toward that effort as we strive for empathy and cross-cultural understanding and “seek to make a difference in the lives of learners around the world” (Reeves & Lin, 2020, p. 1999). The limitations of this research are presented below.

Limitations

A limitation of this research is the relatively small sample size of the validating data (N = 80) and instructors at multiple levels from K12-20. Further, the confirmatory factor analysis model fit could have been better with a larger sample. It is desirable to collect more data by upscaling this research with a sizeable inter-institutional, international, large-scale study to add to the data analysis to improve the model specifications, and to develop the CIID scale iteratively. Since most participants were Caucasians (70%), not all ethnicities were equally represented in this research. In the following section, we present suggestions for future research.

Future research

Future research may continue to iteratively design and refine the CIID instrument and collect large scale data across various institutions globally. Further, various data analyses may be undertaken to examine differences between different sectors, regions, populations, K-12, and higher education to build on the contributions of this research. A mixed methods study that examines the perspectives of faculty from different disciplines who are engaged in online teaching and learning, and their ways of embracing cultural inclusivity in the design and development of their online courses may have the potential to inform future research. Conclusions from our study are presented below.

Conclusion

The results from this study provided reliable and valid information about the CIID as a measure of the construct of the culturally inclusive instructional design (Gunawardena et al., 2019; McLaughlin & Oliver, 2000; Thomas et al., 2002), a seven point-Likert scale with 31 items.

Findings from two samples of K12-20 educators were cross-validated against the hypothetical model and statistical model with excellent reliability indices at the factor level and the total item level, and the factor loadings were all significant. The sample included a few individuals (N = 5 out of 80) from K-12; the rest were from higher education involved in some aspect of the design, development, or delivery of online courses.

A culturally inclusive instructional design scale for distance learning will benefit the scholarly community by providing a valid and reliable measurement instrument. It addresses the need for a measurement instrument that can effectively measure introspection, intention, and interaction (learner-instructor, learner-content, and learner-learner) and lead to further research efforts for reiteration and refinement. As online learning faces challenges in terms of student retention (Dziuban et al., 2018), especially for students of color and marginalized populations, it is imperative to design a culturally inclusive online learning wisdom community where every learner has a sense of belonging with their intersectional and overlapping cultural identities; who feels valued; and who is motivated to contribute, strive, and accomplish their personalized life goals. Cultural inclusivity is a worthwhile goal to pursue for instructional designers in various sectors to improve human performance and educational opportunities for all global learners!

Biographies

Nandita Gurjar

is an Assistant Professor in the College of Education at Rhode Island College. Gurjar’s research explores authentic learning, social justice, and equity in online learning through cultural inclusivity and feminist pedagogy.

Haiyan Bai

is Professor in the Learning Sciences department at the University of Central Florida. She specializes in measurement, evaluation, and statistical analysis

Culturally inclusive instructional design (CIID) scale/instrument

Your Name: _____________________________Date: ______________________

The following 31 statements present educators’ perceptions and attitude toward culturally inclusive instructional design for online learning. Please rate your agreement with the following statements on a scale from strongly agree (7) to strongly disagree (1). Please refer to the Likert Scale key below as you make your selection.

7: Strongly Agree

6: Agree

5: Somewhat Agree

4: Neither agree nor disagree

3: Somewhat disagree

2 Disagree

1: Strongly disagree

The Likert Scale ranges from strongly agree (7) to strongly disagree (1)

Items	7	6	5	4	3	2	1
Humanizing learners begins by examining one's privileges
Each learner has their unique (overlapping) cultural identities
My identity (values and beliefs) influences the way I teach
Cultural differences are benefits in learning
Leveraging learners' cultural contributions (cultural wisdom, talents, resources) enriches learning
Learners' native language influences their communication in English
Communication styles (dialect, intonation, gestures) vary by cultures
Embed divergent cultural perspectives into the course
Support learners negotiate power-relations (e.g., help-seeking behavior) rooted in specific cultural contexts
Empower learners by leveraging their native cultural heritage
Develop (or use) relevant instructional materials where learners see themselves being represented
Utilize relevant Open Educational Resources (OERs) for equitable access
Implement inquiry-based collaborative learning that leverages diverse cultural perspectives for problems of practice
Learners explore meaning situated in their unique cultural context
Learners utilize mind-mapping tools to visualize connections
Learners integrate cultural resources in their collaborative inquiry
Learners synthesize knowledge through critical reflection
Learners negotiate (affirm) their collaborative meaning making
Learners create (preserve) their collective knowledge as a finished product
Learners feel comfortable sharing multiple perspectives in the learning community
Contribution of learners' cultural resources creates a sense of belonging
Honoring learners' cultural gifts/talents creates social comfort
Cultivating comfortable social space supports collaboration
Collaboration is cultivated for a shared purpose
The learning community leverages its wisdom for the social good (shared vision)
Clarity of communicating course expectations (e.g., cultural norms and help-seeking) enhances learning
Providing timely, constructive feedback scaffolds academic progress
Using equitable, inclusive technology (accessible) scaffolds learning
Facilitating learner autonomy with choice and flexibility addresses diverse learner needs
Facilitating reflection (metacognition) supports self-regulated learning
Leveraging cultural perspectives in learning activities motivates learners

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.