EFL learners’ engagement in online learning context: development and validation of potential measurement inventory

Abstract

Research has proved learner engagement is a strong predictor of academic achievement, especially in the online learning environment. The lack of any reliable and valid instrument to measure this construct in the context of online education made the researchers of the current study develop and validate a potential measurement inventory to assess EFL learners' engagement in the online learning environment. For this purpose, a comprehensive review of the related literature and careful investigation of the existing instruments were carried out to find the theoretical constructs for learner engagement which led to the development of a 56-item Likert scale questionnaire. The newly developed questionnaire was piloted with 560 female and male EFL university students selected based on nonprobability convenience sampling. The results of the factor analysis indicated the reduction of items to 48 loaded on three main components, namely behavioral engagement (15 items), emotional engagement (16 items), and cognitive engagement (17 items). The results also revealed that the newly developed questionnaire enjoyed a reliability index of 0.925. The findings of the current study will undoubtedly help teaching practitioners assess EFL learners’ engagement in the online learning context and make principled decisions when it comes to learners' engagement.

Introduction

The past decades have witnessed a revolution in the education system due to the rapid advancement of information technologies that resulted in the integration of different media into the computer system and the application of digital technologies and virtual communication in teaching and learning. Now, more people and institutions have access to computers and technologies thanks to this advancement. Not only these technologies are used for presentation and practice, but also interaction and collaboration. Moreover, the implementation of computers into educational contexts to promote the efficacy of teaching and learning gained attention. As a result of this growing use of computers, the Internet, and different forms of distance learning, language educators and learners express an increasing interest in the use of computers as a supportive and facilitator tool for language learning [86].

Recently, the COVID-19 pandemic leads to schools and universities being closed all over the globe. Consequently, there is a dramatic change in education with the widespread use of online learning whereby education occurs remotely via various digital platforms. Therefore, this pandemic results in the dire need of applying technologies such as laptops, tablets, smartphones, and Wi-Fi connections in education [46]. Online learning provides learners with opportunities to learn a language more independently [83]. To build such fruitful and efficient online learning, different factors come into play, such as learners’ engagement [67], which is integral in the process of teaching and learning since it is a way of encouraging students’ active participation in learning-related activities, sharing ideas and experiences and accomplishing collaborative tasks [66].

The concept of learner engagement has gained increased attention over the last decades. Despite the popularity of this topic, there is a lack of consensus on the conceptual definition of learner engagement and its types [20]. In the language education field particularly, questions remain about why this construct is so critical and how it can be developed [20, 70, 74, 76]. In spite of the restrictions in the existing literature, there is constant evidence that engagement is closely linked to successful learning [43]. Learner engagement is a multidimensional construct defined as learners’ active participation, involvement, and emotional quality in learning-related activities [20]. In the online education context, student engagement deals with the energy and time invested by the students in the online learning process [55]. Several studies considered engagement as one of the strong predictors of students’ academic accomplishment [93]. The multidimensional construct of engagement has been defined as having three or four dimensions [9]. Most scholars theorized the construct of student engagement as three separate but inter-reliant dimensions, comprising behavioral, emotional, and cognitive engagement [31, 44, 54]. In the online learning environment, behavioral engagement reflects learning-related activities such as participating in online interactions and communications and asking questions, cognitive engagement is concerned with mental efforts that learners make to learn a particular skill or acquire intricate knowledge, while emotional engagement deals with students’ positive emotional feelings about their class, peers and teachers, and their online courses [54]. Highly engaged learners are more successful in absorbing knowledge and skills. Therefore, it is crucial to promote learners’ engagement, especially in online classes [58].

Considering the significance of learners’ engagement and its important role in academic achievement, it is worth investigating learners’ level of engagement. For doing so, a valid and reliable instrument is needed. Researchers have utilized various instruments for measuring learners’ engagement such as portfolios, semi-structured interviews [27], observation [3, 47], course and professor evaluation [41]. However, implementing these instruments for investigating learner engagement in an online education environment is not feasible and they are mainly utilized for providing qualitative data. Hence, an instrument that is easily applicable in an online learning context that provides quantitative data is required. Numerous researchers have developed and validated instruments for evaluating learners’ engagement [6, 8, 10, 22, 32, 37, 48, 50, 56, 96, 100, 101]. However, some of these instruments focus on measuring just one or two learner engagement dimensions (e.g., [6, 48]). Most of the instruments investigate learners' general engagement (e.g., [8, 96]) instead of engagement in specific subject areas (e.g., [50, 100]) such as foreign language learning or science. In addition, these instruments are developed to measure school students’ engagement in actual and face-to-face classes [10, 101] rather than in the online learning environment.

A plethora of research proved the domain specificity of motivational constructs, especially situation- and subject-relevant ones [102]. Some primary studies also suggest that student engagement is domain-specific; however, more research is required to identify how this concept varies across subject areas [57]. Sinatra et al. [90], for instance, stated that epistemological cognition, participation in science and mathematics activities, attitudes, and topical reactions are domain-specific features of science engagement that are essential to take into account.

Despite the studies mentioned above, learners’ engagement is not fully investigated in the online educational context due to the lack of reliable and valid instruments for assessing this construct in online learning. Therefore, the goal of the current attempt is to create and validate an instrument to evaluate learners' engagement in the online educational setting. The novelty of this study not only relies on developing and validating a multidimensional questionnaire exploring learners’ behavioral, emotional, and cognitive engagement but also focuses on assessing learners’ engagement in the context of online education as one of the main educational concerns in today’s world affected by the COVID-19 pandemic.

Literature review

Learner engagement

The concept of learner engagement has gained increasing attention in educational literature in recent years. Relied on educational psychology, learner engagement reflects learners’ behavior and their psychological relationships with education or established learning [72]. Learner engagement is also defined as learners’ attention, interest, and self-confidence during learning or receiving instruction. To put it another way, engagement is regarded as a motivational behavior revealing the learners’ motivation for learning and knowledge enhancement [77].

Empirical studies revealed a positive correlation between learners’ engagement and academic performance and achievement [31, 72, 81], and learners who involve actively in tasks planned by educational institutions advantage more than those who do not do as much [35, 51, 72]. Accordingly, Kidwell [49] asserted that “If students are not engaged in the learning process, all the testing, data analysis, teacher meetings and instructional minutes will not motivate students to learn” (p. 30).

Scholars have introduced various typologies of engagement and some researchers have employed a twofold engagement typology, recognizing the behavioral and emotional dimensions of this concept [98]. The twofold engagement typology acknowledged by Finn [28] was established based on his participation-identification model. According to this model, learner engagement reflects learners’ active involvement and participation in classroom activities, along with their sense of treasuring and belonging to educational institutions.

Other scholars theorize learner engagement as threefold, including behavioral, emotional, and cognitive dimensions in which each dimension correlates with the other, and the three create a single compound construct. In the same vein, Fredricks et al. [31] described learner engagement as “a multi-dimensional construct of motivation that includes three interrelated components: behavioral, emotional, and cognitive” (p. 305). This suggests that these three dimensions should be considered while examining learners’ engagement. According to Fredricks et al. [31] “behavioral engagement involves doing work and following the rules, emotional engagement incorporates interest, value, and emotions; and cognitive engagement includes motivation, effort, and strategy use” (p. 306).

Moreover, Philp and Duchesne [76] conceptualized a task engagement model including four dimensions: cognitive, emotional, behavioral, and social. While cognitive engagement consists of procedures like constant attention, emotional engagement reflects the expression of different emotions such as eagerness, interest, satisfaction, dissatisfaction, apprehension, and frustration. Behavioral engagement is described as continuing task as demonstrated by the extent of on-task talk, whereas social engagement refers to the degree of interaction and reciprocality between learners while doing task-based interaction [17].

Additionally, Dunleavy [25] examined Canadian learners’ engagement and conceptualized learner engagement as a three-dimensional construct including (1) behavioral (e.g., participation and attendance in school activities, attendance), (2) academic-cognitive (e.g., doing homework, responding to learning-related challenges), and (3) social-psychological (e.g., interest, motivation, sense of belonging). Furthermore, Reschly and Christenson [78] proposed a four-dimensional model of learner engagement comprising academic, behavioral, cognitive, and psychological which is not a common model. However, Appleton et al.’s [7] meta-analysis of numerous studies on learner engagement revealed that learner engagement is a meta-construct including multiple facets which have been mostly conceptualized as behavioral, emotional, and cognitive dimensions.

Theoretical framework

Fredricks et al.’s [31] three-dimensional model of learner engagement is adopted as the theoretical framework of the current study since this model is considered a more appropriate one in the language learning area due to the combination of behavioral, emotional, and cognitive dimensions that almost embraces some most-researched fields in language instruction research, such as motivation, learning strategies, affective factors, and cognitive qualities [104]. Fredricks et al. [31] analyzed various definitions and taxonomies of engagement and recognized three major dimensions of engagement: behavioral, emotional, and cognitive. These dimensions are defined and explained meticulously in the following section.

Behavioral engagement

According to the threefold model of learner engagement, behavioral engagement reflects learners’ active participation in learning-related tasks, such as doing homework or asking questions in class, attending in class, and showing a decent manner that relies on the notion of involvement, association with academic activities, and successful academic accomplishments [12]. Other researchers explain behavioral engagement in a positive manner, such as not showing troublemaking behavior, abiding by the rules, and following the classroom norms [29, 30].

Instances of behavioral engagement in language learning contexts contain learners’ intentional participation in speaking, interactional inventiveness, time on task, and perseverance on task without asking for support or guidance [76]. While all areas of engagement include some extent of action, current studies consider behavioral engagement as the amount of endeavor that learners make in learning activities, their involvement quality, and their active participation level in the process of learning [40]. Research that focused on learners’ behavioral engagement indicated that learners can either show positive behaviors (obeying the classroom rules), which are signs of a high level of learners’ behavioral engagement, or they can show negative behaviors (being troublemaking in the classroom), which are signs of low level of behavioral engagement or disengagement [71].

Considering the positive impact of behavioral engagement on learners’ achievement [89, 99], and on reducing learner dropout rates [80, 103], a considerable number of studies addressed recognizing the features of classrooms that are related to behavioral engagement. Two school-level features, namely school size and firm rules, have an excessive deal of research on their connection with learner behavioral engagement [88]. Though schools have a vital impact on learner behavioral engagement, engagement fluctuates by classroom within a school as well [14]. Consequently, numerous studies focused on finding classroom instructional factors that may affect learners’ behavioral engagement such as the quality and quantity of learners’ interactions with the teacher, peers, and content [71].

Emotional engagement

The second dimension of learner engagement is emotional engagement which deals with learners’ emotional reactions in the learning process. It is also defined as students’ feelings of interest, happiness, anxiety, and anger while accomplishing activities related to learner attainment [91]. That is to say, emotional engagement is the degree to which learners show a sense of concern about their learning and education [18] and their attitude toward school, teachers, and classmates [104]. It is apparent that actual emotional engagement impacts learners’ inclination to do the activity and creates an emotional attachment to the educational context [13].

In second language learning contexts, emotional engagement is often displayed in learners’ affective responses when they take part in language learning-related activities. Learners who are engaged emotionally are considered as having a “positive, purposeful, willing, and autonomous disposition” toward language-related learning activities, and peers [94], p. 247]. Declaration of positive emotions such as satisfaction, interest, and eagerness are supposed to be signs of learners’ emotional engagement; however, negative emotions such as apprehension, tediousness, frustration, and irritation reveal emotional disengagement [61].

Emotional engagement has a crucial effect on other dimensions of learner engagement due to the importance of subjective perceptions or attitudes that learners have during class activities or through language-related tasks [39]. Emotional engagement is, consequently, associated with learners’ points of view toward educational settings, the individuals in those settings, the learning activities, and their involvement in learning [40]. Pekrun [75] stated that emotions affect “a broad variety of cognitive processes that contribute to learning, such as perception, attention, memory, decision making, and cognitive problem solving” (p. 26), and Skinner and Pitzer [92] considered emotion as “the fuel for the kind of behavioral and cognitive engagement that leads to high-quality learning” (p. 33).

Cognitive engagement

Cognitive engagement is the third dimension of learner engagement that reflects implementing various learning strategies and self-regulation [20, 104]. Cognitive engagement is considered a mediator variable that has a positive and direct impact on the learners' academic achievement. It is also recognized as strategic and thoughtful to apply the required effort for comprehension of intricate ideas [34].

Cognitive engagement deals with learners’ mental energy, activity, and effort in the learning process. Learners are cognitively engaged when they show careful, discerning, and constant attention to accomplish a given activity or learning objective [40]. In the context of second or foreign language learning, studies on learner cognitive engagement focus chiefly on verbal manifestations such as peer interactions, learners’ questioning, reluctance and repetition, offering answers, expressing ideas, giving feedback, directing, and explaining. Along with such negotiation of meaning, other scholars consider language-related incidents as appropriate indicators of learners' cognitive engagement [52, 95]. Private speech, nonverbal communication, and exploratory talk that happen when learners try to make sense of learning are regarded as other indicators of this dimension. Besides these more concrete communications prompts, it is also likely to examine cognitive engagement via nonverbal signals such as eye movements, facial expressions, body language, and body positioning [40].

Online language learning

During the last 2 years, there has been a dramatic change in education due to the COVID-19 pandemic leading to the closure of educational institutions all over the globe and the sudden rise of online learning through which instruction takes place virtually on various digital platforms. Even before the COVID-19 outbreak, online learning had been a growing business in its own right in various educational fields and language learning was not an exception [82]. It is worth distinguishing between online learning approaches such as Massive Open Online Courses (MOOC) and Virtual classes. Although MOOCs have gained popularity over the past few years, the COVID-19 pandemic has made virtual classes almost universally adopted since they can meet a variety of pressing demands that arose as a result of the limits that were put in place. The name MOOC was first used by Cormier in 2008 and includes three key characteristics: first, they are open in that anyone may enroll and access the information, second, they are online courses, and third, they are massive in terms of the number of students they can accommodate [1]. According to another description, MOOC is an online course that offers free registration to users and has a publicly available curriculum with open-ended learning objectives [59].

On the other hand, virtual classes can be delivered in one of two ways: synchronously or asynchronously. In asynchronous virtual education, learning resources and materials are delivered asynchronously via platforms for learning management systems (LMS), such as Moodle or BB. The use of resources like email, discussion boards, blogs, wikis, or video/audio recordings can facilitate asynchronous online communication, which does not require the instructor and students to participate in real-time [42], whereas delivering course materials in real time is a part of the synchronous mode. Using synchronized software, students and teachers can converse verbally, type messages back and forth, upload PowerPoint presentations, send videos, or browse the web simultaneously [60]. Synchronous online learning, which is frequently supplemented by chat and video conferencing, is said to have the ability to aid e-learners in the formation of learning communities. It provides whiteboard, streaming video and audio, and file sharing [15, 79]. The synchronous conferencing sessions' nonverbal cues help learners feel more connected to and engaged with their classmates and teachers [4].

The context of online learning provides learners with flexible and self-paced learning without place and time restrictions [64, 65, 68] which results in the promotion of knowledge construction and the creation of online learning atmospheres that boost meaningful learner engagement [82]. Learning language, similar to other educational goals, is far more fruitful when joined with proper educational technology. The growing application of modern technology in educational contexts contributes to connecting teachers to learners, to resources, to professional content, and to systems planned to enhance their lessons and personalize instruction [62]. Research revealed that educational programs that take place in an online atmosphere are preferred by teachers because they seem to correlate with positive learners’ engagement with the teachers, and educational material and boost language learning [2, 46]. The main benefits of online classrooms are their flexibility, communication, and support for those with restricted movement or lack of resources. Such classes can enhance learners’ engagement and provide them with more opportunities for interaction along with their sense of community [38]. However, learners and teachers may encounter some challenges in online education such as (1) adaptability of teachers and learners (transition from traditional face-to-face education to online education may result in some struggle for teachers and learners) [73], (2) the lack of technological literacy (i.e., teachers and learners lack required knowledge and skills in applying computer and software in online classes) [53, 73], (3) the lack of technological infrastructure especially in developing countries [5], (4) the cost of online education, (5) classes time management (i.e., the online class takes a lot of time for teaching and assignment) [5, 36], (6) lack of motivation (i.e., both teachers and learners are less motivated in online classes due to the less active process of this classes compared with face-to-face education [5, 36, 87], and (7) learners’ engagement (i.e., students’ nature of engagement in online classes is considered the main challenge. Although some students actively engage in discussions and arguments in online classes, others incline toward being reluctant and just listening and observing [53]. Therefore, for online language learning to be successful, it is essential to make sure that learners are adequately engaged. It is extensively recognized that engagement is connected to improved productivity and learning advance. Some studies indicated that engagement is flexible, and appropriate instructive involvements, learning designs, and feedback can develop learner engagement [19, 69].

Eventually, given the significance of learner engagement as a robust predictor of learner academic accomplishment especially in the online education context, the review of the related literature, and previous studies provided here, the nonexistence of a comprehensive way of assessing EFL learners’ engagement in online learning was the gap targeted for which a questionnaire was developed and validated in the course of the present investigation.

Method

Participants

The participants of this study were 560 Iranian undergraduate university students of English majors studying at Karaj Islamic Azad University. In this sample, 53% of the participants were female and 47% were male between the ages of 18 and 25 years. The participants were selected through convenience sampling [11] to include the participants who were available and willing to take part in this study.

Instruments

A questionnaire was designed to assess EFL learners’ engagement in online classes. To determine the theoretical framework of the study and to develop the items of the questionnaire, a comprehensive review of related literature and a critical investigation of the existing questionnaires were done. To illustrate the purpose of the newly developed questionnaire and the way to answer its items, clear instruction was provided, and the rating scale was clarified as well. The first draft of the questionnaire included 56 items on a five-point Likert scale rating from never (rated 1) to always (rated 5).

Some items were constructed according to the previous study conducted by one of the researchers of the current study [63] focusing on EFL learners' progress from limited to elaborate behavioral, emotional, and cognitive engagement through conversation analysis techniques such as item 13 (When I do a task in the class, I try to relate the task to the prior knowledge). Some other items such as item 37 (When I do well in the class it’s because I work hard.) were borrowed and modified from frequently used questionnaires such as Appleton et al. [6] that have been recognized with psychometrically valid properties. This step is very crucial not only for generating the items but also for supporting the enhancement of reliability and validity evidence [23]. Additionally, some items are created by the researchers of the current study to best suit the online language learning contexts such as item 12 (I fully charge my telephone, laptop, and internet to not disconnect during the class in order to lose the material). Reviewing the related literature comprehensively and investigating the existing learner engagement questionnaires critically, an initial pool of 56 items was constructed assessing three constructs: behavioral (items 1–17), emotional (items 18–36), and cognitive (items 37–56).

Procedure

The researchers of the current study followed a standard, step-by-step process to develop a questionnaire for assessing EFL learners' engagement in online classes. This process was initiated with meticulous scrutiny of the related literature on numerous variables about various perceptional, methodological, and systematic variables. Accordingly, it was essential to develop an appropriate, efficient, and flexible framework for learner engagement. Reviewing the literature on learner engagement, a three-dimensional model of learner engagement focusing on behavioral, emotional, and cognitive dimensions [31] was selected from among various available frameworks on learner engagement. In order to operationalize the indicators and collect the questionnaire items, various existing questionnaires on learner engagement were investigated considering the three dimensions of learner engagement. In addition, some items are created and some other items were modified by the researchers to concern the online learning context.

Taken all together, the initial number of 56 five-point Likert scale questionnaire items (Appendix 1) was developed based on a comprehensive review of related literature and a thorough investigation of available questionnaires. Items were written in straightforward language without complexity in order to avoid any inconsistency between participants’ interpretation and scale intention. Furthermore, for examining the content validity of the newly developed questionnaire, a panel of experts was asked to evaluate the items and their appropriateness and relevance to the objectives of the study. Following their viewpoints and modifications, the ultimate items for the inventory were selected.

Moreover, to examine the reliability and construct validity of the questionnaire, it was piloted with 560 participants. Because of the COVID-19 outbreak, an online questionnaire implementing the Google Forms platform was conducted and distributed via email or other social media along with a brief overview of the purpose of the study and a description of the instrument. Students were given enough time to answer and also opportunities to edit their responses. In the process of designing the questionnaire, care was taken to make the questionnaire and its item as comprehensive as possible in terms of language and wording. In addition, researchers of the current study provide the participants with their email addresses to send them email in case of any questions. The researchers did not observe any difficulty in administering and collecting the questionnaire developed to measure engagement for a course during a semester for 15 sessions each lasting for 2 h. The ease with and also the speed of responding to the questionnaire and the availability of the researchers to assist students any time they face problems or the need for clarification give us the vision that the questionnaire can be utilized and adapted widely. Finally, for estimating the reliability and construct validity of the instrument, Cronbach’s alpha and exploratory factor analysis were run employing SPSS software.

Design

As mentioned before, this study is an attempt to design and develop a reliable and valid instrument to assess EFL learners' engagement in the online learning context. A sequential exploratory mixed methods design was used in this process. The main goal of using mixed methods research is often to join qualitative and quantitative approaches in a variety of ways and utilize each method's strength within a single inquiry [16]. An exploratory sequential mixed methods study consists of two phases, the first of which is the collection and analysis of the qualitative data. The quantitative data are then gathered and examined in the second phase using the findings of the previous phase. That is to say, a three-phase method should be used to create and construct a questionnaire, which includes looking at the qualitative data, creating the instrument, and then administering the instrument to a sufficient number of participants [16]. The design is demonstrated in the following figure (Fig. 1).

Fig. 1.

Exploratory sequential mixed methods [16], p. 220]

Results

Reliability

This study aims at exploring the reliability and construct validity of the online classroom engagement questionnaire (OCEQ). The OCEQ has 56 items measuring three constructs of behavioral (items 1–17), emotional (items 18–36), and cognitive (items 37–56). Table 1 displays Cronbach’s alpha reliability for the overall OCEQ and its three components. The overall OCEQ enjoyed a reliability index of 0.897. The reliability indices for the behavioral, emotional, and cognitive engagements were 0.881, 0.894, and 0.897 respectively.

Table 1.

Reliability statistics: online classroom engagement and its components

	Cronbach's Alpha	No. of items
Overall OCEQ	0.897	56
Behavioral	0.881	17
Emotional	0.894	19
Cognitive	0.897	20

It should be noted that Tseng et al. [97], Dörnyei and Taguchi [24] believe that 0.70 is the adequate reliability index for an instrument. George and Mallery [33] believe that “there is no set interpretation as to what is an acceptable alpha value. A rule of thumb that applies to most situations is: 0.9 = excellent, 0.8 = good, 0.7 = acceptable, 0.6 = questionable, 0.5 = poor and 0.5 = unacceptable” (p. 244). Based on these criteria, it can be concluded that the reliability indices (Table 1) for the OCEQ and its components were almost excellent.

Table 2 displays the item–total correlations for the three components of OCEQ. The results showed that all items had large, i.e., ≥ 0.50, contributions to their constructs except for: items 8 and 12 (behavioral engagement), items 10, 14, and 17 (emotional engagement), and items 9, 13, and 17 (cognitive engagement).

Table 2.

Item–total correlations: components of online classroom engagement questionnaire

Behavioral	Item–total correlation	Emotional	Item–total correlation	Cognitive	Item–total correlation
Beh1	0.692	Emo1	0.642	Cog1	0.715
Beh2	0.633	Emo2	0.661	Cog2	0.627
Beh3	0.631	Emo3	0.667	Cog3	0.679
Beh4	0.698	Emo4	0.674	Cog4	0.629
Beh5	0.626	Emo5	0.714	Cog5	0.745
Beh6	0.617	Emo6	0.662	Cog6	0.700
Beh7	0.643	Emo7	0.642	Cog7	0.642
Beh8	0.035	Emo8	0.616	Cog8	0.646
Beh9	0.615	Emo9	0.658	Cog9	0.033
Beh10	0.676	Emo10	-0.079	Cog10	0.699
Beh11	0.667	Emo11	0.625	Cog11	0.751
Beh12	− 0.016	Emo12	0.659	Cog12	0.691
Beh13	0.611	Emo13	0.715	Cog13	0.048
Beh14	0.674	Emo14	0.037	Cog14	0.727
Beh15	0.619	Emo15	0.691	Cog15	0.657
Beh16	0.632	Emo16	0.654	Cog16	0.728
Beh17	0.650	Emo17	0.040	Cog17	− 0.033
		Emo18	0.710	Cog18	0.742
		Emo19	0.693	Cog19	0.653
				Cog20	0.743

Table 3 displays Cronbach’s alpha reliability indices after dropping these defective items out. The reliability index for the overall OCEQ increased to 0.925. The reliability indices for the behavioral, emotional, and cognitive engagements increased to 0.921, 0.936, and 0.940 respectively. All these reliability indices can be considered excellent based on the criteria discussed above.

Table 3.

Reliability statistics: online classroom engagement and its components (after removing defective items)

	Cronbach's Alpha	No. of items
Overall OCEQ	0.925	48
Behavioral	0.921	15
Emotional	0.936	16
Cognitive	0.940	17

Construct validity

Tables 4, 5, and 6 display the results of the construct validity of the OCEQ. Based on the results displayed in Table 4, it can be concluded that the present sample size of 560 was adequate for running factor analysis (KMO = 0.944 > 0.60) and the significant results of the sphericity test (χ² (1540) = 15,613.78, p < 0.05); the correlation matrix based on which the factors were extracted was a factorable one.

Table 4.

KMO and Bartlett's test: online classroom engagement questionnaire

Kaiser–Meyer–Olkin measure of sampling adequacy	0.944
Bartlett's test of sphericity
Approx. Chi-Square	15,613.781
df	1540
Sig.	0.000

Table 5.

Total variance explained: online classroom engagement questionnaire

Factor	Initial eigenvalues			Extraction sums of squared loadings			Rotation sums of squared loadings
	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %
1	11.460	20.464	20.464	11.005	19.651	19.651	9.062	16.182	16.182
2	7.666	13.690	34.154	7.217	12.888	32.539	8.009	14.301	30.483
3	6.544	11.685	45.839	6.057	10.816	43.355	7.170	12.804	43.287
4	1.253	2.238	48.077	0.501	0.894	44.249	0.483	0.862	44.149
5	1.202	2.146	50.223	0.451	0.805	45.054	0.443	0.792	44.941
6	1.150	2.053	52.277	0.407	0.727	45.781	0.434	0.775	45.716
7	1.081	1.931	54.208	0.378	0.675	46.456	0.392	0.699	46.415
8	1.062	1.897	56.105	0.364	0.650	47.106	0.387	0.691	47.106

Extraction method: principal axis factoring

Table 6.

Rotated factor matrix: online classroom engagement questionnaire (after removing items with low contributions)

	Factor
	1	2	3
Cog14	0.774
Cog11	0.772
Cog5	0.772
Cog18	0.766
Cog20	0.757
Cog16	0.755
Cog1	0.750
Cog6	0.745
Cog10	0.728
Cog12	0.725
Cog3	0.708
Cog19	0.693
Cog15	0.688
Cog8	0.675
Cog7	0.674
Cog4	0.661
Cog2	0.652
Emo13		0.742
Emo5		0.740
Emo18		0.738
Emo19		0.736
Emo15		0.731
Emo4		0.719
Emo3		0.703
Emo2		0.702
Emo16		0.694
Emo6		0.691
Emo9		0.685
Emo12		0.681
Emo7		0.670
Emo1		0.662
Emo11		0.652
Beh4			0.728
Beh1			0.721
Beh10			0.720
Beh14			0.719
Beh11			0.704
Beh7			0.698
Beh17			0.690
Beh16			0.672
Beh6			0.668
Beh2			0.661
Beh3			0.654
Beh9			0.653
Beh5			0.652
Beh13			0.647
Beh15			0.646

Eight factors accounted for 47.10% of the variance (Table 5). That is to say, the 56 items of the OCEQ measured eight factors with an accuracy of 47.10% were extracted.

The results of the rotated factor matrix (Table1, Appendix 1) indicated that all items related to cognitive, emotional, and behavioral engagements loaded under the first, second, and third factors respectively, except for the following items whose factor loadings were lower than 0.30; cognitive (items 9, 13, and 17), behavioral (item 8), and emotional (items 10 and 14). Three items, i.e., emotional 8 and 17, behavioral 12, loaded under two factors.

After removing the items which had low contributions to their factors, or had loadings on more than one factor, the factor loadings became as displayed in Table 6. The items related to cognitive, emotional, and behavioral engagements are loaded under the first, second, and third factors. All factor loadings enjoyed large effect sizes; i.e., ≥ 0.50.

Discussion

The current study aimed to design and validate a measurement instrument to assess EFL learners’ engagement in the online learning context. After a comprehensive review of the related literature and scrutiny of the existing instruments, the theoretical constructs concerning EFL learners’ behavioral, emotional, and cognitive engagement were obtained and were later employed as the underpinning theoretical framework of the current attempt that resulted in the development of a valid and reliable questionnaire. To examine the psychometric qualities of the newly developed questionnaire, it was administered to 560 undergraduate university students studying English majors at Karaj Islamic Azad University. The results of Cronbach’s alpha revealed that the overall OCEQ enjoyed a reliability index of 0.897. The reliability indices for the behavioral, emotional, and cognitive engagements were 0.881, 0.894, and 0.897, respectively. Moreover, the results of the factor analysis revealed no construct-irrelevant factor, and all items contributed to their constructs except for items 8 and 12 (behavioral engagement), items 10, 14, and 17 (emotional engagement), and items 9, 13, and 17 (cognitive engagement). After eliminating these defective items, the overall OCEQ reliability index improved to 0.925. The reliability indices for the behavioral, emotional, and cognitive engagements increased to 0.921, 0.936, and 0.940 respectively.

In spite of the existence of numerous instruments for assessing learners’ engagement, some instruments that measure engagement, such as the UWES developed by Schaufeli and Bakker [84] have been criticized due to the lack of appropriate construct definitions and dimensionality to their application to university students. The results of the present study are supported by those of a study conducted by Appleton et al. [6] who developed the Student Engagement Instrument (SEI), which is one of the most prominent instruments utilized for measuring learner engagement. However, this 35-item instrument is a four-Likert scale planned to assess just two dimensions of school students' engagement namely cognitive and psychological engagement.

The outcomes of the present study are in line with Gargallo et al.’s [32] study in which the Student Engagement Questionnaire developed by Kember and Leung [48] including 17 items on 6 points Likert scale was adopted to assess the behavioral dimension of the classroom. This instrument was validated by Gargallo et al. [32] with a Spanish sample of 805 university students. This questionnaire was utilized for two main purposes: (1) to analyze university students’ capabilities to acquire during the course of study and (2) to evaluate the learning environment created by the teacher in class to facilitate the learning process. Therefore, this inventory focuses on just one aspect of learner engagement, behavioral engagement.

The results of the current study are in agreement with those of previous studies concerning the main factors of learner engagement such as Balfanz [10] who designed the High School Survey of Student Engagement which includes 121 items measuring three dimensions of cognitive engagement (65 items), behavioral engagement (17 items), and emotional engagement (39 items) of secondary school students. Despite the wide use of this instrument, it lacks information on its reliability and validity. Similarly, Maroco et al. [56] developed the University Student Engagement Inventory (USEI) drawing on the conceptualization of engagement as a multifaceted concept, comprising emotional, cognitive, and behavioral engagement. The participants of this study were 609 Portuguese university students majoring in Social Sciences, Biological Sciences, and Engineering. The initial USEI included 35 items but the validated USEI was composed of 15 items loaded on three main components of learner engagement.

All the instruments mentioned above were designed and developed to measure learners’ engagement in actual and face-to-face classrooms. Despite the importance of learners’ engagement in the online learning context, there are very few instruments for assessing this crucial construct in the context of online education. The results of the current study are also consistent with those of the study conducted by Dixson [22] in which the online Student Engagement scale (OSE) was developed. The OSE includes 19 items of a 5-point Likert scale that reports how well thought, behavior, and the feeling were characteristic of them or their behavior. This instrument was utilized to find the correlation between learners' self-reports of engagement and following data of learner behaviors in an online course. According to the findings, learner engagement on the OSE significantly correlated with two types of student behaviors: observational learning behaviors and application learning behaviors.

As stated earlier, although numerous investigations were conducted with a focus on learner engagement, no studies were carried out to design and validate a questionnaire utilizing which EFL learners’ engagement in the online learning context can be evaluated. Therefore, explorations such as the current one will be very beneficial to enable those who are responsible for improving learners’ engagement in online learning environments.

Conclusion

Learner engagement and academic achievements are so inevitably intertwined that is really hard to conceptualize learners’ accomplishments without considering the crucial role of learner engagement, especially in the online learning context where learners may feel isolated. In this regard, teaching practitioners are required to be able to measure learners’ level of engagement and enhance it in order to achieve desired learning outcomes. Consequently, developing and validating an assessment inventory for assessing EFL learners’ engagement in the online educational context lies at the heart of the current attempt. This study resulted in the development of a questionnaire with a final number of 48 items across three components namely behavioral, emotional and cognitive engagement which were scored on a five-point Likert scale ranging from 1 to 5 representing the extent to which the statement was reckoned to be true for the respondent learner.

The results of the current attempt will potentially be beneficial for several stakeholders in the fields of English language teaching and learning. Teachers, supervisors, administrators, course designers, curriculum and materials developers, and policymakers all may take advantage of the results of this study since they will be equipped with a reliable and valid instrument to assess EFL learners’ engagement in the online learning environment. To achieve the desired learning outcomes, teachers, for instance, will be able to measure their students’ level of engagement during the online course of study and provide learners with more engaging activities and tasks. Considering the importance of the content of materials in online courses, course materials developers will also be cognizant of Iranian EFL learners’ level of engagement to develop more engaging materials. Administrators and policymakers can use this questionnaire as a yardstick to evaluate the efficiency of the online learning environment when it comes to learner engagement.

However, the researchers of the current study tried to follow established methods to develop and validate this measurement inventory, some caveats need to be noted regarding this study. First, even though both inductive and deductive approaches were implemented to substantiate the conceptualization of learner engagement, some intervening matters were not controlled due to time and practical restrictions. For instance, including a larger sample size or carrying out the research countrywide could result in better outcomes. In addition, the participants of this study were selected through convenience sampling which is considered a limitation. Therefore, other scholars can replicate and obtain a better understanding of learner engagement in the online learning environment and more generalizable results by implementing other sampling approaches. Moreover, the qualitative part of the present study was mainly carried out through a deep review of the related literature and a careful investigation of the existing instruments. The lack of data triangulation and usage of different sources of data collection is another limitation. The use of various data collection sources can open up new research horizons and reveal more research areas. Future studies can reflect on and identify various aspects of learner engagement in the online learning context by observing their classes or providing them with focus group discussions.

Furthermore, it should be noted that individual differences are not considered in this study since it was carried out during the COVID-19 outbreak. Hence, some research procedures such as piloting the questionnaire were conducted online. Consequently, paying attention to the participants’ individual differences was not possible. Therefore, the findings of future studies would certainly complement those of the current study in terms of controlling the impact of individual differences and clarifying how these features may contribute to learners’ engagement in the online learning environment.

Appendix 1

Learner Engagement in Online Learning Questionnaire

	Statements	1	2	3	4	5
Behavioral engagement
1	I treat my classmates and teachers with respect either in Skyroom or in the WhatsApp group	1	2	3	4	5
2	I comply with the rules in the class by doing all my homework and being punctual	1	2	3	4	5
3	I work with great effort to answer the teacher’s questions by typing in the chatbox or by clicking the Raising Hand icon	1	2	3	4	5
4	When I encounter a hard homework exercise, I continue working on it till I think I’ve answered it	1	2	3	4	5
5	I participate actively in the activities such as presenting a lecture and participating in class discussions by asking the teacher to turn on my microphone	1	2	3	4	5
6	When I’m in class, my mind wanders	1	2	3	4	5
7	I discover that when the teacher is teaching, I think of other things and don't really listen to what is being said	1	2	3	4	5
8	When I am in class, I listen carefully	1	2	3	4	5
9	I turned in a homework assignment late or not at all	1	2	3	4	5
10	I never use my mother tongue in class, I keep speaking English	1	2	3	4	5
11	When I do not understand something in class, I request the teacher to clarify it	1	2	3	4	5
12	I fully charge my telephone, laptop, and internet to not disconnect during the class in order to lose the materials	1	2	3	4	5
13	When I do a task in class, I try to relate the task to prior knowledge	1	2	3	4	5
14	I tutored or taught the class materials to other students in the WhatsApp group after the class	1	2	3	4	5
15	When I participate in class discussions or present a lecture, I pause frequently to remember a word or I laugh	1	2	3	4	5
16	I fully attend the class by affirming the teacher by typing “yes” or “that’s right” in the chatbox	1	2	3	4	5
17	I speak fluently in class	1	2	3	4	5
Emotional engagement
18	I enjoy learning new things in online classes	1	2	3	4	5
19	The online class is one of my favorite places to be	1	2	3	4	5
20	When I run out of the internet or lose the connection, I worry a lot about it	1	2	3	4	5
21	I enjoy learning new things in class	1	2	3	4	5
22	When I have a project to do, I worry a lot about it not being disconnected	1	2	3	4	5
23	My teacher praises me most of the time when I work hard	1	2	3	4	5
24	I am interested in the work, discussions, and exercises in my class	1	2	3	4	5
25	When I’m not interested in the class I excuse myself for a camera or microphone	1	2	3	4	5
26	When the teacher wants to teach new things, I complain by typing “no, please”	1	2	3	4	5
27	I feel relaxed in doing exercises or during a discussion	1	2	3	4	5
28	I believe I can do it well during answering the exercise	1	2	3	4	5
29	I am all ears when my teacher teaches a new thing	1	2	3	4	5
30	My classmates always affirm my correct answers by typing “yes” or “very good”	1	2	3	4	5
31	Students in my class are there for me when I need them	1	2	3	4	5
32	I believe in myself when I want to present a lecture	1	2	3	4	5
33	When I present a lecture, I need my teacher’s affirmation by saying “that’s right” or nodding her head while her camera is on	1	2	3	4	5
34	I Volunteer to present in the class by clicking on the Raising Hand icon	1	2	3	4	5
Cognitive engagement
35	Because I work hard, I succeed in class	1	2	3	4	5
36	I check my classwork to ensure that I have done it correctly	1	2	3	4	5
37	When I study for an exam, I attempt to combine the information from the book and the class	1	2	3	4	5
38	Finding the main ideas of a text is difficult for me	1	2	3	4	5
39	When my classmates do an exercise or present a lecture, I try to justify their answers or lecture	1	2	3	4	5
40	When I study or learn new things, I attempt to put the ideas in my own words	1	2	3	4	5
41	I make my own examples to understand the important concepts I learn from the class	1	2	3	4	5
42	When I study for a test, I repeat the important points several times	1	2	3	4	5
43	When I am studying a topic, I try to connect everything properly	1	2	3	4	5
44	When I am studying my coursebook, I try to outline its chapters to learn better	1	2	3	4	5
45	I try to give directions to my classmates when they do not know how to do the task	1	2	3	4	5
46	When lessons are difficult, I either stop studying or study only the easy parts	1	2	3	4	5
47	I work on practice exercises and answer the end-of-chapter questions even when I don't have to	1	2	3	4	5
48	Even when lessons are uninteresting, I continue studying till I finish	1	2	3	4	5
49	I try to use dictionaries or reference books in doing tasks and exercises	1	2	3	4	5
50	I prevent exchanging ideas with my peers or teachers during a discussion	1	2	3	4	5
51	I attempt to think carefully about topics and find what I’m expected to learn from them, instead of studying topics by just reading them over	1	2	3	4	5
52	When I am studying course materials, I try to join different pieces of information together in innovative ways	1	2	3	4	5
53	When I finish working on an exercise, I check my answer to see if it is reasonable	1	2	3	4	5
54	English class is important for achieving my future goals	1	2	3	4	5

1 = never or almost never; 2 = rarely; 3 = sometimes; 4 = often; and 5 = always or almost always.