Matters of Frequency, Immediacy and Regularity: Engagement in an Online Asynchronous Course

Abstract

Many models of online student engagement posit a “more is better” relationship between students’ course-related actions and their engagement. However, recent research indicates that the timing of engagement is also an important consideration. In addition to the frequency (how often) of engagement, two other constructs of timing were explored in this study: immediacy (how early) and regularity (in what ordered pattern). These indicators of engagement were applied to three learning assessment types used in an online, undergraduate, competency-based, technology skills course. The study employed advanced data collection and learning analytics techniques to collect continuous behavioral data over seven semesters (n = 438). Results revealed that several indicators of engagement predicted academic success, but significance differed by assessment type. “More” is not always better, as some highly engaged students earn lower grades. Successful students tended to engage earlier with lessons regardless of assessment type.

Higher education continues to see steady growth in distance education (Seaman et al., 2018), which has accelerated due to the COVID-19 pandemic (O’Keefe et al., 2020). Despite this growth, educators remain concerned about student success and a lack of engagement (Lee et al., 2015; Soffer & Cohen, 2019). Engagement has been defined as time and effort expended by students in their academic pursuits (Ma et al., 2015). When engagement is lacking, students can become isolated from teachers and peers (Nguyen et al., 2018) and dissatisfied with course design, instructors and workload (Stott, 2016), which are associated with higher dropout rates (Lee et al., 2015). In contrast, higher levels of engagement have been associated with course completion (Cohen, 2017; You, 2016), higher exam grades (Soffer & Cohen, 2019), and increased self-regulation (Sun & Rueda, 2012).

Given the importance of engagement, researchers have explored a variety of indicators designed to track engagement in online settings. Examples of these indicators include the number of questions asked (Henrie et al., 2015) or assignments submitted (Soffer & Cohen, 2019), as well as writing and reading messages (Li & Tsai, 2017; You, 2016). These indicators suggest that the quantity of course-related actions such as questions asked or assignments submitted provide insight into engagement. Typically, indicators of engagement focused on quantity posit a “more is better” relationship between course-related actions and engagement. In other words, more course-related student output indicates more engagement.

In addition to quantity, recent research indicates that the timing of engagement is a significant consideration (Tang et al., 2019). In this context, the timing of engagement refers to when students take initiative regarding course assignments and related work. Two constructs related to this form of timing are immediacy and regularity. Immediacy describes how early students engage with course assignments (Wong & Chong, 2018). For example, You (2016) found that students who show high levels of engagement within the first four weeks perform better. Regularity refers to what ordered pattern forms as students engage with a course, and has been shown to be a significant predictor of success (Conijn et al., 2017; Tang et al., 2019; Wong & Chong, 2018). In short, indicators of engagement focused on timing provide another way of understanding engagement in online contexts.

Applying indicators of quantity and timing to the study of engagement in online contexts was difficult in the past for logistical and technical reasons. Today, however, advanced data collection and learning analytics techniques can be integrated into online courses, providing continuous data capture for an entire course (Cipher et al., 2018; Soffer & Cohen, 2019; Wong & Chong, 2018; You, 2016). The current study leveraged these techniques to investigate the relationship between various indicators of engagement and student success online. Seven semesters of data from an asynchronous course were analyzed to determine how the amount of engagement (how often), the immediacy of engagement (how early), and the regularity of engagement (in what ordered pattern) predicted success as measured by course grade. In addition, these indicators were applied to different types of lessons used in the course, a decision motivated by evidence that course-level characteristics such as lesson type are associated with behavioral engagement (Nguyen et al., 2018). The goal of the study was to better understand the relationships between indicators of engagement, learning assessment types, and student course performance. Insight into these relationships may help researchers, instructors and instructional designers create new ways of promoting and sustaining engagement in online courses.

Background

Engagement in Online Courses

Engagement is referred to as commitment, participation or involvement (Fredricks et al., 2004), and in academic contexts, as time and effort expended by students in their academic pursuits (Ma et al., 2015). While engagement has been theorized as a complex meta-construct encompassing cognitive, behavioral and emotional aspects (Fredricks, 2011; Fredricks et al., 2004), this study addresses these as a collective quality that produces “learners’ participation in learning, including effort, persistence, and concentration” (D’Mello et al., 2017, p. 105).

Higher levels of engagement generally predict better academic performance (Soffer & Cohen, 2019; Li & Tsai, 2017) found that learners engaged in an online course fell into one of three activity clusters: consistent use, intensive use and less use. Consistent use students, those most actively engaged with course materials, outperformed the other two clusters. Similarly, students with higher levels of engagement, such as overall time spent in an online course (Cipher et al., 2018), activity in forums (Tang et al., 2018), regular study and frequency of course logins (You, 2016), performed better, whereas low activity levels (Cohen, 2017) were associated with course dropout.

Research examining student perspectives on engagement emphasizes its importance and variability in online contexts. For example, many students report struggling with engagement in online courses (Stott, 2016; Sun & Rueda, 2012; You, 2016). Still other research suggests students need to feel their course-related activity is important and that they are learning and improving (Manwaring et al., 2017). Related research has found engagement to be malleable (Fredricks et al., 2004) and influenced by course characteristics such as the instructor’s course preparation, assistance, and a student’s own course viewing activities (Ma et al., 2015).

Indicators of Timing

Higher levels of engagement, typically measured through indicators of quantity, have been found to predict student success (Conijn et al., 2017; Soffer & Cohen, 2019). However, research on temporal effects or indicators of timing may offer more nuanced views (Canal et al., 2015; Knight et al., 2015; Nuhfer, 2005a, b; Reimann, 2009; Tang et al., 2018, 2019). The rationale for this premise is the idea that learning unfolds over time (Reimann, 2009) and that the timing of events can reveal details about the learning process (Tang et al., 2018). For instance, Cerezo et al., (2017) found that indicators of timing, such as the time taken to complete tasks and when content was accessed (i.e., how early or late as measured in days), predicted course completion. Asarta & Schmidt (2013) found that the frequency of accessing content was less important to student success than timing and consistency. Some experts assert that research on learner performance without the study of time may be methodologically flawed (Barbera et al., 2012) because it assumes a constant state (Reimann, 2009) and captures only a cross section of activity (Barbera et al., 2012; Canal et al., 2015). Related work has highlighted important perspectives on the temporal aspects of learning, distinguishing between constant, cyclic and fractal patterns of change (Nuhfer, 2005a, b).

Indicators of timing in online courses have been studied using various measures. Examples include the amount of time spent on a course website, and the order and sequence of actions taken throughout a course (Reimann, 2009; Wong & Chong, 2018) found that engagement could be measured by duration, immediacy and regularity. One form of immediacy (how early) is the measure of time elapsed between when courses begin and when students first access those courses, which has been found to be a significant predictor of success (Cipher et al., 2018; Conijn et al., 2017). Similarly, regularity of study, a pattern of relatively smaller intervals of time between engagements, reflects time-management behaviors shown to predict success (Asarta & Schmidt, 2013; You, 2015, 2016) and course completion (Cohen, 2017). Regularity of engagement is important because many online learners struggle with time-management issues like procrastination (Cerezo et al., 2017), suggesting indicators of timing such as regularity of study are worthy of investigation (You, 2016).

Learning Analytics and Learning Record Stores

Self-reporting has been a common method for collecting data about engagement in online courses (Henrie et al., 2015). However, in more recent years, advances in technology have made it possible “to track and store students’ learning activities” (Avella et al., 2016, p. 14) without the need for self-report measures. In online courses, it is common to capture and record activities such as student logins, clicks, page views and assignment submissions (Henrie et al., 2015). Broadly speaking, the process of systematically collecting and analyzing large sets of data from online courses is known as learning analytics (Brown, 2012), which aims to identify behavior patterns and improve student learning (Avella et al., 2016).

An important technological enabler of learning analytics has been learning record stores (LRS). LRSs are systems that preserve learner activity, enabling detailed tracking of learners’ course usage (Berking, 2016). Storing rich learner data has enabled researchers to better understand various learning-related phenomena such as student motivation (Li & Tsai, 2017), procrastination (You, 2015), social interactions (Agudo-Peregrina et al., 2014), dropout rates (Cohen, 2017) and engagement behavior (Wong & Chong, 2018). Taken together, learning record stores and learning analytics provide longitudinal data to analyze patterns over time (Tang et al., 2019) and in context (Anseeuw et al., 2016), a combination that can help researchers explore the relationship between engagement indicators and student success in authentic, online learning contexts.

The Current Study

In an effort to contribute to the ongoing discourse about the relationship between engagement indicators and student success, this study used data from an online course to address three questions: (1) To what degree does the frequency (how often) of engagement behavior predict success in an asynchronous online course? (2) To what degree does the immediacy (how early) of engagement behavior predict success in an asynchronous online course? and (3) To what degree does the regularity (in what ordered pattern) of engagement behavior predict success in an asynchronous online course?

Methods

Course and Context

Data used in this study were collected from an asynchronous, undergraduate course from a large, public university in the United States. Data came from seven semesters between Spring 2016 and Spring 2019. During this time 696 students were enrolled in the course and 438 (63%) consented to having their data used. 58% of students whose data were studied were female and 41% were male.

The course aimed to teach students computer skills for academic success. With skill mastery its main goal, the course employed a competency-based approach. Competency-based education is an instructional technique that measures learning through the demonstration of competencies as opposed to traditional credit hours (Uranis et al., 2019), which allows students to progress at their own pace as they master content (U.S. Department of Education, 2013). In competency-based courses, students are often provided with traditional tests such as multiple-choice, true/false or short answer quizzes to assess their foundational knowledge, (Colson & Hirumi, 2018). Such courses enable students to move quickly through familiar content, take more time on unfamiliar areas, and resubmit assessments to demonstrate competency (Colson & Hirumi, 2018).

The course consisted of 52 lessons and corresponding student learning assessments. In early iterations of the course, before the current study, students could complete lessons at any time, as long as the corresponding assessments were submitted by the end of the semester. Unfortunately, many students struggled to be successful with this approach, finding the flexibility of the competency-based course unfamiliar and confusing. To address this issue, instructors decided different lesson types were needed in order to provide more structure for students. The resulting course design, which produced the data for the current study, included lessons that varied in content, assessment format and assessment due date.

More specifically, the course included three types of lessons: initial, deadline-driven, and no-deadline. The initial lesson type was a single activity designed to introduce students to the course content and structure. Submission of the initial lesson’s assessment was required to unlock the remainder of the course. This assessment was due, on average, on the 16th day of the semester (note: average day is used due to slight variations in the calendar between semesters). Seventeen deadline-driven lessons, called “core” on the course website, were grouped into four themes. Assessments for the deadline-driven lessons were required and needed to be submitted by specific due dates throughout the semester. On average, the deadline-driven lesson assessments for the four themes were due on the 36th, 59th, 83rd, and 105th day of the semester.

Thirty-four no-deadline lessons, called “electives” on the course website, were optional and could be completed at any time before the end of the semester. On average, the no-deadline lessons were due on the 113th day of the semester. Although unusual, these lessons were included in the study as they served as a unique opportunity to examine engagement without traditional course deadlines. Figure 1 shows a student’s view of the deadline-driven (“core”) and no-deadline (“elective”) lessons in the first theme of the course.

Fig. 1.

Student view of a group of lessons (i.e., a theme) before and after passing assessments

To earn an A, students needed to submit 31 lesson assessments: one initial assessment, 17 deadline-driven assessments, and 13 no-deadline assessments. Since the course was competency-based, students were reminded that they could work ahead of deadlines to finish the course early. Historically, the vast majority of students used an entire semester to complete the course but because they were allowed to work ahead at their own pace, a few students have completed the course in a week.

Data Collection, Measures, and Analysis

The course platform incorporated the Experience Application Programming Interface Standard (Advanced Distributed Learning Initiative, 2022) to track and store detailed student activity data. Known as xAPI, this standard allows “packaging and transmission of learner actions called ‘Activity Statements’ between a tool (such as this course site) and a learning record store” (Kevan & Ryan, 2016, p. 144). In total, 453,083 activity statements were analyzed.

Eight independent variables were defined and operationalized by applying the three indicators of engagement behavior (frequency, immediacy, regularity) to the course’s three student lesson/assessment types (initial, deadline-driven, and no-deadline). Table 1 provides a summary of the names and definitions of the study’s indicators of engagement and student lesson types.

Table 1.

Names and definitions of the indicators of engagement and student lesson types

Category	Name	Definition (and calculation)
Indicator of engagement	Frequency	I measure of how often students viewed course lessons (calculated as number of views per lesson)
	Immediacy	I measure of how early students submitted assessments (calculated as the average day of the semester assessments were submitted)
	Regularity	A measure of the ordered pattern between students’ assessment submissions (calculated as the average number of days between submissions)
Student lesson type	Initial	First student lessons and learning assessment due in the course
	Deadline-Driven	Student lessons and learning assessments other than the first with specific due dates throughout the semester
	No-Deadline	Student lessons and learning assessments with no specific due dates that must be completed by the end of the semester

The first set of independent variables was derived by applying the frequency (how often) of engagement indicator to each lesson type. These variables were operationalized as the number of times students viewed the initial, deadline-driven, and no-deadline lessons. The second set of independent variables was derived by applying the immediacy (how early) of engagement indicator to each lesson type. These variables were operationalized as when, calculated as the average day of the semester, students submitted the initial, deadline-driven, and no-deadline lessons’ learning assessments. The third set of independent variables was derived by applying the regularity (in what ordered pattern) of engagement indicator to each lesson type. These variables were operationalized as the period, calculated as the average number of days students took between lesson assessment submissions. Because there was only one assessment for the initial lesson type, regularity of engagement was not applicable. Table 2 presents a summary of the study’s independent variables.

Table 2.

Independent variables

Indicator of engagement behavior	Lesson type
	Initial	Deadline-driven	No-deadline
Frequency	IV1: Frequency of engagement with initial assessment	IV2: Frequency of engagement with deadline-driven assessments	IV3: Frequency of engagement with non-deadline assessments
Immediacy	IV4: Immediacy of engagement with initial assessment	IV5: Immediacy of engagement with deadline-driven assessments	IV6: Immediacy of engagement with no-deadline assessments
Regularity	N/A	IV7: Regularity of engagement with deadline-driven assessments	IV8: Regularity of engagement with no-deadline assessments

The dependent variable was students’ course grade (A-F). Grades were assigned based on the number of learning assessments students passed. Students who passed a total of 32 lesson assessments earned an A, 26 or more a B, 16 or more a C, and 6 or more a D. Of the 438 students included in the study, 302 (69%) earned an A, 78 (18%) earned a B, 27 (6%) earned a C, 10 earned a D (2%) and 21 (5%) earned an F. Descriptive statistics and multiple regression were used to examine the relationship between indicators of engagement, lesson types, and grades.

Results

Descriptive Statistics

The first step in analysis was comparing students’ frequency (how often) of engagement—operationalized as number of page views—with final grades. One would expect more engaged students to view lessons more often. If true, students earning higher grades would view course lessons more frequently. The analysis revealed that the average number of initial lesson views was 10.11 (SD = 6.74). The average number of deadline-driven lesson views was 8.85 (SD = 6.74) and the average number of no-deadline lesson views was 22.45 (SD = 11.91). Table 3 presents descriptive statistics for the frequency (how often) of engagement, by lesson type and course grade.

Table 3.

Page-views by lesson type and course grade

Course grade	N	Lesson type
		Initial		Deadline-driven		No-deadline
		M	SD	M	SD	M	SD
A	302	10.82	7.20	9.08	4.07	20.41	9.11
B	78	9.26	5.47	8.62	3.46	24.61	11.01
C	27	6.81	5.28	6.01	2.64	30.70	18.94
D	10	8.40	3.47	6.80	2.86	33.57	16.89
F	21	8.10	4.87	5.30	4.23	27.92	22.91
Total	438	10.11	6.74	8.58	4.01	22.45	11.91

When visualized graphically, patterns emerge between number of page-views, lesson type, and course grade. For example, students earning an A viewed the initial lesson the most (M = 10.82, SD = 7.20), whereas students earning a C viewed it the least (M = 6.81, SD = 5.28). A similar pattern exists for the deadline-driven lesson views. Students earning an A viewed the deadline-driven lessons the most (M = 9.08, SD = 4.07), whereas students earning an F viewed them the least (M = 5.30, SD = 4.23). Interestingly, this pattern was reversed for no-deadline lesson views. Students earning an A viewed no-deadline lessons the least (M = 20.41, SD = 9.11), whereas students earning a D viewed them the most (M = 33.57, SD = 16.89). Figure 2 shows a comparison of average views by lesson type and course grade.

Fig. 2.

Frequency of engagement by lesson type and course grade

The next step was to examine students’ immediacy (how early) of engagement by lesson type as measured by the average day (of the semester) students submitted each lesson’s corresponding assessment. One might expect students earning higher grades to submit assessments earlier compared to students earning lower grades. In other words, students earning higher grades would submit assessments earlier and therefore have lower immediacy scores.

Analysis revealed that the average day the initial lesson’s learning assessment was submitted was 11.15 (SD = 13.21). The average day deadline-driven lessons’ learning assessments were submitted was 59.34 (SD = 13.40). The average day no-deadline lessons’ learning assessments were submitted was 62.37 (SD = 26.10). Table 4 presents descriptive statistics for the average day of the semester learning assessments were submitted by lesson type and course grade.

Table 4.

Average day of the semester assessments were submitted by lesson type and course grade

Course grade	N	Lesson type
		Initial		Deadline-driven		No-deadline
		M	SD	M	SD	M	SD
A	302	9.56	15.79	57.88	13.56	57.56	24.45
B	78	13.29	7.10	64.31	11.50	74.62	24.96
C	27	16.96	8.13	65.94	11.88	78.56	25.32
D	10	17.30	8.43	55.95	15.53	67.33	24.99
F	21	15.67	8.37	55.05	11.87	63.00	32.72
Total	438	11.15	13.21	59.34	13.40	62.37	26.10

When visualized graphically, patterns emerge between the average day of the semester students submitted learning assessments and their course grade. For example, on average, students earning an A submitted the initial lesson’s assessment by day ten (M = 9.56, SD = 15.79), whereas students earning a C, D or F submitted the initial lesson’s assessment on day 16 or 17. Interestingly, students earning an A submitted both the deadline-driven and the no-deadline lessons’ assessments by day 58 (on average). In contrast, students earning other grades completed the assessments for no-deadline lessons up to 13 days later than assessments for the deadline-driven lessons. Figure 3 compares the average day of submission by lesson type and course grade.

Fig. 3.

Immediacy of engagement by lesson type and course grade

The next step was examining students’ regularity (in what ordered pattern) of engagement by lesson type as measured by the average number of days between lesson assessment submissions. One might expect high performing students to submit lesson assessments on a fairly regular basis. In other words, students earning higher grades would wait fewer days between lesson assessment submissions. No regularity score was calculated for the single initial lesson’s assessment.

Analysis revealed that the average number of days between deadline-driven lesson assessment submissions was 4.93 (SD = 1.05). The average number of days between no-deadline assessment submissions was 5.94 (SD = 4.25). Table 5 presents descriptive statistics for the regularity of engagement by lesson type and course grade.

Table 5.

Regularity of engagement by lesson type and course grade

Course grade	N	Lesson type
		Initial		Deadline-driven		No-deadline
		M	SD	M	SD	M	SD
A	302	--	--	4.77	0.67	5.38	3.13
B	78	--	--	5.15	0.71	6.59	4.05
C	27	--	--	5.03	0.99	9.05	6.91
D	10	--	--	4.99	1.53	6.33	4.98
F	21	--	--	6.20	3.30	7.39	9.44
Total	438	--	--	4.93	1.05	5.94	4.25

When visualized graphically (Fig. 4), patterns emerge in the average number of days between assessment submissions by lesson type and course grade. Somewhat unexpectedly, all students showed similar regularity in submitting the deadline-driven lessons’ assessments. For the no-deadline lessons’ assessments, students earning an A averaged the lowest number of days between submissions (M = 5.38, SD = 3.13) and students earning a C averaged the highest (M = 9.05, SD = 6.91).

Fig. 4.

Regularity of engagement by lesson type and course grade

Regression Analysis

The next analysis step was to determine which of the eight independent variables (IVs), if any, predicted course grade. A multiple linear regression was calculated to predict final course grade based on frequency (how often) of engagement with the initial lesson type (IV1), the deadline-driven lesson type (IV2), the no-deadline lesson type (IV3), immediacy (how early) of engagement with the initial lesson type (IV4), deadline-driven lesson type (IV5), no-deadline lesson type (IV6), and regularity (in what ordered pattern) of engagement with deadline-driven lesson type (IV7) and no-deadline lesson type (IV8). Using the standard method of entering all IVs simultaneously (the “enter” method), a significant regression equation was found (F (8,429) = 15.549, p < .001), with an R² of 0.225, R² Adjusted = 0.21.

Students’ predicted final course grade was equal to 5.39 − 0.016 (IV1) − 0.013 (IV2) − 0.001 (IV3) + 0.012 (IV4) − 0.263 (IV5) + 0.003 (IV6) − 0.008 (IV7) − 0.014 (IV8). As summarized in Table 6, the analysis revealed five significant predictors of final course grade: immediacy of engagement with the initial lesson, Beta = -0.013, t(437) = -3.133, p = .002; immediacy of engagement with the deadline-driven lessons, Beta = 0.012, t(437) = 2.759, p = .006; regularity of engagement with the deadline-driven lessons, Beta = -0.263, t(437) = -5.684, p < .001; frequency of engagement with the no-deadline lessons, Beta = 0.003, t(437) = 4.191, p < .001; and the immediacy of engagement with the no-deadline lessons, Beta = -0.008, t(437) = -3.678, p < .001.

Table 6.

Independent variables found to be significant predictors of course grade

Indicator of engagement behavior	Lesson type
	Initial	Deadline-driven	No-deadline
Amount			IV3: Frequency of engagement with non-deadline assessments**
Immediacy	IV4: Immediacy of engagement with initial assessment*	IV5: Immediacy of engagement with deadline-driven assessments*	IV6: Immediacy of engagement with no-deadline assessments**
Regularity	N/A	IV7: Regularity of engagement with deadline-driven assessments**

^*p < .01. ^**p < .001

Discussion

Results revealed that the three indicators of student engagement were significant predictors of academic success as measured by course grade. Specifically, the frequency, immediacy and regularity of engagement significantly predicted student success. However, the significance of these engagement indicators differed based on the three lesson types (initial, deadline-driven and no-deadline).

Frequency of Engagement Behavior

The study’s first question inquired if the frequency (how often) of engagement behavior, an indicator of quantity, would predict course grade. Measured by the number of page-views per lesson, the predictive nature of this variable varied by lesson type. Specifically, frequency of engagement with no-deadline lessons predicted course grade, whereas frequency of engagement with initial and deadline-driven lessons did not.

Initial Lesson

Frequency of engagement with the initial lesson was not a significant predictor of course grade, although some unexpected differences were found. This particular lesson and its assessment involved multiple steps, therefore students needed to view the related lesson page multiple times. Students earning an A averaged 10.82 (SD = 7.20) views of the lesson, whereas students earning an F averaged 8.10 views (SD = 4.87). Interestingly, students that earned a C exhibited the lowest frequency of engagement with an average of 6.81 (SD = 5.28) views.

Deadline-Driven Lessons

The number of page-views of deadline-driven lessons was not a significant predictor of course grade. However, page-views of the no-deadline lessons, which will be discussed in the following section, were. This difference may be explained by the differing nature of the deadline-driven and no-deadline lessons’ assessments. The deadline-driven lessons’ assessments were often automatically graded quizzes and could be taken an unlimited number of times. Students may have felt that the most efficient way to pass a quiz was through multiple attempts rather than reviewing the lessons, resulting in fewer page-views. This is a form of “gaming the system”, a phenomenon where students systematically exploit properties and regularities of the learning system, rather than learning the material (Baker et al., 2004).

No-Deadline Lessons

As seen in Fig. 2, students generally viewed the pages of no-deadline lessons more frequently than deadline-driven lessons. The number of page-views for no-deadline lessons was a significant predictor of course grade. Unexpectedly, students earning the highest course grades (A’s and B’s) had the fewest page-views and students earning lower course grades had the highest. Specifically, students who earned an A averaged the lowest average number of page-views. This was lower than the overall average frequency of engagement with no-deadline lessons and substantially lower than students earning a D in the course, who averaged the highest number of page-views.

The higher number of page-views of no-deadline lessons, as opposed to deadline-driven lessons, may be related to the fact that most no-deadline lesson assessments required completion of a manually graded assessment, instead of an automatically graded quiz. Because most of the no-deadline, manually graded assessments involved multiple tasks and could not be completed through trial-and-error (like quizzes), students may have revisited the lessons multiple times. Additionally, students may have viewed no-deadline lessons more often while deciding which lessons to complete because, unlike the required deadline-driven lessons, students could choose from a pool of no-deadline options. Thus, students may have spent more time “browsing” the no-deadline lessons to find ones aligned with their interests or appeared “easy” to complete. Notably, students with higher grades viewed the no-deadline lessons less frequently than lower performing students.

Why would students earning higher grades have the lowest frequency (how often) of engagement with no-deadline lessons? A view of engagement that is strictly quantity-based (i.e., more is better) would expect higher performing students to have the highest number of page-views for all lesson types. Such a view would suggest that more engaged students would view the no-deadline lessons more frequently than less engaged students. However, the data show the opposite: higher performing students, presumably more engaged, viewed the no-deadline lessons the least.

Possible explanations for this finding might have to do with course characteristics and student characteristics. Regarding course characteristics, one explanation may relate to the relationship between deadline-driven and no-deadline lessons. Deadline-driven lessons were designed to provide a foundation for the content in subsequent no-deadline lessons. Since students with higher grades viewed deadline-driven lesson pages more often, the amount of engagement spent on these lessons may have had a “downstream” impact on the amount of engagement needed to complete the no-deadline assessments. In short, more engagement with the deadline-driven lessons may have enabled students to work more efficiently on the related, no-deadline lessons.

The finding that high grade-earning students viewed no-deadline lessons the least might be explained by student characteristics. It is possible that higher-performing students processed lesson content more efficiently. For example, they may have applied more advanced cognitive and/or metacognitive strategies as they worked through the lessons. Because the course’s competency-based approach requires students to self-regulate in order to be successful, those earning higher grades may have been implementing strategies associated with better information processing (e.g., note-taking). Relatedly, they may have been better at selecting no-deadline lessons that aligned with their existing skills or knowledge, thus giving them an advantage.

While frequency (how often) of engagement has been found to be significant in multiple studies (Conijn et al., 2017; Soffer & Cohen, 2019), these results point to the need for more nuanced usage of indictors of quantity in online courses. As others have discussed (Asarta & Schmidt, 2013; Barbera et al., 2012; Canal et al., 2015), simply counting a general activity indicator such as page-views or course logins may not be as predictive as more specific and/or multiple measures combined. For instance, You (2015) found that the frequency of course logins did predict course achievement, but not as strongly as regularity of study and the number of late submissions. General quantity-based measures alone may be inadequate for predicting student success. As a result, incorporating more detailed analyses of student and course characteristics may prove valuable in understanding the predictive role of such measures.

Immediacy of Engagement Behavior

The study’s second question focused on the extent to which immediacy (how early) of engagement, measured by the average day of the semester students submitted an assessment, predicted course grade. Notably, immediacy of engagement, or submitting assessments earlier, significantly predicted course grade for all three lesson types.

Initial Lesson

For the initial lesson, students earning an A had immediacy scores of approximately 10, meaning they submitted the initial lesson’s assessment on day 10 of the semester though the assessment was not due until day 16. Stated differently, they submitted the lesson’s assessment about six days before the due date. A similar pattern was found for students earning a B. These students earned immediacy scores around 13, meaning they submitted the initial lesson’s assessment on day 13, three days before the due date. Students with lower grades (C, D, F) had immediacy scores that were higher than 16, meaning they submitted after the due date. Procrastination, the antithesis of early engagement, appears to be a factor for those with lower grades, as the average submission date was after the deadline.

These results support findings emphasizing the importance of students’ initial course engagement in student success (Cipher et al., 2018; Conijn et al., 2017; Wong & Chong, 2018; You, 2016). Because the initial lesson’s assessment must be passed to “unlock” the entire course, it is an essential milestone. Success in online courses requires higher self-regulation (Broadbent, 2017), and late submission of this initial assignment may indicate a lack thereof. For instructors, late submission of an initial lesson may serve as an early way to identify students likely to struggle.

Deadline-Driven Lessons

The immediacy of engagement with deadline-driven lessons was significant in predicting course grades. It seems logical that higher performing students would submit assessments earlier, and therefore have lower immediacy scores. This was borne out by the immediacy scores of students who earned A’s (see Table 3).

Unexpectedly, students earning D’s and F’s had lower immediacy scores than those with B’s and C’s. This finding may be explained by the nature of the course grading system, which is based on the number of lesson assessments successfully completed. Those students who “give up” or greatly reduce their efforts in the latter half of the semester submit most of their lesson assessments early, thus biasing their immediacy scores. Therefore, the D and F students earned lower grades and lower immediacy scores.

No-Deadline Lessons

The immediacy of engagement with no-deadline lessons was also a significant predictor of course grade. Students’ immediacy scores on the no-deadline lessons’ assessments followed a similar pattern to those on the deadline-driven lessons’ assessments. Students earning an A demonstrated more immediacy (earlier submissions) through a lower score, than those with B’s and C’s. Those with D’s and F’s, however, had lower immediacy scores than those with B’s and C’s. As with the deadline-driven lessons’ assessments, this is likely because students who “gave up” submitted relatively fewer assessments later in the semester, so that their early, yet too-few submissions kept their immediacy scores low.

Because these lessons’ assessments can be completed any time throughout the semester, engagement with no-deadline lessons may reflect the self-regulation of students who earn A’s, B’s and C’s. Those who submit no-deadline assessments earlier in the course do better. It can be reasoned that these students have better time management and planning, skills that are a particularly important for online students (Inan et al., 2017).

The course is structured such that the deadline-driven (“core”) lessons are presented alongside two related no-deadline (“elective”) lessons (see Fig. 1). This association of core and elective lessons, combined with the finding that A students submitted both deadline-driven and no-deadline lessons’ assessments, on average, by day 58 of the semester may mean that A students work on both types of lessons at the same pace and likely at the same time, rather than returning to elective lessons later.

This idea is additionally supported by regularity of engagement patterns. All students averaged approximately five days between their submissions of the deadline-driven lessons’ assessments. However, as seen in Fig. 4, differences emerged with submission of the no-deadline lessons’ assessments. Students earning an A exhibited similar regularity for deadline and no-deadline lessons. In comparison, students earning a B took longer between submissions of the no-deadline lessons’ assessments, and C students took even longer.

Regularity of Engagement Behavior

The study’s third question asked about the extent to which regularity (in what ordered pattern) of engagement behaviors, as measured by the number of days between assessment submissions, predicted students’ course grades. Regularity of engagement was found to be significant for deadline-driven lessons, but not for those without deadlines. For all students, the number of days between the deadline-driven lessons’ assessment submissions, as shown in Table 5, averaged 4.93 (SD = 1.05). Students that earned an A waited the fewest number of days between the deadline-driven lessons’ assessment submissions, whereas F students waited the highest number. Lower regularity was better, which suggests the importance of self-regulation and supports other findings on the importance of regular, incremental study (Conijn et al., 2017; Li & Tsai, 2017; You, 2016; Yu & Jo, 2014).

Given a 120-day semester and the requirement to complete 17 deadline-driven assessments by the 105th day, students have approximately five days per deadline-driven assessment after submitting the initial assessment. The most regular pattern possible would show a student completing one deadline-driven assessment every five days. Heat maps can be used to illustrate patterns in the regularity of assessment submissions. As shown in Fig. 5, each square represents one day of the semester and each square’s amount of shading indicates the number of assessments submitted that day. A completely white square (no shading) means zero lesson assessments were submitted whereas a completely black square (complete shading) means every student submitted an assessment. Using these maps, a highly regular pattern of submissions shows consistent shading throughout the semester. Students earning an A demonstrated regularity through a consistent shading pattern. This is less evident in the shading patterns of students earning successively lower grades.

Fig. 5.

Assessment submissions by day of the semester

Interestingly, the regularity of the no-deadline lesson submissions did not predict course grade. Based on the immediacy results discussed earlier, submitting the no-deadline lessons’ assessments earlier as opposed to more regularly, was more important for predicting student success. Recall that earning an A required 13 no-deadline submissions by the end of the semester. Students had 95 days following the initial assessment deadline to complete those 13 submissions. Therefore, the most regular pattern would have students submitting a no-deadline assessment every 7.31 days. Students who waited to submit no-deadline assessments until late in the semester may have run out of time. This is consistent with findings that procrastinating significantly predicts performance (Asarta & Schmidt, 2013; You, 2015).

Limitations

Several limitations to this study should be considered. First, the study involved a retrospective analysis of previously collected data, of which a subset was used. Data collection was conducted on a custom platform that provided more granular data than most learning management systems can provide. Despite a large sample size, data was collected from a single course at one institution, reducing generalizability. The data represented student interactions with the course website, which is not a complete representation of engagement behavior. In addition, important information pertaining to student and course characteristics (e.g., year of study, where the course sits in the curriculum) was not analyzed. Another limitation is the unequal distribution of course grades. This was inevitable as a competency-based course. As a result, 87% percent of students earned a grade of A or B and the sample size for lower grade earners was relatively small so results should be interpreted with caution.

Conclusion

This study applied three indicators of engagement (frequency, immediacy, and regularity) to three lesson types (initial, deadline-driven, and no-deadline) to identify predictors of student success in an asynchronous course. The findings provide useful guidance for instructional designers, instructors, learners and researchers.

Frequency (how often) of engagement did not by itself consistently predict student success. Therefore, researchers may want to consider the measure (such as page-views) and the content type (such as deadline-driven versus no-deadline lessons) when studying engagement. The need to consider multiple factors suggests that indicators of quantity may not be the most straightforward measures of engagement. Moreover, the finding that higher-performing students viewed no-deadline lessons less than their lower-performing counterparts emphasizes the value of integrating indicators of time into engagement research. This finding may also suggest the purpose of no-deadline lessons, as well as how to approach them, was unclear to some students— a point relevant to designers and instructors of competency-based courses.

Immediacy (how early) of engagement, or early submission of assessments, significantly predicted course grades across all three assessment types. The initial lesson, in particular, was a useful tool for gauging immediacy and predicting student success. This suggests that how students engage with the first assessment of an online course might be used to identify those needing additional support. It is important to note, however, that the initial lesson in this course differed in structure and content from the other lessons. For this reason, further analysis is needed to determine if particular aspects of the lesson itself (e.g., topic, difficulty, assessment type) made it predictive.

That said, instructors and course designers may benefit from including an initial lesson and assessment in their course designs. Because such an assessment is completed early in the course, the predictive value of students’ submission timing (immediacy) may be valuable. This could be particularly important in asynchronous courses where instructors receive fewer interaction cues than when engaging synchronously.

Regularity (in what ordered pattern) of engagement was found to be significant for deadline-driven lessons, but not for those without deadlines. Deadline-driven lessons were most similar to typical course assignments where deadlines are common. In contrast, no-deadline lessons could be submitted throughout the semester and on students’ personal schedules. Therefore, it is plausible that regularity of engagement is predictive in more typical course designs that have assigned due dates. In practice, instructors and course designers might find it helpful to know how much time has elapsed between student submissions, helping them identify students who may need support. Previous studies have shown the value of regularity (Conijn et al., 2017; You, 2016; Yu & Jo, 2014) and this study supports such findings, adding that deadlines may be an important contributor to regularity. In terms of instructional strategies and student support, it may be beneficial to communicate that students should engage early and regularly in their courses.

This study points to opportunities for further research such as comparing and contrasting students’ intended immediacy and regularity behaviors with their actual behavior. It is possible that some students are poor self-regulators, especially in online environments, and that course designs could include structures to assist in improving immediacy and regularity of engagement. For example, providing students with opportunities to plan and reflect on performance throughout the semester might improve their self-regulation and performance. In addition, instructors could devote more time to providing students with strategies for choosing and planning for no-deadline assessments. Another research opportunity could compare students’ intended immediacy and regularity behaviors with mathematically optimal levels. Seeing the behavior patterns of A students could prompt other students to reflect or modify their own goals and behavior.

The results also suggest interesting interactions between lesson type and how and when students engage with course content. It is clear that students handled assessments with concrete deadlines differently than “far-off” end-of-the-semester deadlines. This points to the need to support students in planning and persisting regularly toward longer-term goals. Further research might explore how providing visualizations of student behavior data via dashboards or progress reminders might impact their ability to remain committed to goals throughout the semester. Future studies might also compare and triangulate learner record data with self-report and interview data to better understand student engagement behaviors. The availability of learning record stores and learning analytics techniques make online asynchronous courses like the one used in this study ideal platforms for investigating students’ intended and actual engagement behaviors.

Biographies

Dr. Daniel L. Hoffman

is an assistant professor of Learning Design & Technology at the University of Hawai‘i at Mānoa. His research focuses on the design of interactive media and its impact on learning and engagement.

Dr. Faye Furutomo

is a program manager for the Technology & Distance Programs at the University of Hawaiʻi at Mānoa. Her research interests include technology innovation in higher education, online learning and organizational culture.

Dr. Ariana Eichelberger

is a specialist faculty member with the department of Learning Design & Technology at the University of Hawai‘i at Mānoa. Her research areas of interest include online teaching, online course and program development, technology integration and faculty professional development.

Dr. Paul McKimmy

is the Interim Associate Vice Provost for Academic Excellence at the University of Hawai‘i at Mānoa, College of Education. His research interests include distance learning, program development and technology policy.

Authors’ Contributions

All authors made substantial contributions to the work and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Data Availability

Data for this study are unavailable due to data protection and privacy concerns.

Declarations

Ethical Approval

This research was conducted within guidelines set by the Institutional Review Board of the authors’ university and under protocols approved by that Board. Informed consent was obtained from all participants. The authors have taken steps to protect the participants, including anonymizing all study related data.

Consent to Publish

The authors confirm that the work described has not been published before, is not under consideration for publication elsewhere; has been approved by all co-authors, and that its publication has been approved by the responsible authorities at the institution where the work is carried out. In addition, individuals appearing in the work’s figures have given consent to publish. 

Conflict of Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.