In-lecture quizzes improve online learning for university and community college students

Abstract

Online classes are now integral to higher education, particularly for students at two-year community colleges, who are profoundly underrepresented in experimental research. Here, we provided a rigorous test of using interpolated retrieval practice to enhance learning from an online lecture for both university and community college students (N = 703). We manipulated interpolated activity (participants saw review slides or answered short quiz questions) and onscreen distractions (control, memes, TikTok). Our results showed that interpolated retrieval enhanced online learning for both student groups, but this benefit was moderated by onscreen distractions. Surprisingly, the presence of TikTok videos produced an ironic effect of distraction—it enhanced learning for students in the interpolated review condition, allowing them to perform similarly to students who took the interpolated quizzes. Moreover, we showed in an exploratory analysis that the intervention-induced learning improvements were mediated by a composite measure of engaged learning, thus providing a mechanistic account of our findings. Finally, our data provided preliminary evidence that interpolated retrieval practice might reduce the achievement gap for Black students.

Interpolating a lecture with quiz questions enhanced online learning for both community college and university students. The benefit is influenced by level of distraction, is associated with engagement, and can reduce racial achievement gaps.

Introduction

Online learning has become integral to higher education, and the COVID-19 pandemic made it nearly ubiquitous¹. The convenience and flexibility of online courses attract a diverse student body, particularly those who hold a job, have family obligations, or live too far from a college campus to physically attend lectures^2,3. Although online education has made distance learning widely accessible, it also introduces unique pedagogical challenges. Relative to students in live classes, online learners are frequently distracted by external stimuli⁴ and are more prone to mind wandering⁵. Consequently, discovering interventions to improve learning from online lectures is crucial for enhancing educational outcomes. Recent research has shown that embedding brief quizzes into a learning episode (i.e., interpolated retrieval practice, IRP) can reduce mind wandering and enhance learning^6–8. In the present high-powered study, we aimed to provide a rigorous test of this intervention. An key element of this study was the involvement of community college students, who are twice as likely (~41%) as four-year university students (~20%) to enroll exclusively online⁹ but are profoundly underrepresented in experimental research.

The increasing prevalence of online instructions has raised questions about their efficacy relative to traditional face-to-face instructions. Although some studies have demonstrated that online classes can be comparable to in-person classes^10–12, many have not. For example, relative to students who enroll in face-to-face courses, students who enroll online are more likely to engage in off-task activities¹³ or drop the course^14,15, and less likely to achieve a passing grade^2,16 or obtain a college degree¹⁷. Although factors that underlie these poor learning outcomes are numerous and might evade causal inferences due to their correlational nature, one potential contributor is that students are prone to experiencing inattention either from mind wandering⁸ or from external distractions when viewing online lectures¹⁸.

Involuntary mind wandering—a spontaneous shift of attention away from the target task to some unrelated thoughts^19–21—is detrimental to comprehension and learning^22–26. This issue is exacerbated in longer study sessions like video lectures, during which maintaining focus becomes increasingly difficult over time—a pattern known as vigilance decrement²⁷. Indeed, students tend to mind-wander more and learn less as a lecture progresses^28,29. Inserting brief quizzes into a learning task (e.g., asking a few short-answer questions), however, has been shown to reduce mind wandering⁷, increase engagement³⁰, and promote retention of the tested material³¹. These IRP opportunities can also promote new learning of content presented after the interpolated quiz across myriad scenarios—a finding termed test-potentiated new learning (TPNL)⁶. For example, TPNL has been observed when people study word lists^32–35, paired associates^36–39, spatial routes⁴⁰, text passages^41,42, and video lectures^7,43–45. IRP has been touted as a potential remedy for inattentiveness and poor learning outcomes in online education^6,46,47. It is especially promising given its cost-effectiveness, scalability, and simple implementation.

Several explanations have been proposed to account for the TPNL effect. Some researchers argued that IRP promotes new learning by increasing learners’ attention toward the material (i.e., the attention hypothesis)^7,48. In contrast, other researchers suggested that switching from encoding to retrieval induces a mental context change, which helps students segregate the materials learned pre- and post-retrieval and reduces memory load^35,49,50. Lastly, IRP has been proposed to promote new learning because it helps learners optimize their encoding and retrieval strategies^36,51. Although these various accounts are not necessarily mutually exclusive, (e.g., sustained attention may be necessary to support the development of optimal retrieval strategies), we focus specifically on the attention hypothesis. We address the theoretical relevance of our experiment when we introduce its methodology.

Although ample evidence suggests that IRP is a potent learning enhancer, the vast majority comes from laboratory studies. Despite a meta-analysis showing that online (g = 0.69) and lab (g = 0.76) studies produce comparable effect sizes, it is nonetheless difficult to draw firm conclusions because only three of the included studies were conducted online⁶. Moreover, extant online studies have either recruited participants from specialized data pools (e.g., MTurk)^44,52,53 that can produce unreliable outcomes^54–56, employed simplistic materials (e.g., word lists, paired associates) that differ from typical classroom content^52,57,58, or have produced inconclusive results⁵⁹. Therefore, it was essential to examine the impact of IRP for online learning using materials and participant populations that resemble real-world situations.

An additional concern when drawing broad conclusions from existing data is that all studies that have linked IRP to a reduction in mind wandering were conducted in the laboratory^7,60–62. Critically, unlike the “messy” environments under which online learning occurs, the laboratory places participants in a “pristine” environment. For example, students are strongly discouraged, if not prohibited, from using digital devices, talking with other participants, or partaking in unrelated activities. The laboratory is constructed to foster focused attention on a single task, which creates a nearly ideal learning condition. In contrast, online learners might view a lecture in a noisy cafe, at home with pets or housemates, or in the presence of myriad other potential distractions (e.g., online shopping, instant messaging, social media, notifications from other connected devices, etc.).

The different attentional demands of laboratory and online studies present a unique challenge to using IRP as a learning intervention. As stated previously, a prominent account for TPNL is that IRP reduces inattention. This account is supported by evidence that IRP reduces mind wandering⁷, increases students’ expectations that they will be tested again⁴⁸, and lengthens students’ voluntary study duration^30,31. IRP has been proposed to reduce inattention for several reasons. First, when attempting to answer a question, participants must, at the very least, temporarily reorient their attention to the learning task. Second, experiencing retrieval failure might prompt participants to stay engaged with subsequent materials. Third, answering questions allows students to express what they know, a form of active learning that might foster engagement. Despite progress in understanding the relationship between attention and TPNL, current theoretical models do not yet permit precise predictions about whether IRP’s benefit would be enhanced or diminished in highly distracting environments. On the one hand, IRP might buffer learners against the distractions that they must ignore, leading to a more pronounced benefit. On the other hand, the opposite might occur if the “messy” conditions of online learning overwhelm the advantages of interpolated testing. Despite the lack of precision of the attention hypothesis, and unlike the context change or strategy optimization accounts mentioned earlier, the account unambiguously predicts that the magnitude of TPNL would be influenced by factors that can affect attention during encoding.

This study aimed to evaluate IRP as an intervention for online learning. All participants completed the experiment online, which should be more distracting than laboratory environments, but we also manipulated the level of onscreen distractions (none, static meme pictures, dynamic TikTok videos) to examine their effects on learning and mind wandering. Although we suspected that the benefits of IRP might be mitigated in online relative to in-person learning, we could not directly assess this possibility because we did not include an in-person group. This omission was intentional, as our objective was to study online learning. Therefore, we manipulated onscreen distraction levels to ensure that we could draw causal arguments about the effects of distractions on TPNL.

Community colleges educate approximately one-third of the undergraduate population in the United States, and about half of the students who enroll exclusively online. Further, the COVID-19 pandemic has had an especially lasting impact on the education delivery of two-year colleges. Although universities have largely returned to in-person education, for many community colleges, a majority of their courses have not returned to in-person delivery^63–65. Despite the essential role that community college plays in post-secondary education, students at these institutions are profoundly underrepresented in educational research in general^66,67 and experimental research in particular⁶⁸. Researchers have often promoted retrieval practice as an intervention for online learning, and yet community college students – perhaps the largest stakeholders for online education – are conspicuously absent in this literature. Moreover, students at two-year colleges are far more diverse than their four-year counterparts, both in socioeconomic status and race/ethnicity⁶⁹, so the involvement of community college students also provides an opportunity for us to investigate whether IRP can reduce racial achievement gaps in online learning. Studies have documented differences in performance on working memory⁷⁰ and executive function tasks across racial groups in nationally representative samples^71,72. Given the role of these cognitive functions in learning efficiency⁷³, metacognition⁷⁴, and retrieval practice benefits^75–78, it is important to assess our intervention’s effectiveness across diverse student populations.

To permit a comparison between university and community college students, we recruited half of our participants from four-year public universities (Iowa State University in the United States—ISU, Toronto Metropolitan University in Canada—TMU) and half from two-year community colleges (American River College in California—ARC, Des Moines Area Community College in Iowa—DMACC, Mohawk College in Canada—MC, Columbus State Community College—CSCC, and Cuyahoga Community College—Tri-C in Ohio).

Methods

Design and Participants

Iowa State University’s institutional review board (IRB) provided ethics oversight for data collected in the United States under the protocol 21-037. Toronto Metropolitan University’s IRB provided ethics oversight for data collected in Canada under the protocol REB-2020-285. In addition, ethics approval has been obtained from all community colleges involved.

The experiment employed a 2 (interpolated activity: review, quiz) × 3 (onscreen distraction: control/no-distractor, medium/meme distractor, strong/TikTok distractor) between-subjects design, with institution type (university, community college) as a subject variable. A critical objective was to ensure that our study resembled real-world online learning while retaining the advantages of an experiment. To this end, we developed a collection of high-quality, ~20 min STEM video lectures (computer science, animal ecology, physics, and statistics), with a different pair of man and woman instructors for each topic. Two instructors were middle-aged adults, and six were young adults. Two were Black, and six were White (see Figure S1 in Supplementary Information). All lecture videos are available on the OSF page. Participants viewed one of these videos in the presence of no distractors, medium distractors (memes), or strong distractors (TikTok videos), and were required to engage in either interpolated retrieval practice or review following the first three segments of the lecture. All participants completed an interpolated quiz following the fourth and final segment of the lecture and then a final cumulative test approximately 24 hours later (Fig. 1). To assess mind wandering, all participants responded to one mind wandering probe (MWP) during each segment of the lecture. Finally, all participants indicated how likely they were to be tested before each lecture segment, their level of engagement following each lecture segment, and were free to take notes during the lecture.

Fig. 1. Screenshots of the lecture video and experimental flowchart.

In the control, no-distractor condition, the area to the right of the lecture video was left blank. In the medium distractor condition, a different set of three meme pictures were shown for the duration of each lecture segment. The memes included a picture with a brief caption meant to convey humor. In the strong distractor condition, a fresh set of three TikTok videos were shown per minute of the lecture segment. The videos generally depicted humorous, light-hearted encounters, adorable animals, and nature scenes. The criterial test for Segment 4 is highlighted with a thick red border. The photographic image of the instructor was replaced with a line-drawing for publication purposes. Participants watched a real human serve as the instructor.

We conducted a pilot experiment (N = 439) to determine a possible effect size for our power analysis. The pilot data showed a moderate TPNL effect, t(437) = 4.37, p < .001, d = 0.42. Based on this effect size, we needed a minimum of 91 participants per condition to achieve .80 power. Because it took multiples of eight participants to complete a full counterbalance, we rounded up each condition to 96 participants (note that we omitted this consideration in our preregistration). We opted to over-provision our data collection goal by 20% to account for attrition and our preregistered data exclusion criteria. Therefore, we aimed to collect data from 115 participants per condition (for 690 participants across the six conditions), with approximately half of the participants from community colleges.

We recruited 898 participants in the experiment and excluded data from 195 (see Table 1 for exclusion reasons). We approached recruitment practices by employing the most efficient method permitted by each college’s institutional review board. At ISU and Tri-C, we recruited participants through mass emails. At ARC, DMACC, and MC, a staff member sent recruitment emails or embedded our study’s advertisement in electronic newsletters. At CSCC, the advertisement was posted to students’ learning management system. At TMU, we first attempted to recruit students on the TMU subreddit. Unfortunately, this approach resulted in very poor-quality data (e.g., many Reddit participants did not follow instructions). Consequently, we excluded all of the data acquired from Reddit (N = 61) and recruited students at TMU via flyers posted around campus.

Table 1.

Number of participants excluded from analysis

Exclusion reason	N
Missed all four mind wandering probes	143 (39)
Did not attempt to answer any questions on the criterial test	27 (18)
Low English proficiency	11 (2)
Not taking the experiment seriously at all	10 (1)
Browsing other websites during the entire study	4 (1)

Number in parentheses refer to participants who were from the TMU subreddit.

Ultimately, we retained data from 703 participants for analysis. Participants’ self-reported characteristics are displayed in Table 2. Of these participants, 622 completed both Sessions 1 and 2, and 81 did not complete Session 2 (i.e., cumulative test). A programming error affected the data of 27 participants. Specifically, we had included one incorrect question on the criterial test for these participants. To ensure that all of the data were based on a common set of questions, we removed the incorrect question from analysis for these participants so that their criterial test performance was based on three instead of four questions.

Table 2.

Number of participants as a function of institution and demographic variables

Institution	N
Iowa State University	187 (27%)
Toronto Metropolitan University	186 (27%)
American River College	98 (14%)
Columbus State Community College	96 (14%)
Des Moines Area Community College	52 (7%)
Cuyahoga Community College	52 (7%)
Mohawk College	31 (4%)

Gender	N	Mean Age
Woman	418 (61%)	24.3
Man	239 (35%)	22.7
Non-cisgender	30 (4%)	22.2

Race	N from Universities	N from Community Colleges
Asian/Asian American/Asian Canadian	48 (14%)	28 (9%)
Black/African American/African Canadian	28 (8%)	39 (13%)
Hispanic/Latinx	13 (4%)	36 (12%)
Multi-racial	9 (3%)	26 (8%)
Others	4 (1%)	3 (1%)
White	246 (71%)	175 (57%)

Parental Social Economic Status	N from Universities	N from Community Colleges
Upper Class	39 (11%)	14 (5%)
Middle Class	235 (67%)	146 (52%)
Working Class	71 (20%)	100 (35%)
Fully Supported by Government Assistance	5 (1%)	23 (8%)

These data include all participants who completed Session 1. Percentages are displayed in parentheses. We could not determine the college for one participant due to a program error (the participant could have attended ARC, CSCC, or Tri-C). For the demographic data, 48 participants chose not to report their race, 16 chose not to report their gender, and 70 chose not to report their SES. All demographics information was based on self-reports.

We retained all 703 participants for analysis to maximize statistical power for the Session 1 data, a decision justified by the absence of an interaction between Session 2 participation and other independent variables on criterial test performance. Specifically, participation during Session 2 did not produce a significant interaction with interpolated activity, F(1, 691) = 0.04, p = .843, η_p² < .001, 95% CI = [0.000, 0.005], BF₀₁ = 5.22, onscreen distractor, F(2, 691) = 1.17, p = .310, η_p² = .003, 95% CI = [0.000, 0.015], BF₀₁ = 8.00, or a three-way interaction with both variables, F(2, 691) = 0.17, p = .845, η_p² < .001, 95% CI = [0.000, 0.006], BF₀₁ = 7.13. Participants who completed both sessions received a $20 Amazon electronic gift card, whereas participants who finished only Session 1 received a $15 electronic gift card.

Materials and Procedure

The experiment included two sessions. In Session 1, participants were first presented with the informed consent form. Those who did not provide their consent were sent to a thank-you page and the experiment did not start. If participants provided their consent, they started the experiment by watching a ~ 20 min lecture video broken into four ~5 min segments. A head-and-shoulder view of the instructor appeared in front of the lecture slides (see Fig. 1). To ensure a consistent presentation style and to minimize potential instructor effects on engagement and learning, we instructed our actors (ISU Department of Theater instructors and students) to deliver the lecture flatly. Similar to an actual lecture, the latter segments presented content that built upon earlier segments. Participants were encouraged to take notes in the area below the video lecture and were informed that their notes would not remain visible across lecture segments. For participants in the control, no-distractor condition, the rightmost 20% of their screen was left blank. For participants in the medium distractor condition, the same area was occupied by three static memes, each with a short caption. These images conveyed humor and were meant to simulate online advertisements or click-baits. A new set of memes was shown for each segment. Therefore, each participant saw a total of 12 memes. For participants in the strong distractor condition, the rightmost area was occupied by three silent, captioned TikTok videos depicting humorous encounters, adorable animals, or calming nature scenes. To maximize distractions, each video cycled for approximately one minute and was then replaced by a new video, so each participant was presented a total of 60 TikTok videos.

Before the experiment, participants were told that they would watch a four-segment lecture video and receive either quiz questions or review slides after each segment. Participants were also told that a cumulative test would follow at the end of the experiment. In reality, participants were randomly assigned to either an interpolated review or interpolated quiz condition, which determined the activity for Segments 1 to 3. After each lecture segment, participants in the quiz condition answered four brief factual questions (e.g., what term does Python use to represent characters?). They had 20 s to answer each question and were then presented with the correct answer (e.g., str) as feedback for 3 s. Participants in the review condition were shown four review slides following each segment, with 23 s allocated to study each slide. The slides presented the same information as the interpolated quiz questions but with the answers embedded (e.g., Python uses “str” to represent characters).

In addition to the interpolated activity, participants were informed that they would encounter mind-wandering probes (MWP) at an unpredictable point during each lecture segment. Participants were shown an MWP and instructed to familiarize themselves with the five options: (1) what the professor is saying right now, (2) something related to an earlier part of the lecture, (3) relating the lecture to my own life, (4) something unrelated to the lecture, and (5) zoning out. Consistent with prior literature, we considered the first three options task-related thinking and the last two off-task⁶⁰. Each probe appeared silently for 10 s, covering the instructor’s face. The instructor continued to teach during the entire probe duration. We opted for this approach because we wanted to catch participants being completely off-task by missing the MWP entirely.

Participants were instructed to focus on the lecture and ignore any irrelevant images/videos on screen should they appear. Participants were also encouraged to type notes below the lecture, as they would for an actual class. Before watching each segment, participants were asked to rate the likelihood of being quizzed after the segment on a scale from 0 (sure that there will not be a quiz) to 100 (sure that there will be one). Consistent with prior studies⁴⁸, interpolated quizzes increased test expectations across segments (from 68% to 83%), F(3, 1047) = 37.83, p < 0.001, η_p² = 0.047, 95% CI = [0.028, 0.073], whereas interpolated review reduced them (from 65% to 53%), F(3, 1056) = 29.82, p < 0.001, η_p² = 0.037, 95% CI = [0.021, 0.058].

Immediately after each lecture segment and before the interpolated activity, participants completed an abbreviated, six-item Math and Science Engagement Scales⁷⁹ (see Table S1 of Supplementary Information). In addition to measuring subjective engagement, which we hypothesized to be related to the benefits of interpolated testing, the questionnaire also served as a brief (~1 min) retention interval. After viewing Segment 4, all participants completed the engagement questionnaire and then a quiz for Segment 4. Performance on this criterial test provided the key measure of interest. Next, participants answered demographic and data quality control questions (e.g., how seriously they took the experiment). Participants were assured that their responses would not impact compensation. Twenty-four hours later, participants were sent an automated email instructing them to participate in Session 2 within 12 h.

During Session 2, participants first answered four inference questions. They then completed a cumulative test, which contained all of the questions from the interpolated and criterial quizzes. At the end of the session, participants were debriefed and thanked. Session 1 took approximately 45 min and Session 2 about 15 min.

Data Analysis and Preregistration Statement

We first address the effects of IRP on test performance and mind wandering for university and community college students; we then examine the influence of onscreen distractions on test performance. Finally, we present two preregistered exploratory analyses: we first report a mediation analysis that tests the attention account; we then examine whether IRP can reduce racial achievement gaps.

All analyses were conducted using two-tailed tests with an alpha level of .05. We report Cohen’s d and partial eta square (η_p²) for effect sizes. Neither the criterial test data, W = 0.92, p < 0.001, nor the cumulative test data were normally distributed, W = 0.96, p < 0.001. This outcome is not surprising given that the criterial test contained only four questions and the cumulative test contained 16 questions. For null results, we also report Bayes Factor (BF₀₁), which quantifies support for the null hypothesis (H₀) against the alternative hypothesis (H₁), with a greater value indicating greater support of the null. For all Bayesian analyses, we used the default parameters in JASP⁸⁰ for our priors—a point-null (at zero) for H₀ and a zero-centered, two-sided Cauchy distribution with a 0.707 scaling factor for H₁.

This study was preregistered on the Open Science Framework at 10.17605/OSF.IO/URWVM on March 25, 2022. We deviated from our preregistered analyses on two occasions. First, we noticed an error in our preregistration, in which we stated that we would conduct a 2 (interpolated activity) × 3 (onscreen distraction) × 2 (on-task or off-task) ANOVA, without specifying the dependent measure and how we would measure mind wandering. That analysis actually described a 2 (interpolated activity) × 3 (onscreen distraction) ANOVA, and mind wandering (on-task or off-task) was the dependent variable. In the results section, we report the outcomes of this exploratory analysis by combining five measures related to engaged learning, including a dependent measure tied to mind wandering. Second, we had planned to conduct an ANOVA to examine whether our manipulations of interpolated activity and distraction would affect inference question performance. At the time of preregistration, we considered using performance on the inference questions as a measure of participants’ ability to integrate pieces of information within and across lecture segments. In hindsight, these questions were very difficult to produce and we could not ascertain that they actually measured integration. Moreover, participants generally performed poorly on them. An exploratory analysis showed that university students (M = 0.45) outperformed community college students on these inference questions (M = .39), t(620) = 2.65, p = .008, d = 0.21, 95% CI = [0.06, 0.37]. But none of our planned analyses revealed significant effects of any independent variables on these questions, so we will not mention them further.

Finally, for completeness, we also preregistered two planned exploratory analyses: one on whether instructor gender affected our dependent measures, and the other on temporal changes to mind wandering based on interpolated activity. We report the outcomes of these analyses in the Supplementary Information under Supplementary Notes 1 and 2.

Results

Did IRP Enhance New Learning for University and Community College Students?

Figure 2 shows that IRP enhanced new learning for both university and community college students. A 2 (interpolated quiz, interpolated review) x 2 (university, community college) ANOVA showed that prior retrieval (M = 0.55) improved Segment 4 test performance relative to review (M = 0.47), F(1, 699) = 9.56, p = 0.002, η_p² = 0.013, 95% CI = [0.002, 0.035]. Moreover, university students (M = 0.56) outperformed community college students (M = 0.45), F(1, 699) = 24.45, p < .001, η_p² = 0.034, 95% CI = [0.012, 0.064]. But perhaps most importantly, the two variables did not interact, F(1, 699) = 0.01, p = 0.905, η_p² < .001, 95% CI = [0.000, 0.004], BF₀₁ = 8.40. Both university and community college (CC) students benefited from IRP, t_university(371) = 2.20, p = 0.029, d = 0.23, 95% CI = [0.02, 0.43]; t_CC(328) = 2.17, p = 0.030, d = 0.24, 95% CI = [0.02, 0.46]. Therefore, inserting brief quizzes into an online lecture promoted learning of the last segment of the lecture for both student groups - even in an uncontrolled online environment.

Fig. 2. Criterial test performance as a function of interpolated activity and institution type.

Error bars are 95% confidence intervals. University: N = 373. Community College: N = 330. Each dot represents the data of an individual participant. The dots were displaced horizontally to improve visibility.

In addition to the criterial Segment 4 test, participants completed a cumulative test a day later. Similar to the criterial test, prior retrieval enhanced cumulative test performance (M = 0.62) relative to review (M = .54), F(1, 618) = 15.98, p < .001, d = 0.32, 95% CI = [0.01, 0.06], and university students (M = .62) outperformed community college students (M = .53), F(1, 618) = 16.07, p < .001, η_p² = 0.025, 95% CI = [0.007, 0.055]. The interaction was not significant, F(1, 618) = 0.25, p = 0.617, η_p² < 0.001, 95% CI = [0.000, 0.009], BF₀₁ = 7.06.

For mind wandering, we examined the proportion of probes for which participants claimed to have engaged in task-related thinking, which included “what the professor is saying,” “something related to an earlier part of the lecture,” and “relating the lecture to my own life.” This dependent measure captured all thoughts associated with learning from the lecture⁶⁰, and the data from all four MWPs were combined to increase reliability⁸¹. We conducted the same 2×2 ANOVA, but unlike the data on test performance, we found no credible evidence that interpolated activity affected engagement with lecture content (M_review = .67, M_quiz = .69), F(1, 699) = 0.49, p = 0.485, η_p² < 0.001, 95% CI = [0.000, 0.010], BF₀₁ = 9.06, nor did it interact with institution type, F(1, 699) = 0.04, p = .846, η_p² = .005, 95% CI = [0.000, 0.021], BF₀₁ = 8.53. The main effect of institution type was also not significant, F(1, 699) = 3.56, p = .060, η_p² < .001, 95% CI = [0.000, 0.005], BF₀₁ = 1.99.

Before concluding that there was little credible evidence that IRP influences mind wandering, it was important to validate our measure. If the mind-wandering responses accurately reflected participants’ cognitive state, then participants with more task-related thinking responses should exhibit better test performance. Indeed, the proportion of task-related thinking across the four mind-wandering probes was correlated with both Segment 4 test performance (r = .32, 95% CI = [0.25, 0.38], p < .001) and cumulative test performance (r = .33, 95% CI = [0.26, 0.40], p < .001). These results are noteworthy because of their predictive validity, such that participants’ real-time responses to the MWPs predicted their test performance both minutes (Segment 4 criterial test) and a day (cumulative test) later. In sum, unlike previous laboratory studies, we found no credible evidence that IRP reduced mind wandering despite it having improved learning. We further investigated the role that attention plays in the TPNL effect later in an exploratory analysis.

Effects of Onscreen Distractions

We now examined the influence of interpolated activity (quiz, review) and onscreen distraction (control, meme, video) on test performance (see Fig. 3). The key findings for this analysis are those pertaining to the distraction manipulation, because the main effect of interpolated activity is the same as that reported in the prior section. The main effect of distraction was not significant, F(2, 697) = 1.38, p = .252, η_p² = .004, 95% CI = [0.000, 0.016], BF₀₁ = 11.71. The interaction between distraction and interpolated activity also did not reach significance, F(2, 697) = 2.58, p = .077, η_p² = .007, 95% CI = [0.000, 0.023], BF₀₁ = 3.11, but it exhibited an unexpected pattern. To further scrutinize this result, we conducted an exploratory analysis for each distractor condition. These comparisons revealed a significant TPNL effect for both the no-distractor control condition, t(233) = 2.83, p = .005, d = 0.37, 95% CI = [0.11, 0.63], and the meme condition, t(233) = 2.62, p = .009, d = 0.34, 95% CI = [0.08, 0.60], but not for the TikTok condition, t(231) = 0.06, p = .955, d < 0.01, 95% CI = [-0.25, 0.26], BF₀₁ = 6.97. In fact, participants in the TikTok condition exhibited numerically identical performance following interpolated review and quizes (both Ms = .51). We suspect that the interaction’s lack of significance was driven by two of the three distraction conditions (i.e., control and meme) having produced the same, moderate TNPL effect, thereby increasing the power needed to detect an interaction.

Fig. 3. Criterial test performance as a function of interpolated activity and onscreen distraction.

Error bars are 95% confidence intervals. Participants in the control condition did not have any onscreen distractors (N = 235). Participants in the medium distractor condition had three memes on the right side of the screen during the lecture (N = 235). Participants in the strong distractor condition had three TikTok videos playing on the right side of the screen throughout the lecture (N = 233). Each dot represents the data of an individual participant. The dots were displaced horizontally to improve visibility.

At first glance, the null effect in the strong distractor condition suggests that TPNL diminished with onscreen distractions, but a closer examination of Fig. 3 suggests a more nuanced interpretation. Specifically, the TikTok videos reduced accuracy (M = 0.51) relative to no distractors (M = 0.54) for participants in the interpolated quiz condition, but this slight drop in performance was not the main contributor to the elimination of the TPNL effect. Instead, for participants in the interpolated review condition, the onscreen distractions unexpectedly enhanced, rather than reduced, their criterial test performance. Indeed, interpolated review participants who were given video distractors (M = 0.51) actually outperformed their no-distractor counterparts (M = 0.43), t(239) = 2.10, p = 0.037, d = 0.27, 95% CI = [0.02, 0.52]. Therefore, rather than IRP merely failing to benefit memory amid onscreen distractions, our data revealed an ironic effect of distraction – the TikTok videos might have kept students engaged with the lecture. We must note that this is an unexpected finding, so further research is needed to ascertain its reliability and generality. For example, our onscreen distractions was silent, so participants could listen to the lecture content while watching the TikTok videos, which might ironically help participants stay engaged with the content. We believe that the same benefit might not occur for participants who received interpolated quizzes because the quizzes already kept them engaged with the lecture.

We now present results from the cumulative test. Onscreen distraction did not have a main effect on cumulative test performance, F(2, 616) = 1.72, p = 0.180, η_p² = 0.006, 95% CI = [0.000, 0.021], BF₀₁ = 11.18. If anything, participants in the video distractor condition (M = .60) produced the best performance overall (M_Control = 0.57, M_Meme = 0.56). Further, interim activity and distraction did not interact, F(1, 616) = 0.51, p = .600, η_p² = .002, 95% CI = [0.000, 0.011], BF₀₁ = 18.43.

Exploratory Analysis on Engaged Learning

Contrary to several laboratory studies, our data showed that taking interpolated quizzes did not credibly reduce mind wandering. In the following exploratory analysis, we aimed to obtain a more reliable measure related to the broader concepts of attention, engagement, and effort by combining multiple variables that were individually less reliable into a composite measure^82,83. The five variables were participants’ STEM engagement score for video lecture Segment 4, the number of Segment 4 idea units captured by participants’ Segment 4 notes, participants’ expectation that they will be quizzed following the Segment 4 lecture, the task-related thinking score from the four MWPs, and participants’ answer to the post-experiment question about how seriously they took the experiment (5-point scale).

We used artificial intelligence (OpenAI’s GPT4) to code for the presence of idea units in participants’ notes. To ensure that ChatGPT’s codings were satisfactorily accurate, we compared GPT’s coding with two trained undergraduate coders on a randomly selected subset of 40 participants’ data per topic. Each lecture segment had between 14 and 19 idea units. Therefore, each research assistant coded 10,200 data points. Across these data points, GPT’s coding produced agreement rates comparable to that between the human coders (GPT vs. Human A: 84%, GPT vs. Human B: 81%, Human A vs. Human B: 87%). We will detail the AI-assisted coding methodology in a separate report.

Consistent with the idea that the five measures were related to the same latent construct, they converged onto a single factor in an exploratory factor analysis with principal axis factoring, eigenvalue = 1.89, proportion of variance = 0.26 (see Table S2 of Supplementary Information). For exposition purposes, we termed this composite measure engaged learning, which was computed for each participant by averaging the Z scores of each measure because we did not have a priori notions about differential weights. As shown in Fig. 4, a 2 (interpolated activity) × 3 (distractor) ANOVA revealed that interpolated retrieval increased engaged learning, F(1, 697) = 32.27, p < 0.001, η_p² = 0.044, 95% CI = [0.019, 0.078]. Neither the main effect of distractor, F(2, 697) = 0.41, p = 0.662, η_p² = 0.001, 95% CI = [0.000, 0.009], BF₀₁ = 30.77, nor the interaction was significant, F(2, 697) = 1.69, p = 0.185, η_p² = 0.005, 95% CI = [0.000, 0.018], BF₀₁ = 5.45.

Fig. 4. Engaged learning composite score as a function of interpolated activity and onscreen distraction.

Error bars are 95% confidence intervals. Participants in the control condition did not have any onscreen distractors (N = 235). Participants in the medium distractor condition had three memes on the right side of the screen during the lecture (N = 235). Participants in the strong distractor condition had three TikTok videos playing on the right side of the screen throughout the lecture (N = 233). Each dot represents the data of an individual participant. The dots are displaced horizontally to improve visibility. The y-axis is truncated at -1 and +1 to promote visibility of the central tendency.

If IRP potentiates new learning by promoting engagement with the lecture content, it should mediate the effect between interim activity (our independent variable) and test performance. Note that we have preregistered this mediation analysis conceptually, although the preregistration did not specify exactly how we would represent the concept of engaged learning. After removing participants with missing data (in either the mediator or the DVs), a mediation model⁸⁴ (see Fig. 5) with 622 participants revealed a significant indirect effect for criterial test performance, β = 0.15, 95% CI = [0.09, 0.21], proportion-mediated = 0.65, p < 0.001, as well as cumulative test performance, β = 0.14, 95% CI = [0.08, 0.20], proportion-mediated = 0.45, p < 0.001. A sensitivity analysis showed that the smallest proportion-mediated our sample could detect with 0.50 power was 0.08. We chose 0.50 power for the sensitivity analysis based on a recommendation for exploratory analysis⁸⁵. Therefore, unlike the mind wandering data alone, the composite engaged learning score revealed a mediation between interpolated activity and criterial test performance.

Fig. 5. Mediation model (N = 622) depicting the relationship between interpolated activity, engaged learning composite score, and criterial and cumulative test performance.

Values are standardized parameter estimates.

Exploratory Analysis on Achievement Gaps

The racial diversity of the community college samples provided an opportunity to explore whether IRP could reduce race-based achievement gaps (note that this analysis was also preregistered conceptually but not precisely). Because we needed to break down the data by race, we collapsed across onscreen distraction conditions to increase power. Moreover, we only included data from race groups with at least 20 participants per condition (see Table 2). We acknowledge that this threshold is arbitrary, but it was necessary to ensure reliable outcomes.

As can be seen in the left panel of Fig. 6, there was a substantial achievement decrement in criterial test performance for Black students (M = 0.33) relative to White (M = 0.50), Asian (M = 0.51), and Hispanic/Latinx (M = 0.49) students in the interpolated review condition, F(3, 301) = 2.77, p = 0.042, η_p² = 0.027, 95% CI = [0.000, 0.065]. Strikingly, this achievement gap was greatly reduced in the interpolated retrieval condition (see right panel of Fig. 6), F(3, 304) = 1.10, p = .348, η_p² = .011, 95% CI = [0.000, 0.036], BF₀₁ = 9.09, such that Black students (M = 0.51) performed similarly to White (M = 0.58), Asian (M = 0.56), and Hispanic/Latinx students (M = .49). To be fully transparent, the 2 (interpolated activity) × 4 (race) interaction was not significant, F(3, 605) = 0.94, p = 0.420, η_p² = 0.005, 95% CI = [0.000, 0.017], BF₀₁ = 12.87, perhaps because the smaller sample sizes of the minority groups limited statistical power (see Table 2). So it is important to interpret these results with caution. A sensitivity analysis showed that the smallest effect size our sample could detect with 0.50 power was .018 (N = 304 for this analysis, and the remaining analyses in this section all had equal or slightly larger samples).

Fig. 6. Criterial test performance as a function of interpolated activity and race.

AS refers to Asians/Asian-Americans/Asian-Canadians (N = 76), BL refers to Black (N = 67), H/L refers to Hispanic/Latinx (N = 49), WH refers to White (N = 421). Error bars are 95% confidence intervals. Each dot represents the data of an individual participant. The dots were displaced horizontally to improve visibility.

To ensure that the achievement gap findings were not driven by sampling differences—which is a distinct possibility given the small sample sizes for the minority groups in this analysis – we conducted the same analyses as above but replaced criterial test performance with participants’ self-reported GPA. If sampling differences had contributed to the achievement gap and its reduction, such that the Black students in the interpolated review condition happened to be lower-achieving and those in the interpolated quiz condition higher-achieving, we should observe lower GPA for the Black students relative to other students in the interpolated review condition but not in the interpolated quiz condition. Critically, unlike test performance, GPA exhibited racial achievement gaps regardless of whether participants were in the interpolated review, F(3, 281) = 4.51, p = 0.004, η_p² = 0.046, 95% CI = [0.005, 0.095] (M_Black = 2.96, M_White = 3.43, M_Asian = 3.56, M_Hispanic = 3.32), or interpolated quiz condition, F(3, 284) = 12.17, p < 0.001, η_p² = 0.114, 95% CI = [0.049, 0.181] (M_Black = 2.76, M_White = 3.36, M_Asian = 3.74, M_Hispanic = 3.19). If anything, the GPA gap between Black and White students was at least as large in the interpolated quiz condition (d = 0.85) as in the interpolated review condition (d = 0.72). Therefore, the beneficial effects of IRP on racial achievement gaps was not driven by a sampling error.

For cumulative test performance, we did not include Hispanic/Latinx students because they did not meet the sample size cutoff. Unlike the criterial test results, there was an achievement gap for students in both the review, F(2, 245) = 3.85, p = .023, η_p² = .030, 95% CI = [0.000, 0.080], and quiz conditions, F(2, 250) = 3.71, p = 0.026, η_p² = 0.029, 95% CI = [0.000, 0.077], with Black students (M_review = .40, M_quiz = .55) underperforming Asian students (M_review = .52, M_quiz = .61) and White students (M_review = 0.55, M_quiz = 0.67). Despite this result, we emphasize that Black students exhibited at least the same gains from retrieval practice as the other racial groups; therefore, IRP either reduced achievement gaps (for criterial test performance) or did not magnify it (for cumulative test performance).

Together, these results demonstrate that interpolated retrieval practice might, under some circumstances, reduce race-based achievement gaps. Although these data are intriguing and promising, they must be interpreted with caution because of the small sample size for the racial minority groups (N ≈ 30 per condition) relative to White participants (N > 180 per condition). Nevertheless, the sample sizes of the minority groups here are comparable to many experimental intervention studies^86,87. Lastly, one might wonder if our data also demonstrated achievement gaps based on socioeconomic status (SES). They did not. Specifically, neither participants in the interpolated review, t(321) = 0.23, p = 0.817, d = -0.03, 95% CI = [-0.25, 0.20], BF₀₁ = 7.86, nor those in the interpolated quiz conditions showed an SES-based achievement gap, t(321) = 0.98, p = 0.330, d = -0.11, 95% CI = [-0.33, 0.11], BF₀₁ = 5.13. The negative effect sizes here indicate that lower SES participants performed slightly better than higher SES participants. These null effects might be driven by pronounced range restrictions in the socioeconomic data, given that nearly 90% of the participants reported that they were from middle or working class. For transparency and completeness, these data are presented in full in Supplementary Note 3 and Table S3.

Discussion

Several key findings emerged from this experiment. First, IRP improved new learning relative to interpolated review, and this result applied equally to university and community college students. However, the benefit of IRP varied with onscreen distractions, such that it was eliminated for participants in the strong-distraction condition. Second, we observed the benefits of IRP on test performance even when it did not significantly reduce the frequency of mind wandering, but an exploratory composite variable that combined multiple attention/engagement-based metrics revealed that the TPNL effect was mediated by task engagement. Third, IRP enhanced delayed, cumulative test performance regardless of onscreen distractions. Lastly, we found preliminary evidence that IRP might reduce race-based achievement gaps between Black and White students. We now discuss the implications of these findings.

Lab-based research has provided ample evidence that IRP can enhance learning, but as we discussed earlier, much of this research occurred under “pristine” laboratory conditions. Here, we showed that IRP enhanced online learning even when we imposed no “guard rails” against distractions. However, the IRP advantage over interpolated review was absent for participants who watched the lecture with onscreen TikTok videos. This finding and the mediation analysis results suggest that attention during lecture viewing was crucial for the test-potentiated new learning effect.

Online learning and attention

To gain a better understanding of the TPNL phenomenon, it is important to contextualize our effect size against the broader literature. Specifically, even in the no-distractor condition (d = 0.37), our effect size was more modest than in many laboratory studies. For example, Chan et al. reported a meta-analytic effect size of g = 0.75 (k = 84)⁶. We do not believe that the moderate TPNL effect observed here was specific to our lecture materials, because other studies have demonstrated powerful TPNL benefits using similar lectures in the lab (ds > 1.00)^7,60,88,89. Nor do we believe that the modest effect was specific to our public university and community college samples (e.g., participants in other studies were from elite universities^7,60), since both groups exhibited similar effect sizes, despite the university students having significantly outperformed community college students.

An intriguing alternative possibility is that the uncontrolled online learning environment, even for participants in the no-distractor condition, already acted as a divided attention condition relative to in-lab environments. This hypothesis is consistent with the idea that IRP potentiates new learning because it promotes attentive encoding after retrieval, so environments (online learning) or manipulations (onscreen distractions) that interfere with attentional control should reduce the TPNL effect. If one assumes that the typical online learning environment is highly distracting, then entertaining stimuli that promote engagement with the experiment itself might enhance learning, even if they are irrelevant to the lecture. Specifically, in the absence of the onscreen distractions or interpolated quizzes, participants in the interpolated review condition might be prone to disengaging entirely from the experiment (e.g., web browsing, texting, mind wandering). But the TikTok videos might ironically keep some participants engaged (Fig. 4), thereby promoting learning for participants in the interpolated review condition. This ironic effect of distraction aligns with the practice of using humor or offering breaks to raise student engagement, despite these approaches seemingly running counter to typical pedagogical objectives^90–92.

Limitations

Given the unexpected nature of this ironic effect, we must interpret it with caution. Further, it is likely that the match between the modality of the distractor (silent videos or pictures) and target material (e.g., voiced lecture) would interact. For example, the beneficial effects of our video distractors might not hold if both the distractor and the target material were voiced, because stimuli that occupy the same perceptual domain often produce powerful interference⁹³. An additional discrepancy between our study and other divided attention studies was that we aimed to implement distractors that resembled the typical passive distractors encountered by online learners (e.g., click-baits, other opened applications), so participants could ignore them - as they could in the real world. In contrast, most lab-based divided attention studies required participants to dual-task, such that they must respond to multiple stimuli simultaneously^94,95. Future research can address whether our results – particularly the interaction between interpolated activity and distractions – would generalize to dual-task situations when students must respond to irrelevant stimuli (e.g., chatting on instant messaging platforms).

A potential intervention to reduce achievement gaps

The large sample of the present study and the involvement of the (more diverse) community college students allowed us i) to demonstrate race-based achievement gaps in our data and ii) to show that IRP could reduce these achievement gaps for Black students. However, it is interesting that IRP reduced achievement gaps for criterial test performance but not for cumulative test performance. Based on this dissociation, one might suggest that our results are less than impressive. We argue that, to the contrary, these findings should be investigated further because they suggest the possibility that a low-cost, scalable intervention can be leveraged to reduce some race-based achievement gaps, which are notoriously prevalent and intractable⁹⁶. For example, despite having spent billions of dollars on several educational reforms, the achievement gaps in the U.S. have remained stubbornly consistent for decades⁹⁷.

Constraints on generality

A promising aspect of the present finding is that the intervention of IRP had an immediate impact on students’ learning, and it did not require a multi-day commitment or extensive training from students and instructors⁸⁶. A shortcoming of our results is that we had a small sample size for the minority groups (e.g., 67 Black students took the criterial test and 58 took the cumulative test), so these data are by no means representative, but we must also emphasize that experimental intervention research aimed at reducing racial achievement gaps often included similar⁸⁷ or even smaller samples⁸⁶. Moreover, the number of intervention studies in education has declined rapidly in recent years, but researchers have made ever more practice or policy recommendations based on correlational data^98,99. This problematic trend can only be reversed through more experimental research. We believe that interventions aimed at reducing the long-standing racial achievement gaps warrant further experimental investigations.

Conclusions

In the present study, we aimed to provide a rigorous evaluation of interpolated quizzes as an online learning intervention. To this end, we produced four high-quality STEM lectures taught by eight instructors, and we examined the efficacy of our intervention with students across seven institutions, including community college students, who have been profoundly underrepresented in experimental research. Using a realistic manipulation of onscreen distractions, we also examined a key theoretical account of the test-potentiated new learning effect—that IRP enhances new learning because it promotes engagement with the material. Together, our study showed that IRP can be implemented in online lectures to boost student learning. Not only did interpolated quizzes promote new learning immediately, they also enhanced delayed recall of the lecture material. Lastly, our data showed that IRP might reduce race-based achievement gaps in some situations.

Author contributions

Chan and Szpunar conceptualized the research. Chan and Ahn produced the materials for the research. Ahn programmed the experiment. Ahn, Assadipour, and Gill collected the data. Chan and Ahn conducted the analysis. Chan wrote the original draft of the manuscript. All authors contributed comments to the manuscript. Ahn was at Iowa State University when this work was conducted.

Peer review

Peer review information

Communications Psychology thanks Andrew Butler and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editors: Troby Ka-Yan Lui. A peer review file is available.

Data availability

All analyses were conducted using JASP. The data and the lecture materials are available at https://osf.io/3fx8d/?view_only=a2228ccf7c4540bb923ed9e54f7a0441.

Code availability

The R codes for the figures are available at 10.17605/OSF.IO/3FX8D or https://osf.io/3fx8d/?view_only=a2228ccf7c4540bb923ed9e54f7a0441.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s44271-025-00234-5.