Interpolated memory tests reduce mind wandering and improve learning of online lectures

Abstract

The recent emergence and popularity of online educational resources brings with it challenges for educators to optimize the dissemination of online content. Here we provide evidence that points toward a solution for the difficulty that students frequently report in sustaining attention to online lectures over extended periods. In two experiments, we demonstrate that the simple act of interpolating online lectures with memory tests can help students sustain attention to lecture content in a manner that discourages task-irrelevant mind wandering activities, encourages task-relevant note-taking activities, and improves learning. Importantly, frequent testing was associated with reduced anxiety toward a final cumulative test and also with reductions in subjective estimates of cognitive demand. Our findings suggest a potentially key role for interpolated testing in the development and dissemination of online educational content.

Online education is quickly becoming a central fixture in the college curriculum. The availability of free online courses with massive enrollments including students from all over the world (e.g., www.edX.org, www.coursera.org, http://2u.com, www.Udacity.com) has developed rapidly and captured widespread public attention. Within brick and mortar colleges, instructors are increasingly making use of flipped classrooms (1), whereby students are encouraged to study lectures on their own time and engage in activities geared toward a more in-depth understanding of the subject matter in the classroom. As such, institutions of higher education have devoted considerable time and effort to making large-scale and highly accessible online repositories of classroom lectures available to both students and the general public. Indeed, recent surveys indicate that students are increasingly using online lectures as a primary learning tool (2). At the same time, little is known about the potential limitations to learning from online lectures and how those limitations can be overcome. For instance, college students frequently report lapses of attention during lectures (3–6), and the tendency to mind wander (7–9) while viewing videotaped lectures has been shown to result in impoverished learning of lecture content (10). Such observations raise the need for rigorous investigations of learning from online lectures. For example, what interventions might remedy the tendency for students to mind wander while viewing online lectures and also allow them to quickly and efficiently extract lecture content? Here we test, and find support for, the hypothesis that interpolating online lectures with memory tests can both reduce the occurrence of mind wandering during lectures and foster task-relevant activities, such as note taking (11), that facilitate learning of lecture content.

Our approach is based on recent studies demonstrating that interpolating the study of lists of words (12), face–name pairs (13), and prose passages (14) with memory tests can substantially improve the typically impoverished learning that takes place toward the end of extended study sequences (15). For instance, in one study students learned five lists of words and were told that each list would be followed by a memory test for the most recent list or an unrelated mental activity (12). Moreover, students were told that a computer program would randomly determine the frequency of testing but that there would be a final cumulative test. In fact, there were two testing schedules. One group was tested after every list, whereas another group was only tested after the fifth and final list. Students who were tested after each list learned the final list twice as well as students who had not been tested until the final list. Interpolated testing facilitated the learning of the final list in the study sequence.

Interpretations of the facilitative effect of interpolated testing on subsequent learning have focused on a number of cognitive mechanisms that are thought to be at work during encoding [i.e., when students are actively learning (16)] and retrieval [i.e., when students bring study materials back to mind after learning (12, 17)]. Here we focus on factors operating during encoding (SI Materials and Methods provides discussion, and evidence, of factors operating at retrieval). In particular, recent evidence from cognitive neuroscience (16) and cognitive psychology* suggests that interpolating extended periods of study with memory tests can motivate students to focus on study materials in a manner that benefits learning. For instance, electrophysiological studies of brain activity have demonstrated that lapses of attention are associated with elevated amplitude in the α-frequency band (8–14 Hz) (18–21) and that interpolating extended periods of study with testing can ward off increases in α-power and sustain high levels of learning over time (16). These and related findings carry potentially important implications for learning in real-world contexts, such as lectures, where lapses of attention are frequently reported to disrupt learning (3–6, 10, 11, 22). Nonetheless, the hypothesis that interpolated testing helps students attend to and retain the contents of a lecture remains to be directly tested. Here we conducted a critical test of this hypothesis by directly assessing the extent to which interpolating online lectures with memory tests helped students to resist the tendency to engage in mind wandering and facilitate task-relevant activities (e.g., note taking) in a manner that improves learning.

In two experiments (n = 80), students were asked to learn a 21-min video lecture (Introduction to Statistics; further details in Materials and Methods). In each experiment, students were given the lecture slides/notes and instructed to learn the lecture as they would in the classroom. Moreover, students were informed that the video lecture would be divided into four segments and that there would be a break between each segment. During each break, students were told that they would first spend 1 min completing arithmetic problems (unrelated to the lecture) and that the arithmetic problems would be followed by either a 2-min test about the most recent lecture segment or two more minutes of arithmetic problems. In experiment 2, students were told that each lecture segment could also possibly be followed by a 2-min period during which test questions were presented along with the answers (i.e., a restudy phase). In both experiments, students were told that the frequency with which they would receive tests would be randomly determined but that there would be a final cumulative test. The actual frequency of testing during the lecture was manipulated in a between-subjects manner.

In experiment 1 (n = 32), half of the students were tested after each lecture segment (“tested group”), and half were only tested after the fourth lecture segment (“nontested group”) (further details in Fig. 1A and Materials and Methods). After the lecture and before the final cumulative test, students used seven-point rating scales to indicate the extent to which they had mind wandered during the lecture and their level of anxiety toward the final test. Direct comparisons between the tested and nontested groups allowed us to assess the extent to which interpolated testing during the lecture helped students to avoid task-irrelevant activities such as mind wandering, engage in task-relevant activities such as note taking (further details in Results), and improve learning of the tail end of the lecture. In experiment 2 (n = 48), we provide a replication of our initial results but also include a more direct measure of mind wandering during the lecture (further details in Fig. 1B and Materials and Methods) and further rule out the possibility that the benefits of testing might be attributable to the reexposure to study materials that accompanies testing. This latter aspect of the experiment was accomplished by including a “restudy group” who, like the nontested group, were only tested after the fourth lecture segment but who were exposed to questions paired with answers after the first three lecture segments (further details in Fig. 1B and Materials and Methods). As with experiment 1, students in experiment 2 completed a number of phenomenological rating scales that indicated their level of anxiety toward the final cumulative test (including positive and negative affective reactions) and also the extent to which they found the experience of learning the lecture to be mentally taxing.

Fig. 1.

Schematic of the design of experiments 1 and 2. (A) In experiment 1, students were instructed that they would learn a 21-min excerpt of a statistics lecture that was divided into four segments (∼5.5 min each). Students were further told that each lecture segment would be followed first by 1 min of arithmetic problems (e.g., 13 × 7; six problems; 10 s per problem), and then either a 2-min quiz about the most recent lecture segment (e.g., “Explain the relation between a population and a sample”; six questions; 20 s per question) or 2 more min of arithmetic (12 problems; 10 s per problem). In fact, there were two testing schedules. One group of students was tested after each lecture segment (tested), whereas another group was only tested after the fourth lecture segment (nontested). To assess the learning of the end of the lecture, we compared the proportion of questions students in the tested and nontested groups correctly answered about the fourth lecture segment. (B) In experiment 2, students were told that each lecture segment, and the arithmetic problems that followed, would be followed by either (i) a 2-min quiz about the most recent lecture segment (tested group; six questions; 20 s per question), (ii) a restudy phase during which participants would be presented with answers to key questions about the most recent lecture segment (restudy group; six question–answer pairs; 20 s per question–answer pair; same questions as tested group), or (iii) more arithmetic problems (nontested group; 12 problems; 10 s per problem). To assess learning of the end of the lecture, we compared the proportion of questions students in the tested, restudy, and nontested groups correctly answered about the fourth lecture segment. Additionally, students in experiment 2 were asked to indicate whether their minds had wandered away from the content of the lecture at four random intervals during the lecture (once per lecture segment).

Results

Experiment 1.

Students in the tested, relative to the nontested, group reported having experienced significantly fewer bouts of mind wandering (Mdn = 4 vs. 5), z = 2.02, P = 0.044, and further indicated that the occurrence of mind wandering was significantly less likely to have increased as the lecture progressed (Mdn = 3.5 vs. 5.5), z = 2.23, P = 0.026. To examine more concretely the effect of repeated testing on attention to lecture details, we further examined the extent to which students took additional notes during the lecture. We operationalized this measure as the number of lecture slides on which students made idiosyncratic notes. Corroborating the results of the mind wandering measures collected after the lecture, students in the tested group took significantly more notes during the lecture than students in the nontested group (M = 17% vs. 6% of slides with additional notes), t(30) = 2.63, P = 0.013, d = 0.93 (Fig. 2A). Importantly, testing for lecture segments 1–3 was associated with improved learning of the fourth lecture segment. Students in the tested group correctly answered significantly more questions about the fourth lecture segment than students in the nontested group (M = 84% vs. 59%), t(30) = 2.87, P = 0.008, d = 1.03 (Fig. 1A; analyses related to factors operating at retrieval are provided in SI Materials and Methods). (Comparisons of performance across the four segments of the lecture for the tested group were not carried out because the order of presentation of lecture segments was not manipulated during this study. This restriction similarly applies to the results of experiment 2.)

Fig. 2.

(A) Experiment 1. Proportion of slides for which students made additional notes during the 21-min lecture (Left), and mean number of questions answered correctly about the fourth lecture segment (Right); NT, nontested group; T, tested group. (B) Experiment 2. Proportion of mind wandering probes during which students reported having engaged in mind wandering away from lecture content (Left), proportion of slides for which students made additional notes during the 21-min lecture (Center), and mean number of questions answered correctly about the fourth lecture segment (Right); NT, nontested group; RS, restudy group; T, tested group. Error bars represent SEMs.

Experiment 2.

A one-way ANOVA revealed a significant effect of condition in relation to the occurrence of mind wandering, F(2,45) = 3.43, P = 0.041, η² = 0.16. During the lecture, students in the tested group reported fewer instances of mind wandering (M = 19% of probes) than students in the restudy (M = 39%) and nontested groups (M = 41%), t(30) = 2.34, P = 0.026, d = 1.05; t(30) = 2.48, P = 0.019, d = 1.01, respectively (Fig. 2B). Notably, theories of mind wandering distinguish between factors that determine the frequency of occurrence of mind wandering from those related to processes that sustain mind wandering over time (23). If testing affects the intensity of mind wandering in addition to the frequency of mind wandering, then the slope that relates mind wandering to comprehension should differ across tested and nontested groups. Accordingly, we conducted an additional analysis to examine whether the slopes (regression coefficients) characterizing the relation of mind wandering to comprehension (i.e., test performance) differed across the tested, restudy, and nontested groups. Irrespective of group membership, the regression slope characterizing the relation of mind wandering to comprehension (b = −2.10) was significant, t(46) = 4.77, P < 0.001. However, comparisons across groups did not reveal a significant interaction, F(5,42) = 1.18, P = 0.32, suggesting that our encoding manipulation did not impact the intensity of the influence of mind wandering on comprehension across groups. Hence, although testing curbed the occurrence of mind wandering, testing was not related to the intensity of influence of mind wandering on learning. Further corroborating these results, a one-way ANOVA revealed a significant effect of condition for note taking, F(2,45) = 9.45, P < 0.001, η² = 0.30. Between-group comparisons revealed that students in the tested group (M = 24% of slides with additional notes) took significantly more notes during the lecture than students in the restudy (M = 9%) and nontested groups (M = 7%), t(30) = 3.55, P = 0.001, d = 1.26; t(30) = 3.93, P < 0.001, d = 1.39, respectively (Fig. 2B). Importantly, a one-way ANOVA revealed a significant main effect of condition for number of questions correctly answered for the fourth lecture segment, F(2,45) = 4.95, P = 0.011, η² = 0.18. Between-group comparisons revealed that students in the tested group correctly answered more questions about the fourth lecture segment (M = 89%) than students in the restudy (M = 65%) and nontested groups (M = 70%), t(30) = 2.88, P = 0.007, d = 1.06; t(30) = 2.85, P = 0.008, d = 1.01, respectively (Fig. 2B; analyses related to factors operating at retrieval are provided in SI Materials and Methods). The restudy and nontested groups did not differ from one another in any respect (ts < 1). Hence, the presence of tests, and not merely the reexposure to study materials that accompanies tests, encouraged students to attend to lecture content in a manner that reduced mind wandering, increased note taking, and facilitated learning.

Phenomenological Ratings.

The use of frequent testing during lectures raises possible concerns regarding test anxiety and cognitive demands placed on students. Notably, there was a significant effect of condition on anxiety, χ²(2, n = 48) = 13.16, P = 0.001, such that students in the tested group (Mdn = 2) reported significantly lower levels of anxiety toward the final cumulative test than students in the restudy (Mdn = 4) and nontested groups (Mdn = 3.5), z = 3.78, P < 0.001; z = 2.29, P = 0.003, respectively. Supporting this result, there was also a significant effect of condition on negative affect toward the final cumulative test in experiment 2, F(2,45) = 8.68, P = 0.001, η² = 0.28, such that students in the tested group reported significantly less negative affect (M = 12/50 on negative affect subscale) in relation to the final cumulative test than students in the restudy (M = 22) and nontested groups (M = 17), t(30) = 4.04, P < 0.001, d = 1.65; t(30) = 2.93, P = 0.006, d = 1.13, respectively (there was no effect of condition on positive affect). Finally, there was a significant effect of condition on subjective estimates of cognitive demand, χ²(2, n = 48) = 9.19, P = 0.01. After the lecture, students in the tested group reported that their experience of learning the lecture was less mentally taxing (Mdn = 2) than students in the restudy (Mdn = 4) and nontested groups (Mdn = 3.5), z = 2.96, P = 0.003; z = 2.15, P = 0.032, respectively.

Discussion

Taken together, the present results demonstrate that interpolating an online lecture with testing can help students to quickly and efficiently extract lecture content by reducing the occurrence of mind wandering, increasing the frequency of note taking, and facilitating learning. Importantly, additional analyses demonstrated that the benefits of testing on subsequent learning extended beyond the lecture. An analysis of the final cumulative test across both experiments demonstrated that students in the tested group retained more information from the fourth lecture segment on the final test (M = 86%) than students in the restudy (M = 66%) and nontested (M = 72%) groups, smallest t = 2.50, P = 0.015, d = 0.63 (analyses related to factors operating at retrieval are provided in SI Materials and Methods). Moreover, students in the tested group also scored better across the entire final cumulative test (i.e., across all four segments of the lecture; M = 90%) than students in the restudy (M = 76%) and nontested (M = 68%) groups, smallest t = 3.31, P = 0.002, d = 1.38. There were no differences between the restudy and nontested groups in either respect, largest t = 1.51, ns. Hence, the act of retrieving relevant lecture content was critical for enhancing retention (24).

Recent research in cognitive psychology suggests that, when implemented appropriately, tests can be used to significantly enhance learning (25). The present results demonstrate one such function of testing and highlight the specific cognitive mechanism by which testing can facilitate learning. In particular, testing can be used to help students sustain attention to lecture content in a manner that discourages task-irrelevant (mind wandering) and encourages task-relevant (note taking) activities, and hence improves learning. Importantly, the benefits of testing for learning were accompanied by reductions in test anxiety (possibly because students became accustomed to testing style or as a result of positive feedback from earlier tests) and subjective estimates of cognitive demand. Future work can further delineate the specific parameters of interpolated tests that might more optimally facilitate the learning of online lecture materials, including frequency of testing, types of retrieval tests, or whether retrieval must be specifically associated with the lecture content (16). The use of online lectures as a learning tool represents a notable advancement in education and brings with it the responsibility of educators to devise techniques that can help students and laypersons make efficient use of their study time. The present results represent an initial step in that direction.

Materials and Methods

Lecture Videos in Experiments 1 and 2.

Harvard undergraduates were required to learn an introductory lecture in statistics that is offered through the Department of Economics. The video was divided into four segments (∼5.5 min each) using iMovie software (Apple). The four lecture segments were interpolated with arithmetic problems and questions designed to test understanding of key concepts (e.g., “What is the relation between a population and a sample?”), and the final sequences of events were presented to students using E-Prime 2.0 software (Psychology Software Tools, Inc.) on a Dell desktop computer. Responses to arithmetic problems and lecture questions were made using a computer keyboard (details regarding scoring of test questions are provided in SI Materials and Methods). All participants provided informed written consent in accordance with the guidelines set by the Harvard University Institutional Review Board.

Experiment 1.

Upon arriving to the experiment, students in experiment 1 (n = 32) were given the lecture slides/notes and instructed to learn the lecture as they would in the classroom. Moreover, students were told that the lecture would be divided into four segments and that after each segment they would answer six arithmetic problems (10 s per problem). After the arithmetic problems, students were told that they would either be asked to answer six questions about key concepts from the most recent segment of the lecture (20 s per question) or complete more arithmetic problems (12 problems; 10 s per problem). An experimenter present in the room ensured that lecture notes were hidden from view during the intervals between lecture segments (i.e., during the math problems and test questions). Finally, students were told that a computer program would randomly determine the frequency of testing during the lecture but that there would be a final cumulative test after the lecture (all 24 questions from the four lecture segments; presented in a new random order; self-paced). In fact, there were two testing schedules. One group of students was tested after each of the four lecture segments (tested), whereas another group of students was only tested after the fourth lecture segment (nontested) (Fig. 1A). To assess the influence of testing on subsequent learning, we compared the tested and nontested groups on the number of correctly answered questions for the fourth lecture segment. After the lecture and before the final cumulative test, students were given a 5-min mental break during which they played an online video game (Tetris, Tetris Online Inc.). Immediately before receiving the final cumulative test, students were asked to complete a number of phenomenological ratings. Specifically, students were asked to use a seven-point scale (1 = not at all; 7 = very much) to rate (i) how much they felt that their minds had wandered during the lecture, (ii) how much they felt that their mind wandering had increased as the lecture progressed, and (iii) how anxious they were about the final test.

Experiment 2.

In experiment 2 (n = 48), we characterized the occurrence of mind wandering more directly by including thought-sampling probes during the lecture. In addition, we evaluated the possibility that the additional exposure to study materials that accompanied testing could account for the observed reduction in mind wandering, increase in note taking, and increase in learning of the fourth lecture segment. The procedure for experiment 2 was identical to that of experiment 1, with the following exceptions. To assess mind wandering, students were informed before the lecture that the experimenter present in the room would, at random points during the lecture, verbally cue them to indicate whether or not their attention had strayed away from lecture content (i.e., “Are you mind wandering?”). Students were told that the lecture video would not be stopped during verbal mind wandering probes and that they should respond to each probe by marking a yes/no on a separate sheet of paper. Students were given no indication of how many probes they should expect. The mind wandering probes occurred once during each segment of the lecture (four total probes). For each lecture segment, the mind wandering probe was administered at some random time point that occurred at least 30 s into the lecture segment and at least 30 s before the end of the lecture segment. To control for exposure to study materials, students were instructed that each segment of the lecture, and the initial arithmetic problems that followed, would be followed by either (i) six questions about key concepts from the most recent segment of the lecture (20 s per question), (ii) six questions about the most recent segment of the lecture accompanied by the answers (20 s per question–answer pair), or (iii) more arithmetic problems (12 problems; 10 s per problem). In fact, there were three testing schedules. One group of students was tested after each of the four lecture segments (tested), one group studied question–answer pairs after lecture segments 1–3 and was tested after lecture segment 4 (restudy), and one group completed additional arithmetic problems following lecture segments 1–3 and was tested after lecture segment 4 (nontested) (Fig. 1B). Finally, in addition to the phenomenological rating of anxiety that students completed before the final test, students also completed the 20-item Positive and Negative Affect Schedule to assess more concretely the extent to which they held positive and negative feelings toward the final cumulative test. Students were also asked to use a seven-point scale (1 = not at all; 7 = very much) to indicate how mentally taxing they found the experience of learning the lecture.