Metacognition of the testing effect: Guiding learners to predict the benefits of retrieval

Abstract

If the mnemonic benefits of testing are to be widely realized in real-world learning circumstances, people must appreciate the value of testing and choose to utilize testing during self-guided learning. Yet metacognitive judgments do not appear to reflect the enhancement provided by testing (Karpicke & Roediger, 2008). In this paper, we show that under judicious conditions learners can indeed reveal an understanding of the beneficial effects of testing as well as the interaction of that effect with delay (Experiment 1). In that experiment, subjects made judgments of learning (JOLs) for previously studied or previously tested items in either a cue-only or cue-target context, and either immediately or after a one-day delay. When subjects made judgments in a cue-only context, their JOLs accurately reflected the effects of testing, both immediately and at a delay. To evaluate the potential of exposure to such conditions for promoting generalized appreciation of testing effects, three further experiments elicited global predictions about re-studied and tested items across two study/test cycles (Experiments 2, 3, and 4). The results indicated that learners’ global naïve metacognitive beliefs increasingly reflect the beneficial effects of testing when learners experience these benefits with increasing external support. If queried under facilitative circumstances, learners appreciate the mnemonic enhancement that testing provides on both an item-by-item and global basis, but generalize that knowledge to future learning only with considerable guidance.

Guiding learners to predict the benefits of retrieval

For research on learning and memory to be relevant to students who wish to enhance their performance in the classroom, that research must acknowledge the fact that a significant portion of learning occurs outside of the classroom, under the supervision of only the student. In circumstances in which no teacher directly guides the learning activities, learners must rely upon their own metacognition to determine what they need to study, how to study, and when to cease study. Self-regulated aspects of learning have significant implications for the effectiveness of students’ learning efforts and achievement in education (Dunlosky & Theide, 1998). For example, how study time is allocated across items often determines how much is remembered (Son & Kornell, 2008; Tullis & Benjamin, 2011). Being an effective learner requires the ability to make appropriate study decisions (e.g., Finley, Tullis, & Benjamin, 2009; Metcalfe, 2009), and the effectiveness of these decisions is directly modulated by the quality of metacognitive monitoring (Metcalfe & Finn, 2008; Thiede, Anderson, & Therriault, 2003). When monitoring judgments are inaccurate or biased, study decisions can result in suboptimal learning (Atkinson, 1972; Kornell & Bjork, 2008; Tullis & Benjamin, 2011).

In this paper, we consider whether learners are sensitive to the mnemonic effects of testing. We will briefly review the testing effect and consider extant research suggesting that learners fail to accurately monitor the mnemonic effects of testing. After that, we report four experiments that evaluate the extent to which learners do accurately monitor the mnemonic effects of testing. Experiment 1 investigates whether learners’ metacognitive judgments reflect the mnemonic benefits of testing under conditions that promote judgments based on mnemonic cues rather than naïve theory (Kelley & Jacoby, 1996; Koriat, 1997). Experiments 2, 3, and 4 address whether learners attribute improved memory performance to testing and whether this knowledge generalizes to global judgments about future learning.

Metacognition and the Testing Effect

Retrieval has enormous potential to enhance long-term retention, particularly if learners appreciate its benefits and utilize it properly during self-regulated learning. However, learners’ metacognitive judgments fail to reflect the advantages that successful retrieval provides for long-term retention. In order to assess whether learners recognize the benefits that testing provides, researchers have surveyed undergraduate students about their real-life study habits. When students free report the study strategies they use, 11% report that they practice retrieval, 40% report using flashcards, and 43% report practicing solving problems (the latter two options could be viewed as a means of self-testing; Karpicke, Butler, & Roediger, 2009). When choosing the study activities they use from a given list of options, 18% of students report using self-testing as a means of studying. The percentage of students who report using self-testing when they have an opportunity to re-study afterwards increases to 42%. However, another survey provides a much higher estimate of the use of self-testing by suggesting that up to 71% of students regularly test themselves with practice problems (Hartwig & Dunlosky, 2012). Even with the highest estimates of self-testing, far from all students report using self-testing to bolster mnemonic performance.

While many learners do not utilize retrieval during study to benefit retrieval, those that do recognize only the indirect benefits of testing. A majority of students who report using self-testing report that they use testing as a means of assessing rather than improving learning, which is an indirect effect of testing. Learners are largely unaware that successful retrieval directly improves learning. When asked “If you test yourself while studying, why do you do it?”, approximately two-thirds of students report they test themselves in order to determine what they do and do not know so they can better allocate future study time (Kornell & Bjork, 2007; Kornell & Son, 2009). Only around 20% of students report that they test themselves because they learn more from testing than from re-studying. Researchers have thus argued that learners do not grasp the immense improvement that testing affords for long-term memory retention (Agarwal, Karpicke, Kang, Roediger, & McDermott, 2008; Karpicke, 2009; Karpicke & Roediger, 2008; Kornell & Son, 2009; Roediger & Karpicke, 2006).

Learners’ appreciation of the mnemonic effects of retrieval has also been assessed using judgments of learning (JOLs) with controlled stimuli such as word lists and prose passages. Roediger and Karpicke (2006) evaluated metamemory of prose passages by having subjects either study a passage multiple times or study it once and then take tests on the material. Subjects then predicted how well they would remember the passages on a test one week later. Subjects rated repeatedly re-studied passages as more memorable than tested passages, even though final free recall performance was greater for the tested passages. Kornell and Son (2009) replicated this finding using word pairs in a flashcard-like procedure. Subjects predicted higher levels of recall for word pairs that were re-studied versus tested, although final cued recall performance was greater for the tested pairs. In fact, in all previous laboratory experiments investigating the effectiveness with which learners monitor the effects of testing, subjects judged re-studied items as more likely to remembered, but actually remembered more of the previously tested items. Karpicke and Roediger (2008) thus argued that “students exhibit no awareness of the mnemonic effects of retrieval practice” (p. 968). Consequently, in laboratory paradigms in which learners control their own learning activities, they often choose to re-study rather than test themselves—a counterproductive act that decreases eventual performance on a memory test (Karpicke, 2009). However, in a survey asking about learners’ study habits, a large proportion of learners report using flash cards in order to help memorize information (Wissman, Rawson, & Pyc, 2012). This may reveal a disconnect between control used in artificial laboratory settings where learners may not value remembering information highly and classroom settings where learners may value remembering information more.

Conditions that Promote Accurate Metacognition

Conditions that promote long-term retention through “desirably difficult” processing during acquisition (Bjork, 1999) often yield metacognitive judgments that inappropriately reflect the difficulty of immediate acquisition, rather than the robustness of long-term learning. Learners’ failure to appreciate the effects of testing thus parallels similar failures to appreciate the benefits of spacing repetitions (Baddeley & Longman, 1978), interleaved practice (Zechmeister & Shaughnessy, 1980), imagery (Shaughnessy, 1981), and release from proactive interference (Diaz & Benjamin, 2011). However, in these cases, the opportunity for a comparative, diagnostic retrieval of material is an important mediating variable in whether learners appreciate the effects of manipulations of learning on long-term memory. Four factors appear to be particularly important. First, JOLs are more sensitive to differences between processing conditions when those conditions are varied within a list than when they are varied between lists or between subjects (Begg, Duft, Lalonde, Melnick, & Sanvito, 1989; Shaw & Craik, 1989). Presumably, such conditions promote metacognitive accuracy because they invite direct comparison between the learning conditions, a comparison that is more difficult between lists (and even more difficult between subjects!). Second, the type of metacognitive monitoring question posed influences the accuracy of the resultant judgments. Aggregate JOLs may depend upon different, often less diagnostic, cues than item-by-item JOLs (Mazzoni & Nelson, 1995). Third, adding a delay between study and JOLs improves the predictive accuracy of JOLs (Nelson & Dunlosky, 1991; Dunlosky & Nelson, 1994) because retrieval at a delay is more diagnostic of later retrieval probability than is immediate recall (Benjamin & Bjork, 1996). Finally, the information present when subjects make JOLs greatly impacts JOL accuracy: JOLs made when both the cue and target from a word pair are present are significantly less accurate at predicting future cued recall than predictions made when only the cue is present (Dunlosky & Nelson, 1992; Dunlosky & Nelson, 1994). It is likely that inclusion of the target during the JOL solicitation either dissuades subjects from attempting to retrieve the target or renders such retrieval nondiagnostic (Benjamin, Bjork, & Schwartz, 1998); consequently, when the target is present subjects fall back on analytic, naïve theories about memorability rather than the more accurate mnemonic strategy otherwise used (Jacoby & Kelley, 1987).

When judgments are made in contrastive, delayed, item-by-item, cue-only contexts, many failures of metacognitive monitoring are allayed: for example, subjects accurately predict the mnemonic benefits of interactive imagery, spacing, and the number of presentations (Begg et al, 1989; Carroll, Nelson, & Kirwan, 1997; Dunlosky & Nelson, 1994). The effects of testing, like most other desirably difficult learning conditions, may be metacognitively misunderstood unless care is taken to elicit judgments under contrastive, delayed, cue-only conditions. Experiment 1 compares metacognitive monitoring of the testing effect under facilitative conditions as outlined above with the more adverse conditions that are normally used to collect these judgments

Experiment 1

Extant data all seem to show a serious failure of metacognitive monitoring of the testing effect. However, these experiments all have characteristics that make the metacognitive demand quite high on the learner. First, all previous studies investigating metacognitive monitoring of the testing effect have elicited JOLs immediately after the final study/test session. This is an important detail, not only because delaying judgments enhances their accuracy, but also because at such short test intervals, re-studying actually is beneficial when compared with testing (Roediger & Karpicke, 2006). Thus these JOLs may accurately reflect the current consequences of testing, which are negative, rather than the future mnemonic strength of items when they will be tested. We elicit JOLs both shortly after testing and at a longer delay in order to determine whether learners can recognize both the mnemonic costs and benefits that testing provides. Second, all previous studies have manipulated testing between subjects or via blocked-list designs. In this study, previously tested and re-studied items are varied within a list so as to enable comparative judgments. Third, previous testing effect studies have relied upon aggregate JOLs, asking subjects to predict final memory performance across groups of tested and groups of restudied items. Prior studies gather aggregate JOLs instead of item-by-item JOLs in order to avoid confounding learning conditions (re-study vs. testing) with item-by-item JOL context (cue-only vs. cue-target). However, by only gathering aggregate JOLs, prior studies have encouraged subjects to base their judgments on analytic naïve theories about learning conditions, rather than on each item’s subjective mnemonic cues (Koriat, 1997). We solicit JOLs on an item-by-item basis in a phase following the re-studying/testing manipulation so that subjects can rely upon subjective processing cues for each item when making JOLs. Finally, we elicit JOLs in both cue-only and cue-target contexts in order to demonstrate that a metacognitive appreciation of the effects of testing depends on the opportunity to engage in diagnostic retrieval of the queried material. We predict that learners will appreciate the mnemonic costs and benefits of testing when JOLs are elicited in facilitative, delayed, cue-only conditions. Our predictions of metacognitive ratings are less clear in cue-target conditions, where learners are re-exposed to all the word pairs after the crucial testing/re-study manipulation has occurred. The added exposure to both the re-studied and tested word pairs may unpredictably alter both memory predictions and final performance.

Method

Subjects

One-hundred and twenty introductory-level psychology students from the University of Illinois at Urbana-Champaign participated in exchange for partial course credit.

Materials

Materials were 64 unassociated English word pairs (e.g., viking - napkin). All cues and targets were nouns and ranged in length from four to eight letters. Target words were selected for high concreteness (M = 572, SD = 32) and high imageability (M = 578, SD = 34), using the Medical Research Council (MRC) Psycholinguistic Database (Coltheart, 1981). Cues varied in written frequency (Kucera & Francis, 1967) from 0 to 286 (M = 43.5, SD = 73.8). Targets varied in written frequency from 1 to 591 (M = 120.8, SD = 138.5). Lack of association between cues and targets was confirmed using the University of South Florida Word Association, Rhyme, and Word Fragment Norms (Nelson, McEvoy, & Schreiber, 1998).

Design

The experiment used a 2 × 2 × 2 mixed design, in which practice condition was manipulated within subjects (Phase 2 = re-study vs. test), JOL context was manipulated between subjects (Phase 3 = cue-only vs. cue-target), and retention interval was manipulated between subjects (delay between Phase 2 and 3 = none vs. 1-day). A schematic of the four between-subject groups is displayed in Table 1. For each condition, n = 30.

Table 1.

The four between-group conditions in Experiment 1.

	Day 1				Day 2
Condition	Phase 1	Phase 2	Phase 3	Phase 4	Phase 3	Phase 4
Cue-only JOL, No delay	Study	½ re-study ½ test	Cue-only JOLs	Test
Cue-target JOL, No delay	Study	½ re-study ½ test	Cue-target JOLs	Test
Cue-only JOL, One-day delay	Study	½ re-study ½ test			Cue-only JOLs	Test
Cue-target JOL, One-day delay	Study	½ re-study ½ test			Cue-target JOLs	Test

Procedure

Subjects were run individually on computers programmed with Matlab using the Psychophysics Toolbox extensions (Brainard, 1997). The procedure consisted of four phases: initial presentation, practice, JOLs, and final test. Subjects were alternately assigned to the four conditions based upon the order in which they were run.

Phase 1: initial presentation

Subjects were shown 32 word pairs randomly chosen from the pool of 64 pairs. Pairs were presented one at a time for 4 s each with an inter-stimulus interval of 0.5 s. Presentation order was determined randomly for each subject.

Phase 2: practice

Subjects re-studied half (16) of the word pairs and were tested on the other half. Word pairs were randomly assigned to practice condition for each subject, with the constraint that exactly half of the word pairs were re-studied and half were tested. On re-study trials, the word pairs were simply re-presented for 4 s each, followed by a 0.5 s inter-stimulus interval. On test trials, the cue (left-hand) word of a pair was presented and subjects were instructed to type the corresponding target (right-hand) word, or to type a question mark if they could not remember the target word. There was no time limit for responding, and each trial was followed by a 0.5 s inter-stimulus interval. Re-study trials and test trials were interleaved, with order determined randomly for each subject.

Delay

Half of the subjects continued the experiment with no delay between Phases 2 and 3. The other half of the subjects left the lab after Phase 2 and returned the following day to complete Phases 3 and 4.

Phase 3: JOLs

Subjects were randomly assigned to make individual JOLs for all 32 word pairs in either a cue-target context or a cue-only context. The JOL phase was separated from the re-study/test phase by either no delay or a long (1 day) delay. Trial order was determined randomly for each subject.

For subjects in the cue-target context, on each trial they were shown the entire word pair for 4 s and then given the following JOL prompt: “How sure are you that you will remember this item on the upcoming final test?”. Subjects responded using a scale ranging from 1 (I am sure I will NOT remember this item) to 4 (I am sure I WILL remember this item). The presented word pair remained visible during the judgment. There was no time limit for responding, and each trial was followed by a 0.5 s inter-stimulus interval.

For subjects in the cue-only context, on each trial they were shown the cue (left-hand) word of a pair and were instructed to type the corresponding target (right-hand) word, or to type a question mark if they could not remember the target word. There was no time limit for responding. Once they had responded, subjects were given the same JOL prompt as that used in the cue-target context, with the same response scale. The presented cue word and the subject's response remained visible during the judgment. There was no time limit for responding, and each trial was followed by a 0.5 s inter-stimulus interval.

Phase 4: final test

Subjects were given cued recall test trials on all 32 word pairs, using the same procedure as the test trials in the practice phase. Order was determined randomly for each subject.

Results

First, we will consider the results for subjects who made JOLs in the presence of the cue only. Then we will consider results for subjects who made JOLs in the presence of the cue-target pair. All statistics reported here are significant at an α < .05 level unless otherwise noted. Effect sizes for comparisons of means are reported as Cohen’s d calculated using the pooled standard deviation of the groups being compared (Olejnik & Algina, 2000, Box 1 Option B). Effect sizes for ANOVAs are reported as ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2$ calculated using the formulae provided by Maxwell and Delaney (2004).

Cue-only JOL Context

Memory performance

In this and all subsequent experiments, subjects’ answers were only considered correct if the answer matched the target exactly. The mean proportion of items recalled during the phase 2 practice test was 0.49 (SD = 0.26). Figure 1 shows mean final cued recall performance and mean JOLs for subjects in the cue-only JOL context. A 2 (mode of practice) × 2 (delay) mixed model ANOVA showed that cued-recall performance significantly declined across the one-day delay (F(1,58) = 24.99, p < .001, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.424$). Critically, mode of practice interacted with delay (F(1, 58) = 43.96, p < .001, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.086$), such that re-studied items were recalled at a higher rate than tested items on the immediate test (t(29) = 4.12, p < .001, d = 0.60), and tested items were recalled at a higher rate than re-studied items after the one-day delay (t(29) = 5.75, p < .001, d = 0.66).

Figure 1.

Mean proportion correct recall on the final cued recall test (top panel) and mean judgment of learning on a scale of 1 to 4 (bottom panel) for cue-only subjects as a function of retention interval and mode of practice during Phase 2 (Experiment 1). Error bars represent the standard error of the difference scores within each delay condition.

Judgments of learning

A 2 (mode of practice) × 2 (delay) mixed model ANOVA was performed on JOLs. The JOLs, shown in the bottom panel of Figure 1, clearly reflect the pattern apparent in actual memory performance: they show both the decline of performance across the delay (F(1, 58) = 10.58, p < .001, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.372$) and the interaction between mode of practice and delay (F(1, 58) = 27.60, p < .001, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.046$), such that re-studied items were given higher JOLs than tested items at no delay (t(29) = 2.73, p = .011, d = 0.36) and tested items were given higher JOLs than re-studied items at the one-day delay (t(29) = 4.90, p < .001, d = 0.34).

Cue-target JOL Context

Memory performance

Figure 2 shows mean final cued recall performance and mean JOLs for subjects in the cue-target JOL context. Performance did not change across mode of practice (F(1,58) = 1.89; p = 0.17, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.002$) or delay (F(1,58) = 0.04; p = 0.84, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2<0$), and the interaction between delay and mode of practice was not significant (F(1,58) = 3.24, p = .08, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.005$). The added re-study opportunity during Phase 3 washed out the testing effect from the prior study phase, particularly when the re-study (Phase 3) and test (Phase 4) events took place the second day, since there was no delay between the last re-presentation of the list and the final test.

Figure 2.

Mean proportion correct recall on the final cued recall test (top panel) and mean judgment of learning on a scale of 1 to 4 (bottom panel) for cue-target subjects as a function of retention interval and mode of practice during Phase 2 (Experiment 1). Error bars represent the standard error of the difference scores within each delay condition.

Judgments of learning

As shown in Figure 3, JOLs given during re-study in Phase 3 were slightly but significantly higher for tested items (F(1, 58) = 21.00, p < .001, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.031$), but the JOLs did not differ across the retention intervals (F(1, 58) = 0.001; p = 0.97, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2<0$). They do not reflect the interactive pattern evident in Figure 3 (F(1, 58) = 1.91; p = 0.17, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.001$), nor should they, since the conditions do not support that interaction in actual retention.

Figure 3.

Number of items predicted to be recalled (out of 16 possible) as a function of study-test cycle and study activity. Top panel shows judgments provided with no external support (Experiment 2); middle panel shows judgments with some external support (Experiment 3); bottom panel shows judgments provided with extensive external support (Experiment 4). Error bars represent the standard error of the difference scores within each cycle.

Discussion

Under conditions that allowed for diagnostic retrieval from long term memory immediately prior to judgments, JOLs accurately reflected the testing effect across retention intervals. Subjects in the cue-only condition accurately accorded re-studied items higher JOLs immediately after practice, but accorded tested items higher JOLs at a delay. Thus, subjects showed sensitivity to the mnemonic strength imparted by testing and the interaction of that effect with retention interval, thereby implicitly recognizing the differential rates of forgetting following study and test trials.

Though judgments mirrored the effects of testing when they were elicited in a cue-only context, they did not when elicited in the presence of both the target and the cue. In a cue-target context, subjects cannot (or do not) infer the effectiveness of learning from the outcome of a concurrent retrieval attempt; rather, they are limited to less diagnostic cues, such as their memories for the outcomes of prior tests (Finn & Metcalfe, 2008). To assess whether subjects relied upon the outcome of the prior Phase 2 test to determine their Phase 3 JOLs, we computed gamma correlations between Phase 2 test performance and Phase 3 JOLs within each subject. This correlation was high in both the cue-only (G = 0.84) and the cue-target (G = 0.86) conditions, indicating either a reliance on the outcome of the prior test for the later JOL, or, more simply, an item effect whereby well learned items are more likely to have been recalled during Phase 2 and are accorded higher JOLs in Phase 3. The interesting results concern the relationship between Phase 3 JOLs and final test performance, which was much higher for the cue-only (G = 0.93) than the cue-target (G = 0.54) condition (t(110) = 5.13). This result suggests that the high correlations between Phase 2 performance and Phase 3 JOLs may have a different basis for the two prediction contexts. Under cue-only conditions, subjects rely on the outcome of the Phase 3 retrieval attempt to make JOLs, which is highly diagnostic of later performance. Under cue-target conditions, they rely upon the outcome of a prior test (one made during Phase 2) to make JOLs, which is not as accurate of a basis.

Relying upon outcomes of prior tests to make cue-target JOLs, however, results in better metacognitive resolution than when no prior tests have been attempted. Learners exhibit very accurate metacognitive resolution in cue-only conditions because they base JOLs on the outcomes of concurrent diagnostic retrieval attempts (tests of each item; Dunlosky & Nelson, 1994). This experiment shows that an added benefit of testing is an improvement in resolution of subsequent metacognitive judgments made in cue-target contexts. The metacognitive resolution of cue-target JOLs in predicting final memory performance, as measured by signal-detection theoretic value d_a (Benjamin & Diaz, 2008), was significantly higher for previously tested than previously re-studied items both at no delay (1.10 vs. 0.64; t(29) = 2.31) and delay (1.08 vs. 0.68; t(29) = 2.08). This extends the memory for past test heuristic literature (Finn & Metcalfe, 2008) by indicating that reliance upon the outcomes of prior tests when making JOLs is a wise strategy: metacognitive resolution is improved in cue-target contexts if learners can base metacognitive judgments on their memory for prior test outcomes. Further, this reveals that students’ reliance upon tests to assess what they know and do not know (Kornell & Son, 2009) is a reasonable study strategy. Basing judgments on the results of prior tests allows learners to improve their future metacognitive accuracy, which may benefit learners’ study choices.

Previously tested word pairs elicited higher JOLs than previously studied words in the cue-target condition even though final recall did not differ, which, though not central to our claims here, differs from extant results in the literature. The cue-target condition utilized within this experiment, however, significantly differs from those in the present literature. While the prior testing effect literature has usually relied upon aggregate JOLs as indicators of metacognitive monitoring, we solicited item-by-item JOLs for both studied and tested items within each subject. Since the judgments are elicited within each subject on an item-by-item basis, the cue-target condition re-exposes learners to all the word pairs following the re-study/test manipulation. Learners who were not able to retrieve items during the Phase 2 test are provided an additional opportunity to learn the correct target for the word pair during the Phase 3 cue-target JOLs. The additional exposure to the word pairs likely alters both predictions and performance across conditions. Further, item-by-item monitoring may allow subjects to base metacognitive judgments upon subjective cues, like cue familiarity, rather than naïve analytic theories about the influence of testing. The reliance on subjective rather than analytic cues may cause subjects to provide higher predictions for previously tested items than studied items, because cue familiarity, an influential subjective cue (Metcalfe, Schwartz, & Joaquim, 1993), for tested items may be greater than that for restudied items. This is likely because subjects spent significantly more time responding to each cue during a test trial during Phase 2 than they spent viewing an item during a re-presentation trial (6.2 sec vs. 4 sec; t(119) = 3.95). The extra time spent processing the cue may have augmented cue familiarity, resulting in higher metacognitive predictions without affecting future memory performance (Metcalfe et al., 1993).

This result adds to and qualifies the existing research on metacognition of the testing effect. Extant research shows that analytic, theory-based judgments about the effects of testing are at best incomplete and likely inaccurate, and these results show that mnemonic-based judgments of the testing effect are very accurate. Under favorable circumstances, metacognitive judgments accurately reflect the mnemonic costs and benefits of testing, as seen in the first experiment. Learners can accurately judge the mnemonic strength imparted by testing. However, it remains unclear whether learners attribute mnemonic differences in performance to prior re-study/test conditions and whether they will generalize this knowledge to future testing predictions. In the next three experiments, we assess learners’ global metacognitive judgments about tested and re-studied items to determine if, and under what circumstances, learners attribute improved memory performance to prior testing. It is of considerable practical interest to know whether subjects can use the accurate mnemonic-based judgments to credit testing with improved memory performance and consequently generalize this knowledge to predict mnemonic benefits of testing during future study episodes. Further, the circumstances under which learners translate accurate item-by-item judgments into accurate aggregate judgments remain unknown. JOLs can reflect a variety of different cues and not necessarily reveal whether learners attribute mnemonic performance to differing study activities (Koriat, 1997). Aggregate judgments may more closely reflect knowledge about study activities than item-by-item JOLs.

In these experiments, we examine changes in learners’ metacognitive beliefs about the effectiveness of testing by measuring learners’ global predictions of the mnemonic consequences of testing and re-studying across a pair of study/test cycles. Even though learners have significant misconceptions about the mnemonic consequences of many aspects of learning, they often modify those beliefs to correctly reflect the differential effectiveness of strategies used to encode items (Brigham & Pressley, 1988), the differential effectiveness of cues used to retrieve items (Bieman-Copland & Charness, 1994), and how item characteristics affect memorability (Benjamin, 2003; Tullis & Benjamin, 2011) through task experience.

Experiment 2

In Experiments 2, 3 and 4, we collected global predictions about performance on re-studied and on tested items to assess general, theory-based beliefs about the differential mnemonic effectiveness of re-studying compared to testing. We collected these global predictions across two study/test cycles and measured changes in predictions after direct experience with the task introduced in Experiment 1. If learners successfully update their predictions concerning the effectiveness of these practice activities, they should make higher predictions for tested than re-studied items during the second cycle. In the following three experiments, learners predicted how many re-studied and tested items they will remember the following day. On the second day, learners were tested on all previously practiced items, practiced a second list of items, and made predictions about how many re-studied and tested items from the second list they would remember the following day. Our focus is on how the global predictions about the effectiveness of re-study and testing change after experience with the task. Performance on List 2 is not measured because we are interested solely in the circumstances that prompt learners to predict benefits of testing.

We vary the metacognitive burden placed on learners across Experiments 2, 3, and 4. In Experiment 2, a high metacognitive burden is placed on learners: in order to use knowledge from the first list to change predictions on the second, learners must remember the study condition for each item at a long delay and must track their memory performance between the study conditions. In Experiment 3, we reduce the metacognitive burden learners face by providing partial external support. Learners are told about the study conditions for each item during the List 1 test. Finally, in Experiment 4, we reduce the metacognitive burden even more by additionally providing an aggregate summary of performance comparing tested and re-studied items. The high metacognitive burden in Experiment 2 may prevent learners from effectively updating their knowledge to predict benefits of testing, while the reduced metacognitive burden in Experiments 3 and 4 may facilitate this change.

Method

Subjects

Thirty-five introductory-level psychology students from the University of Illinois at Urbana-Champaign participated in exchange for partial course credit.

Materials

The same word pair pool was used as in Experiment 1. The word pair pool was randomly split into two equal halves, such that one half (32 word pairs) was studied and practiced on day 1 and the other half was studied and practiced on day 2.

Design

The experiment used a 2 × 2 within-subjects design. The independent variables were practice condition (Phase 2 = re-study vs. test), and list number (1 vs. 2). Global predictions of memory performance for re-studied items and for tested items were collected following practice on each list.

Procedure

The procedure was similar to that used in Experiment 1, but included an additional cycle of study, practice, and predictions for a new word list on day 2. Additionally, Experiment 2 collected global metamnemonic predictions instead of item-by-item JOLs in order to assess general beliefs about the effectiveness of each practice technique.

The initial study and practice phases in Experiment 2 were identical to phases 1 and 2 in Experiment 1, but the prediction phase differed. Following the practice phase on day 1, all subjects made global predictions about how many re-studied items and how many tested items they would remember the next day. Subjects were asked, “From the 16 word pairs that you RE-STUDIED, how many do you think you will remember tomorrow?” and “From the 16 word pairs that you were TESTED on, how many do you think you will remember tomorrow?” The order of the questions alternated between consecutive subjects. Subjects returned a day later to complete the cued recall test.

Once they finished the cued recall task for the first list, subjects completed another cycle of presentation and practice phases with a new list of 32 word pairs. Finally, as they did after practicing the first list, subjects made global predictions about how many re-studied and how many tested items they thought they would remember from the second list on the following day. No second test was administered.

Results

Memory performance

The mean proportion of items recalled during the practice test phase of the first list was 0.36 (SD = 0.23). On the final test for the first list of items, subjects recalled a greater proportion of tested items (M = 0.24, SD = 0.17) than re-studied items (M = 0.19, SD = 0.15), t(34) = 2.31, p = .027, d = 0.31. Of the 35 subjects, 19 showed a testing effect, 8 showed no differences between re-studied and tested items, and 8 showed an advantage for re-studied over tested items. The mean proportion of items recalled during the practice test phase of the second list was 0.41 (SD = 0.24).

Global predictions

The mean global predictions are shown in Figure 3 (top panel). A 2 (prediction cycle: first or second) × 2 (study activity: re-study or test) ANOVA was conducted on the global predictions and no significant main effects were found. Further, a significant interaction was not found between prediction cycle (first or second list) and study activity (re-study or test), F(1,34) = 0.44), p = 0.51, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2<0$. When we restrict the data to only those subjects who show a testing effect, the same pattern of results holds: a significant interaction between prediction cycle and study activity was not found, F(1,18) = 0.00, p = 0.99, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=0$.

Discussion

Subjects did not update their knowledge to reflect the testing effect through direct experience with the task. This finding is in contrast to prior studies that examined updating of the effects of item characteristics (Benjamin, 2003; Tullis & Benjamin, 2011) or encoding regimens (Dunlosky & Hertzog, 2004). However, there are results revealing failures to update with experience (Diaz & Benjamin, 2011), and there are parallels to be drawn between that study and this one. In both tasks a heavy burden was placed on subjects to accurately remember, at test, the origin of individual items—in this case, which items had been previously studied and which had been tested. The long delay between practice and test in the current experiment makes that task even more difficult. In addition to remembering which practice method was used for each item, learners must accurately tally the number of items correctly remembered for each type of practice utilized. By this explanation, the major bottleneck to subjects learning about the testing effect comes from the difficulty of tracking the relationship between the conditions of learning and test performance.

Alternatively, subjects may have based their global predictions on the same considerations as their item-by-item JOLs. That is, they may not seek to assess the relationship between study conditions and test because they already have a theory about the effects of testing (an incorrect one). If so, subjects predict greater recall for re-studied items than tested items for the same reasons that re-studying leads to higher immediate cue-only JOLs than testing typically does.

In Experiment 3, we eliminate some of the extraneous cognitive load in order to evaluate these competing explanations. We reduce the cognitive demands of the knowledge updating task by introducing partial feedback for subjects; we tell subjects which type of study activity was used for each item and whether they were correct on the final test of the first list. If learners predict benefits of testing during the second list in Experiment 3, it will indicate that the difficulty of tracking the relationship between prior study activities and eventual test performance is likely to be the cause of the failed knowledge updating in Experiment 2, and not mere persistence with an incorrect and unmalleable mental model about the effects of testing on memory.

Experiment 3

In Experiment 3, we introduced partial feedback to subjects by providing them with information about their final answers’ correctness and about which study activity was used for each item.

Method

Subjects

Fifty-three introductory-level psychology students from the University of Illinois at Urbana-Champaign participated in exchange for partial course credit.

Materials and Design

The materials and design were identical to those used in Experiment 2.

Procedure

The procedure was very similar to that used in Experiment 2, but the test included more information about each subject’s performance and the study activity used during the practice phase. Feedback given during the cued recall test on the second day included whether the subject’s response was correct or incorrect and whether a given item had been re-studied or tested during the practice phase. Feedback remained on the screen for 3 seconds before it was removed and the next item was tested.

Results

Memory performance

The mean proportion of items recalled during the initial practice test phase was 0.42 (SD = 0.24). Final cued recall performance was higher for previously tested items (M = 0.21, SD = 0.18) than previously re-studied items (M = 0.15, SD = 0.17), t(52) = 3.14, p = .003, d = 0.34. Of the 53 subjects, 35 showed a testing effect, 7 showed no differences between re-studying and testing, and 11 showed a mnemonic advantage of re-study over test. The mean proportion of items recalled during the practice phase of the second list was 0.46 (SD = 0.27).

Global Predictions

Subjects’ predictions of performance are displayed in Figure 3 (middle panel). A 2 (prediction phase: first or second) × 2 (study activity: re-study or test) ANOVA on global predictions shows a marginal interaction between prediction phase and study activity (F(1, 52) = 3.23, p = 0.08, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.002$) and no main effects. Subjects rated re-studying as numerically more effective than testing during the first prediction cycle (t(52) = 1.12, p = 0.27, d = 0.09), but rated testing as numerically more effective than re-studying during the second (t(52) = 1.47, p = 0.15, d = 0.12).

An analysis of the subset of 35 subjects who showed a testing effect reveals the same pattern of data. The 2 (prediction phase) × 2 (study activity) ANOVA on predictions failed to show a significant interaction between phase and study activity (F(1,34) = 1.68, p = 0.20,, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2<0$). Subjects rated restudying as numerically better during the first list and testing better during the second, but neither of these post-hoc tests reached significance (t(34) = 0.55, p = 0.59, d = 0.06; t(34) = 1.56, p = 0.13, d = 0.07, respectively).

Discussion

Subjects may have slightly updated their knowledge to reflect the testing effect through direct experience with the task and experimenter-provided feedback. In contrast to Experiment 2, the ordering of predictions favors re-studying in the first list, but favors testing in the second. The size of knowledge updating may be very small and variable across these lists, so the interaction between cycle and study activity reached only marginal significance. We are still placing a large cognitive burden on learners, as they had to tally the number of correctly answered re-studied and tested items. Providing information about whether each item was re-studied or tested may provide more impetus to change metacognitive ratings than feedback about a response’s correctness. Other studies have shown that providing information at test about prior study condition by blocking similar items helps update metacognitive knowledge (Price, Hertzog, & Dunlosky, 2008), presumably because learners can more easily count how many items from each condition were recalled. Tallying the correctly answered items from each condition may be the last impediment to knowledge updating, as learners may lack the cognitive resources to do so on their own. In Experiment 4, we eliminate this extraneous cognitive load by providing all of the feedback of Experiment 3 and a tally of the number re-studied and tested items they successfully recalled. If learners predict benefits of testing during the second list in Experiment 4, it will indicate that the difficulty of tallying the number of items correctly recalled prevented knowledge updating about the effectiveness of self-testing.

Experiment 4

In Experiment 4, we provided even more external support by tallying the number of correct re-studied and tested items during first list before subjects start the second.

Method

Subjects

Twenty-eight introductory-level psychology students from the University of Illinois at Urbana-Champaign participated in exchange for partial course credit.

Materials and Design

The materials and design were identical to those used in Experiment 2 and 3.

Procedure

The procedure was very similar to that used in Experiment 3, but the test included more information about each subject’s performance and the study activity used during the practice phase. Once all items were tested, subjects were given global feedback about their performance that indicated how many restudied items out of 16 they correctly answered and how many tested items out of 16 they correctly answered. The order of this global differentiated feedback was randomized and remained on the screen until the subject indicated they were ready to proceed.

Results

Memory performance

The mean proportion of items recalled during the initial practice test phase was 0.36 (SD = 0.24). Final cued recall performance was higher for previously tested items (M = 0.27, SD = 0.21) than previously re-studied items (M = 0.17, SD = 0.16), t(27) = 3.45, p = .002, d = 0.54. Of the 25 subjects, 19 showed a testing effect, 6 showed no differences between re-study and test, and 3 showed an advantage of re-study over test. The mean proportion of items recalled during the practice phase of the second list was 0.55 (SD = 0.23).

Global Predictions

Subjects’ predictions of performance are displayed in Figure 3 (bottom panel). A 2 (prediction phase: first or second) × 2 (study activity: re-study or test) ANOVA on global predictions shows a significant interaction between prediction phase and study activity (F(1, 27) = 11.01, p = .003, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.017$) and no main effects. Subjects rated re-studying as more effective than testing during the first prediction cycle (t(27) = 3.48, p = .071, d = 0.27), but rated testing as marginally more effective than re-studying during the second (t(27) = 1.67, p = .099, d = 0.25). An analysis of the subset of 19 subjects who showed a testing effect revealed the same interaction between prediction phase and study cycle (F(1, 17) = 9.57, p = .006, ${\mathrm{\omega ̂}}_{\mathit{\text{partial}}}^2=.024$) as the larger group of subjects. Additionally, post-hoc tests reveal that this subset of subjects initially predicted that re-studying will lead to better performance than testing (t(18) = 2.28, p = .035, d = 0.24), but during the second cycle, reversed their ratings to predict that testing will produce better memory performance than re-studying (t(18) = 2.43, p = .026, d = 0.41).

Discussion

Subjects significantly updated their beliefs about the advantages of testing over re-studying when facilitated by external support. Initially, subjects predicted that re-studying leads to superior memory than testing; through task experience and with feedback, subjects rated testing as greater for promoting long-term memory. This shift suggests that the failure to accurately update knowledge in Experiments 2 and 3 was due to a failure to accurately track the effects of testing and re-studying on items tested after a 24-hour delay and tally the results accordingly.

Results across Experiments 2 through 4

We can combine the results across Experiments 2, 3, and 4 to see how varying the amount of external support influences learners’ predictions across cycles. To do so, we classified learners’ predictions on List 1 and 2 as to whether they predicted an advantage for testing, equivalent performance, or an advantage for restudying. We determined whether this classification matched with their actual mnemonic performance on List 1 across the three feedback conditions. As shown in Figure 4, the proportion of learners whose predictions matched their List 1 performance only increased across the cycles in the full feedback condition. Learners’ predictions seem to be greatly influenced by the tally of their performance and less influenced by the partial feedback given in Experiment 3.

Figure 4.

The proportion of learners whose predictions on Lists 1 and 2 match their mnemonic performance on List 1 by feedback condition (Experiment 2 = none; Experiment 3 = partial; Experiment 4 = full).

Next, we used learners’ metacognitive classifications (predicting a testing effect, equivalent performance, or a restudy advantage) across List 1 and 2 to categorize learners’ predictions as shifting towards or away from the testing effect. For example, if learners predicted equivalent performance on the first cycle, but predicted a testing effect on their second cycle, their predictions were classified as shifting towards the testing effect. Conversely, if learners predicted equivalent performance on the first cycle, but a restudy study advantage on the second, their predictions were classified as shifting away from the testing effect. The proportions of learners shifting towards and away from the testing effect are displayed in Figure 5. With increasing external support, the proportion of learners shifting towards the testing effect increases, while the proportion of learners shifting away from the testing effect drops.

Figure 5.

The proportion of learners whose predictions shifted towards or away from valuing testing more than restudying across the two list cycles by feedback condition (Experiment 2 = none; Experiment 3 = partial; Experiment 4 = full).

The capability of learners to recognize the benefits of differing study activities may depend upon the magnitude of those benefits. Across Experiments 2, 3, and 4, learners showed a relatively small testing effect, such that tested items were remembered significantly, but only slightly, better than re-studied items. Differences in final mnemonic performance between prior tested and re-studied items may have been small because successful retrieval on the initial test was somewhat lower than in other studies (approx. 40%). Low levels of initial successful retrieval reduces the overall advantage for the class of tested items. Increasing successful retrieval during the initial test by increasing the amount of prior learning may increase the size of the testing effect and change what benefits of testing learners recognize. Learners may struggle to detect the advantages of testing because the mnemonic advantage of testing is small throughout these experiments. Consequently, learners need supportive conditions to value testing more than re-studying. If a larger testing effect existed, learners may show knowledge updating under less supportive conditions.

General Discussion

The results here demonstrate that when subjects make judgments under conditions that promote diagnostic self-testing and provide differentiated feedback, their JOLs show sensitivity to the delay-contingent memory effects of testing versus re-studying. Experiment 1 shows that the accuracy of metacognitive judgments for tested items is very similar to metacognitive accuracy for other beneficial study habits, like imaging items and spacing study between repetitions. The accuracy of metacognitive judgments in all cases depends greatly upon the conditions under which the judgments are solicited.

The ability to accurately appreciate the benefits provided by study activities carries vital consequences for learners. Accurate metacognition drives effective study behavior and can improve cognition without significant increases in work or effort, while inaccurate or biased metacognition may lead learners to adopt suboptimal study behaviors and lead to inferior levels of learning. Learners’ metacognitions are very accurate under circumstances in which mnemonic assessment is feasible, contrastive, and diagnostic prior to metacognitive judgments.

However, taking full advantage of such conditions is difficult, in part because the benefits of testing do not arise until long after study choices have been made. In order to benefit from control over learning, learners must recognize the advantages of testing when making choices about study activities. Experiments 2, 3, and 4 investigated the conditions under which such knowledge updating can occur. In Experiment 2, a lack of change in global memory predictions about the effectiveness of testing compared to re-studying across study lists revealed that learners do not update beliefs about the effects of testing when they bear the full burden of tracking the relationship between items and their learning conditions. However, as the experiments provided more and more external support to track these relationships, global memory predictions shifted to predict beneficial effects of retrieval. With adequate support, learners can learn through experience to predict the long-term advantages that testing provides. Other studies also suggest that students recognize the benefits that testing provides when they are provided with extensive external support about study conditions during testing (Einstein, Mullet, & Harrison, 2012). Further, after students recognize the benefits of testing, they report that they incorporate more testing into their self-controlled study activities (Einstein et al, 2012).

The results described in these experiments might help provide a bridge between the many results indicating failures of metacognitive monitoring with respect to the testing effect and suggestive evidence that metacognitive control may reflect the advantages of testing. Learners who experience the mnemonic benefits of generating words (a cognitive activity similar but not identical to testing; Karpicke & Zaromb, 2010) enhance their future processing of read items to render those items as well remembered as generated items (deWinstanley & Bjork, 2004). In deWinstanley and Bjork’s study, learners were presented with short textbook passages which contained both to-be-generated and to-be-read critical items. Learners’ memory for critical items was measured by fill-in-the-blank tests across two study/test cycles. Learners’ memory performance showed a generation advantage during the first cycle, but lacked a generation advantage during the second. The absence of the generation advantage during the second cycle occurred because memory performance for the to-be-read items improved to the level of the to-be-generated items. The authors argued that subjects recognized the mnemonic advantage of generation during the first test and spontaneously developed processing strategies for to-be-read material during the second cycle that rendered to-be-read material as well remembered as to-be-generated material. DeWinstanley and Bjork’s (2004) results suggest that when learners recognize the benefits that testing provides, they may spontaneously engage in better encoding and processing strategies for re-studied items during subsequent lists in order to boost memory performance for those items. Subsequent processing and performance changes may be apparent on re-studied items, where learners could ignore (or actively obscure) the presence of the target in order to test themselves and improve their memory performance. However, we cannot evaluate this claim in our data since memory performance on the second list was not measured.

This set of experiments provides insight into what is required for learners to update their metacognitive strategy knowledge. First, learners’ monitoring must be accurate enough to notice a consistent difference in acquisition or performance between learning conditions. Larger differences in performance may draw more attention to the differences in conditions and allow learners to notice the differences. Next, learners must attribute performance differences to the study conditions and must not disregard differences as arising from idiosyncratic item characteristics. Attributing differences in performance to study conditions requires tracking the study conditions of individual items and tallying the performance of those items across each condition. The ability to track and tally performance, while still recalling items during the test, may require significant working memory resources. The support that the environment provides may modulate the amount of working memory resources needed to track and tally the items accurately. Finally, to modify existing beliefs about strategies, learners must acknowledge the discrepancy between their prior beliefs and actual results.

Interventions designed to educate learners about the effect of testing need to consider the tracking and tallying burden that hampers learning about testing effects and provide support in such a manner so as to allay that burden. The extent to which conditions support such tracking and tallying will determine whether attempts to enlighten metacognition will founder or succeed. The extensive feedback detailing performance on tested compared to re-studied items does not naturally occur in the course of classroom learning because teachers cannot segregate performance as a function of the study activities chosen by the learners during self-regulated learning. The need to fully instruct and support learners concerning the effectiveness of metacognitive strategies mirrors current conceptions about fully guiding learners about content knowledge (Klahr & Nigam, 2004). Many argue that learners do not efficiently or effectively learn content knowledge when required to discover or construct it on their own (Sweller, Ayres, & Kalyuga, 2011). The current study shows that learners may not effectively learn about the differential effectiveness of metacognitive strategies unless provided with full, explicit guidance. This combination of circumstances suggests that automated tutors (see Finley et al., 2009) may be the best source of hope for providing the detailed feedback necessary to translate the appreciation of testing evident in Experiment 1 into the successful theory-based predictions evident in Experiment 4.