Immediate Judgments of Learning are Insensitive to Implicit Interference Effects at Retrieval

Abstract

We conducted three experiments to determine whether metamemory predictions at encoding, immediate judgments of learning (IJOLs) are sensitive to implicit interference effects that will occur at retrieval. Implicit interference was manipulated by varying the association set size of the cue (Exps. 1 & 2) or the target (Exp. 3). The typical finding is that memory is worse for large-set-size cues and targets, but only when the target is studied alone and later prompted with a related cue (extralist). When the pairs are studied together (intralist), recall is the same regardless of set size; set-size effects are eliminated. Metamemory predictions at retrieval, such as delayed JOLs (DJOLs) and feeling of knowing (FOK) judgments accurately reflect implicit interference effects (e.g., Eakin & Hertzog, 2006). In Experiment 1, we contrasted cue-set-size effects on IJOLs, DJOLs, and FOKs. After wrangling with an interesting methodological conundrum related to set size effects (Exp. 2), we found that whereas DJOLs and FOKs accurately predicted set size effects on retrieval, a comparison between IJOLs and no-cue IJOLs demonstrated that immediate judgments did not vary with set size. In Experiment 3, we confirmed this finding by manipulating target set size. Again, IJOLs did not vary with set size whereas DJOLs and FOKs did. The findings provide further evidence for the inferential view regarding the source of metamemory predictions, as well as indicate that inferences are based on different sources depending on when in the memory process predictions are made.

Theories of metacognitive monitoring emphasize that the accuracy of judgments about future memory depends on the accessibility and the diagnosticity of the cues that are accessed for future memory experiences (e.g., Dunlosky & Matvey, 2001; Dunlosky & Metcalfe, 2009; Koriat, 1993, 1997; Koriat & Bjork, 2006). Different kinds of cues are likely to be accessed at different stages of the process of learning and remembering (e.g., Finn & Metcalfe, 2008). Nelson and Narens (1990) proposed a framework of metacognitive monitoring and control involving three basic phases of learning and remembering: acquisition (or encoding), retention, and retrieval. Different kinds of metacognitive monitoring, made during each of these three stages, can inform control processes during those stages, such as selection of encoding strategies during acquisition or termination of search during retrieval.

The present study evaluates the sensitivity of metacognitive judgments to implicit retrieval interference effects. Implicit interference was manipulated by varying the number of words associated with either the cue or target, or associative set size (Nelson, McEvoy, & Schreiber, 1990). The purpose of the experiments was to examine whether metamemory predictions made at different stages in the memory process are influenced by (are sensitive to) this kind of implicit interference. Three types of metamemory predictions were examined: immediate judgments of learning (IJOLs) made during encoding, delayed JOLs (DJOLs) made after encoding an item during the retention interval for that item (sometimes also called predictions of knowing, or POKs; Schreiber & Nelson, 1998), and feelings of knowing (FOKs) made at test after attempted cued recall. The rationale of the study was that implicit interference will not impact judgments at all stages of the memory process. Specifically, because implicit interference effects occur at retrieval, we expected IJOLs to be insensitive to the effects of implicit interference, whereas DJOLs and FOKs should access retrieval outcomes that are influenced by implicit interference. In the following sections, we outline the basis of our theoretical argument after explicitly defining and describing the metacognitive judgments used in our study.

Metacognitive Judgments

Metacognitive judgments are often made for paired-associate items (e.g., BIRD-WINGS), in part because of their affordance for using the same cues for judgments and testing associative memory (e.g., cueing judgments and recall by presenting BIRD). IJOLs are made immediately after encoding of each item. For instance, immediately after studying BIRD-WINGS, the cue BIRD is presented and people report their subjective confidence that they will correctly recall WINGS when cued to do so on the later test. The response format used in the present study was a probability scale ranging from 0 (certain not to recall) to 100 (certain to recall) the target.

DJOLs are predictions that are made during the retention interval between encoding and retrieval. DJOLs can be made with a very short delay (e.g., Nelson & Dunlosky, 1991; Weaver & Kelemen, 1997) or, as in this study, during a separate DJOL collection phase after all items have been studied. Otherwise, DJOLs have a similar format and response scale as IJOLs.

FOKs are predictions made after a cued-retrieval attempt, either immediately after the recall attempt or separately in a later block (Hart, 1965; MacLaverty & Hertzog, 2009; Nelson & Narens, 1990; Schacter, 1983). FOKs assess confidence about future recognition of the target that was paired with the studied cue. In the present study, after recall, item cues were presented in random order and FOKs were made using the same probability scale as JOLs and DJOLs.

Metamemory predictions can be evaluated in a number of ways (Dunlosky & Metcalfe, 2009). In Experiment 1 we examined: (a) sensitivity of JOLs, DJOLs, and FOKs magnitudes (as measured by mean judgments), and (b) resolution, or relative accuracy, as measured by within-person ordinal Goodman-Kruskal gamma correlations between judgments and memory outcomes¹. In Experiments 2 and 3, we focused primarily on sensitivity of the metacognitive judgments to cued recall.

Impact of Implicit Interference

The question posed by these experiments was whether the three types of metamemory judgments are impacted in the same way by implicit interference. In the present study, implicit interference refers to the number of associates of a given cue or target, or associative set (Nelson et al., 1990). Words vary in terms of the number of words that are associated to them; some words have a relatively small associative set (operationalized as from one to nine associates) whereas some have a relatively large set (operationalized as from 15 to 30 associates). Implicit interference arises from competition among associates of either the cue or the target of a paired-associate item (e.g., BIRD-WINGS) during retrieval. Words that have a large set size evoke more implicit interference than those with a small set size. When the target (e.g., WINGS) is studied alone and then cued with an associate (called extralist cueing), recall is lower given a large-set-size cue (e.g., BIRD) than it would be given a small-set-size cue. That is, implicit interference is greater for large- as compared to small-set-size cues or targets (Eakin & Hertzog, 2006; Nelson & McEvoy, 1979; Nelson et al., 1990). According to the Processing Implicit and Explicit Representations model (PIER2; Nelson, McKinney, Gee, & Janczura, 1998), encoding a word generates both (a) an explicit episodic representation of the word and (b) implicit activation of related word concepts. Retrieval success during extralist cueing depends on sampling associated candidates which compete for retrieval; large-set-size cues and targets result in more interference relative to small-set-size cues and targets, resulting in lower probability of recall or set-size effects.

Both cue- and target-set-size effects are eliminated under intralist cueing, in which intact paired associate items are studied (e.g., BIRD-WINGS) and recall of the second, target word is cued by the first word (Eakin & Hertzog, 2006; Nelson & McEvoy, 1979; Nelson et al., 1990). According to PIER2 (Nelson et al., 1998), both the cue and target provide meaning context and only the intersecting set of associates of both words are activated when they are studied together. The result is an effectively equal sampling set for small- and large-set size cues and targets and, therefore, equivalent recall.

Prior research has demonstrated that metacognitive judgments can be affected by implicit interference effects. Schreiber (1998) demonstrated that both DJOLs and FOKs were impacted under extralist cueing by target set size (higher mean judgments for lower set-size items), mirroring implicit interference effects on recall. These effects were eliminated under intralist cueing. Likewise, Schreiber and Nelson (1998) found that both DJOLs and FOKs tracked cue-set-size effects in recall. Eakin and Hertzog (2006) replicated cue-set-size effects on DJOLs and Eakin and Hertzog (2010) replicated the effects on FOKs. In both studies, people’s judgments accurately reflected the impact of implicit interference on recall. Taken together, these studies demonstrate that DJOLs and FOKs are influenced in the same way by implicit interference.

Apparently metamemory predictions made just prior to (i.e., DJOLs) or during (i.e., FOKs) retrieval are sensitive to the implicit interference from competing associates. The question we posed is whether this influence of implicit interference is theory-based prognostication (Koriat & Bjork, 2006) or a direct consequence of retrieval interference for accessibility to information about the target. We argue that this sensitivity to interference occurs because of the retrieval demands that precede both types of judgments.

Experiencing covert retrieval has been cited as an explanation for the substantially greater resolution for DJOLs than for IJOLs (Dunlosky & Nelson, 1994). DJOLs have superior accuracy when only the cue (rather than the cue-target pair) elicits the DJOL (Dunlosky & Nelson, 1994; Nelson & Dunlosky, 1991; Weaver & Kelemen, 1997). The presentation of cue without target invites a retrieval attempt from memory. Delay alone is not the explanation, because presenting intact cue-target items with a delay does not enhance IJOL accuracy. In a similar vein, MacLaverty and Hertzog (2009) found that delaying the FOK after the initial cued-recall attempt did not affect FOK resolution, possibly because in both cases retrieval attempts are required to query the contents of memory.

We hypothesized that one could find a similar dissociation between IJOL and DJOL/FOK judgments under implicit interference. Individuals are not aware of implicit interference effects; they are not able to accurately identify words with small versus large associative sets (Schreiber, 1998). Individuals cannot directly observe the set-size of associates of a given cue or target; nor are they likely to have an implicit theory regarding the relationship of set size and interference. Koriat & Bjork (2005) demonstrated that IJOLs were influenced by whether cue and target were associatively related, but were also insensitive to the direction of association (forward versus backward). Arguably, the direction of association is more readily deduced from the stimuli than is the associative set size of a particular stimulus.

Because a word’s associative set size is neither directly manifest nor easily deduced, we expected that IJOLs would be insensitive to set size effects. Instead, the hypothesis was that metacognitive judgments would only be sensitive to implicit interference if the judgment is based on accessibility when target retrieval is attempted upon presentation of the cue. In other words, implicit interference influences the information that is accessible at the time of retrieval, so that metacognitive judgments that are influenced by retrieval access should, at least potentially, be sensitive to implicit interference. Conversely, IJOLs, which do not benefit from retrieval from secondary memory, should be insensitive to future implicit interference effects. Although the qualia of a large- versus small-set-size cue or target is available when IJOLs are made, without a retrieval attempt, the effect of many versus few associates on retrieval may not be anticipated. In contrast to IJOLs, DJOLs are believed to be influenced by outcomes of attempted target retrievals (Dunlosky & Nelson, 1994; Nelson, Narens, & Dunlosky, 2004). If accessibility is the common factor for DJOLs and FOKs, then – because of the experience garnered via a retrieval attempt – both these judgments should manifest implicit interference effects.

Experiment 1 tested this hypothesis by having three groups of people study related paired associates for which the cue set size varied (small versus large) under either extralist or intralist cueing procedures. We predicted that cue-set-size effects would be obtained in memory under extralist cueing and eliminated under intralist cueing for all three judgment conditions. We anticipated that DJOLs and FOKs would be sensitive to cue-set-size effects, but that IJOLs would not. Experiments 2 and 3 were conducted to contend with some surprising findings due to the methodology of measuring JOLs, as will be discussed below.

General Methods

The procedure for three experiments reported is identical; what changed were the materials used. The general procedure is described here and unique details are provided when describing each experiment.

General Procedure

The tasks were programmed using E-prime version 1.1 (Psychology Software Tools, Inc.) and were executed on standard PCs. Figure 1 depicts the general procedure for the three experiments. All experiments manipulated prediction type (IJOL, DJOL, or FOK) and cue type (intralist or extralist cueing) as crossed between-subjects factors. In all conditions, participants were presented with an item to be memorized for eight seconds, and were instructed to encode the item using visual imagery. They pressed the space bar to indicate when the image was formed and then, after the full eight seconds had passed, rated their images. The response options included vivid (clear with lots of detail), neutral (unclear and vague), or unable to form an image. A total of 44 items were presented in random order. For the intralist cueing procedure, the items were cue-target paired associates (e.g., BROOK – RIVER). For extralist cueing, participants viewed only the target from each paired associate during the encoding phase (e.g., RIVER). An additional six items served as practice before each of the experimental phases.

Figure 1.

The procedure for IJOLs, DJOLs, and FOKs. The bottom figure compares the procedure for the cued and no-cue IJOL extralist conditions used in Experiments 2 and 3.

Procedures for each of the metamemory prediction conditions. Cueing Procedure and Cue Set Size were manipulated within the IJOL, DJOL, and FOK conditions. The bottom figure compares the cued and no-cue IJOL extralist conditions used in Experiments 2 and 3.

IJOL procedure

The IJOL procedure consisted of two phases: a) the encoding phase and b) the recall phase. After the eight-second encoding time, the item was removed and the cue was presented to prompt the IJOL judgment, using the scale described previously, Participants were encouraged to use the full range of the 0–100% confidence scale. After participants rated the image they formed during encoding, the next item was presented.

During the recall phase, each of the cues was presented in a random order, and participants attempted recall of the associated target. Time to recall was unlimited. Participants were encouraged to try hard to recall, and were permitted to guess if they were unable to do so. If unable to recall or guess, participants typed “NEXT” on the keyboard.

DJOL procedure

The DJOL procedure consisted of three phases: a) the encoding phase, b) the DJOL phase, and c) the recall phase. During the encoding phase, all items were studied before the separate judgment phase. During this phase, each cue was presented and DJOLs were elicited, using the using the 0–100 confidence scale. The recall phase was identical to that of the IJOL procedure.

FOK procedure

The FOK procedure consisted of four phases: a) the encoding phase, b) the recall phase, c) the FOK phase, and d) the recognition phase. The encoding phase and recall phase procedures were identical to the DJOL procedure. During the FOK phase, each cue was presented and participants made FOKs about future recognition of the associated target from among five alternatives, using the 0–100 confidence scale.

Finally, a written recognition test was provided on a clipboard, along with a scantron form on which to mark responses. Each of the 44 cues was presented along with five alternatives, one of which was the associated target. Participants filled in the bubble corresponding to the alternative (a, b, c, d, or e) to indicate their response. The recognition test was self-paced. Each of the phases, except the recognition test, was preceded by a practice phase.

Experiment 1

Design and Participants

The design was a 2 × 2 × 3 mixed model factorial design. Cue set size (small, large) was manipulated within subjects. Cueing procedure (extralist, intralist) and Prediction Type (IJOL, DJOL, FOK) were manipulated between subjects. Dependent measures included probability of recall, prediction magnitude (sensitivity), prediction resolution (accuracy), and probability of recognition (FOK only). Participants were 150 undergraduates from Mississippi State University who participated for course credit.

Materials

The key feature of the materials for Experiment 1 was that the cue varied in terms of the number of associates; 22 small-set-size cues and 22 large-set-size cues were used.

Stimulus material

A list of 44 related cue and target word pairs were created using the University of South Florida Word Association Norms (Nelson et al., 1990). Half of the word pairs had small-set-size cues (5 to 9 associates, M = 6.79, SD = 0.16) and half had large-set-size cues (16 to 24 associates, M = 19.75, SD = 0.10). We equated forward association strength (M = .12, SD = .03) and backward association strength (M = 0.03, SD = 0.01) across cue set size and list. The association strength between any given cue and target was relatively low; the target was never the most highly associated member of the cue’s associative set. We also equated target set size, printed word frequency (Kučera & Francis, 1967), concreteness, and connectivity (Nelson et al., 1998). In addition, we used the ListChecker Pro 1.2 program (Eakin, in press) to ensure that each cue was related only to its intended target and not to any other target or cue on the list.

The five alternative forced-choice recognition test presented each of 44 cues (e.g., BROOK) along with its correct (previously studied) target (RIVER) and four associated foils (BABBLING, CREEK, WATER, STREAM). The intended target had the strongest forward association strength with the cue only about 15% of the time. Four versions of the recognition test for each list were created with the cue order and the order of the alternatives differing randomly across versions.

Results

Planned comparisons were conducted to examine the impact of the interaction between cue set size and cueing procedure on each judgment type. Specifically, a significant interaction between the two factors (and an examination of the means) would indicate that cue-set-size effects were obtained for extralist and eliminated for intralist cueing. All reported analyses were conducted using repeated measures, mixed design ANOVAs with prediction magnitude (sensitivity), probability of recall, and resolution (accuracy, as measured by Goodman-Kruskal gamma correlations) as dependent measures. The omnibus F for the full ANOVA for each planned comparison was significant. A criterion of p < .05 was required for significance in all comparisons. The mean prediction magnitude for all judgment types for all three experiments is reported in Table 1. Probability of recall for all three experiments is reported in Table 2. Gammas for Experiment 1 are reported in Table 3.

Table 1.

Metamemory prediction magnitude for all judgment conditions for all three experiments.

			Associative Set
	Prediction Type	Cueing Procedure	Small Set		Large Set
			M	SE	M	SE
Experiment 1	DJOL	Extralist	62	2.42	52	2.40
		Intralist	81	2.52	84	2.50
	FOK	Extralist	73	2.87	61	3.41
		Intralist	85	2.70	85	3.22
	IJOL	Extralist	72	3.30	74	2.99
		Intralist	71	3.12	74	2.82
Experiment 2	DJOL	Extralist	66	2.25	56	2.21
		Intralist	87	2.25	86	2.21
	FOK	Extralist	70	3.06	62	3.00
		Intralist	88	3.16	88	3.10
	Standard IJOL	Extralist	70	3.57	70	3.57
		Intralist	79	3.20	81	3.19
	No-cue IJOL	Extralist	77	2.72	78	2.55
		Intralist	76	2.77	78	2.59
Experiment 3	DJOL	Extralist	79	2.69	74	2.63
		Intralist	91	3.07	91	2.99
	FOK	Extralist	84	1.98	78	2.19
		Intralist	94	1.92	93	2.12
	Standard IJOL	Extralist	78	2.83	76	2.77
		Intralist	83	2.67	83	2.61
	No-cue IJOL	Extralist	78	3.11	79	2.92
		Intralist	85	2.84	84	2.67

Table 2.

Recall performance for all judgment types for all three experiments.

			Associate Set Size
	Judgment Type	Cueing Procedure	Small Set		Large Set
			M	SE	M	SE
Experiment 1	DJOL	Extralist	0.52	0.04	0.37	0.03
		Intralist	0.74	0.04	0.75	0.04
	FOK	Extralist	0.56	0.04	0.41	0.03
		Intralist	0.79	0.03	0.81	0.03
	IJOL	Extralist	0.80	0.03	0.79	0.03
		Intralist	0.83	0.03	0.85	0.03
Experiment 2	DJOL	Extralist	0.53	0.03	0.38	0.03
		Intralist	0.81	0.03	0.81	0.03
	FOK	Extralist	0.56	0.03	0.41	0.03
		Intralist	0.83	0.03	0.83	0.03
	Standard IJOL	Extralist	0.75	0.03	0.75	0.03
		Intralist	0.85	0.03	0.83	0.03
	No-cue IJOL	Extralist	0.62	0.03	0.45	0.03
		Intralist	0.82	0.03	0.82	0.03
Experiment 3	DJOL	Extralist	0.75	0.02	0.66	0.03
		Intralist	0.91	0.03	0.87	0.03
	FOK	Extralist	0.81	0.02	0.70	0.02
		Intralist	0.93	0.02	0.90	0.02
	Standard IJOL	Extralist	0.83	0.04	0.79	0.04
		Intralist	0.83	0.04	0.78	0.04
	NoQJOL	Extralist	0.76	0.04	0.69	0.04
		Intralist	0.88	0.04	0.87	0.04

Table 3.

Metamemory accuracy for all judgments conditions for Experiment 1.

			Semantic Set
	Judgment Type	Cueing Procedure	Small Set		Large Set
			M	SE	M	SE
Experiment 1	IJOL × Recall	Extralist	0.27	0.12	0.43	0.09
		Intralist	0.41	0.13	0.28	0.09
	DJOL × Recall	Extralist	0.72	0.06	0.62	0.08
		Intralist	0.77	0.07	0.75	0.08
	FOK × Recall	Extralist	0.86	0.04	0.86	0.03
		Intralist	0.86	0.04	0.88	0.03

DJOL Results

DJOL magnitude

The interaction between cue set size and cueing procedure was significant, F(1, 46) = 17.38, p < .001, η_p² = .27. Under extralist cueing, predictions were reliably higher for small- (M = 62.19, SE = 2.42) than for large- (M = 51.79, SE = 2.40) set-size cues. Under intralist cueing, cue-set-size effects were eliminated. Mean DJOLs were similar for small- (M = 80.75, SE = 2.52) and for large- (M = 83.85, SE = 2.50) set-size cues, F < 1.

DJOL recall

The interaction between cue set size and cueing procedure was significant, F(1, 46) = 20.62, p < .001, η_p² = .31. Recall was higher for small- (M = .52, SE = .04) than for large- (M = .36, SE = .03) set size cues under extralist cueing. As expected, set-size effects were eliminated under intralist cueing. Recall levels were similar for small- and large-set-size cues (M = .73, SE = .04 and M = .75, SE = .03, respectively), F < 1.

FOK Results

FOK magnitude

The expected interaction between cue set size and cueing procedure for FOK magnitude of all items (recalled and unrecalled) was found, F(1, 49) = 18.69, p < .001, η_p² = .28, with reliable cue-set-size effects under extralist cueing. As had been the case for DJOLs, FOKs were higher for small- (M = 73.35, SE = 2.87) than for large- (M = 61.31, SE = 3.41) set-size cues. Under intralist cueing, no set-size effect was observed (small set size M = 85.35, SE = 2.70 and large set size M = 84.53, SE = 3.22, respectively), F < 1.

FOK recall

The interaction between cue set size and cueing procedure was significant, F(1, 49) = 33.43, p < .001, η_p² = .41, with cue-set-size effects obtained under extralist cueing and eliminated under intralist cueing. Under extralist cueing, the expected difference was found (small set-size M = .53, SE = .03 and large set size M = .41, SE = .03). Again, recall was roughly equivalent under intralist cueing for the two set-size conditions (small set-size M = .79, SE = .03 and large set-size M = .81, SE = .03, respectively), F < 1.

FOK recognition

Recognition was better than recall, M = .85, SE = .03. However, the interaction between cue set size and cueing procedure for recognition memory was not significant, F < 1.

IJOL Results

IJOL magnitude

As predicted, IJOL magnitude did not vary with cue set size for either cueing procedure, and the interaction between cue set size and cueing procedure was not significant, F < 1. JOLs were insensitive to implicit interference.

IJOL recall

To our surprise, recall performance also did not vary as a function of cue set size or cueing procedure, with the interaction between cue set size and cueing procedure, F < 1. Cue set size effects were not obtained for extralist cueing.

Cueing Procedure Results

For the DJOL and FOK conditions, significant main effects of cueing procedure on recall were also obtained, F(1, 46) = 39.25, p < .001, η_p² = .46 and F(1, 49) = 55.15, p < .001, η_p² = .53, respectively. Recall was better for intralist than for extralist cueing. This finding is typical in studies comparing the two (e.g., Nelson, McEvoy, & Schreiber, 1990, Exp. 2; Nelson, McEvoy, Janczura, & Xu., 1993). However, for the IJOL condition, recall was similar for the two cueing procedures, p = 1.51.

Delayed JOL Effect Comparison

Accuracy, as measured by Goodman-Kruskall gamma correlations between judgment and recall was calculated for IJOLs versus DJOLs. The overall mean G = .35, SE = .05, indicating above-chance resolution of IJOLs in predicting recall, t(146) = 24.14, p < .001. For DJOLs, the overall mean resolution was G = .71, SE = .05, a value reliably higher than the resolution seen for IJOLs (G = .35, SE = .05), t(94) = 5.78, p < .001. The typical finding that DJOLs are more accurate than IJOLs (e.g., Dunlosky & Nelson, 1994) was replicated in the present study.

Discussion

As predicted, metamemory predictions made prior to test, after study, reflected the effects of implicit interference on memory. Cue set size effects were obtained in recall in both the DJOL and FOK conditions under extralist cueing and eliminated under intralist cueing. In addition, both DJOLs and FOKs varied with cue set size in a manner similar to cued recall. Predictions were higher given small- than large-set-size cues under extralist cueing and equated under intralist cueing. Similar to prior research on FOKs under implicit interference effects (e.g., Eakin & Hertzog, 2010), FOKs tracked the effects on recall, rather than recognition – the type of memory they were supposed to be predicting. Also similar to prior research, DJOLs were more accurate than IJOLs. The present results are consistent with arguments that DJOLs (but not IJOLs) are thought to be influenced in large part by target accessibility following either explicit or implicit retrieval attempts (Nelson et al., 2004). It appears that both DJOLs and FOKs are influenced by accessibility, given the influence of implicit interference upon them.

IJOLs did not vary with cue set size. Predictions were the same for small- and large-set-size cues, regardless of cueing procedure. Under the assumption that implicit interference effects arise only in the context of retrieval, this outcome was expected. However, in the IJOL condition, recall did not behave as expected; cue-set-size effects were not obtained for either cueing procedure. A closer examination of the procedure for IJOLs suggests a potential reason for the lack of cue-set-size effects in recall for that metamemory condition. According to PIER2, under extralist cueing, cue-set-size effects occur because the cue is not studied in conjunction with the target. Therefore, all of the associates of the cue, including the target, are part of the sampling set and compete for retrieval given the cue at recall. Cue-set-size effects are eliminated under intralist cueing because the cue and target are studied together, effectively reducing the sampling set to associates that are related to both the cue and the target. In the IJOL condition, the sampling set could be constrained in the extralist procedures because, although the target was presented alone under extralist cueing, its associated cue was presented immediately thereafter in order to elicit the IJOL. Therefore, the target and cue were presented in close temporal proximity, unintentionally resulting in a backward intralist cueing procedure (e.g., WINGS → BIRD). This interpretation is supported by the finding that recall was better under intralist than extralist cueing for the DJOL and FOK conditions, but the same for the two cueing procedures for the IJOL condition. Presumably, studying the target in conjunction with the cue that later prompts its recall, as occurs under intralist cueing, aids memory performance relative to studying the target in isolation, as occurs under extralist cueing. The benefit of the backward intralist cueing procedure in reducing implicit interference effects under extralist cueing also aided memory overall in that condition.

The lack of sensitivity of JOLs to implicit interference effects was rendered ambiguous given the lack of implicit interference effects on recall. In Experiment 2, a new method for collecting IJOLs was used to preserve implicit interference effects on memory, enabling a better evaluation of whether IJOLs are influenced in the same way by implicit interference as are predictions at retrieval.

Experiment 2

To avoid the backward intralist effect on memory, Experiment 2 implemented a no-cue IJOL condition, as would be obtained in free-recall JOLs (Castel, 2008). In this condition, participants made IJOLs immediately after the studied target when prompted, but without being provided with the cue word. Because the IJOL cue was not the related cue to be later used in extralist cueing, no winnowing of the target’s associative set should have occurred before extralist cueing. We predicted that set-size effects on IJOLs would be observed in the no-cue IJOL condition, but not in the typical IJOL procedure. Our expectation was also that the two immediate IJOL conditions would produce equivalent IJOL magnitudes.

Design and Participants

The design was a 2 × 2 × 4 mixed model factorial design. Cue set size (small, large) was manipulated within subjects. Cueing procedure (extralist, intralist) and Prediction Type (DJOL, IJOL, FOK, no-cue IJOL) were manipulated between subjects. Dependent measures included probability of recall, prediction magnitude (sensitivity). Planned comparisons were conducted to examine the cue set size and cueing procedure interaction within each of the four prediction types for each of the relevant dependent measures. Participants were 249 undergraduate students at Mississippi State University who participated for course credit.

Materials

The same materials used in Experiment 1 were used in Experiment 2.

Procedure

Experiment 2 followed the general procedure. For the no-cue IJOL condition, participants studied a word pair or target under the two cueing procedures as usual. However, under extralist cueing, they were instructed to predict their memory for the target they had seen just previously; a slide was presented after study that said “Enter your judgment from 0–100.” After making their judgment, they rated their image for vividness and the next word pair or target was presented. For intralist cueing, after studying the cue-target pair, participants were instructed to judge their ability to remember the target they had just seen previously. Judgments were prompted in the same manner as in the extralist condition.

Results

Statistical procedures were identical to Experiment 1.

DJOL and FOK Results

We replicated the basic pattern of results from Experiment 1 for DJOLs and FOKs. The magnitudes of both judgments were influenced by implicit interference in a manner that mirrored cued recall; implicit interference effects were obtained under extralist cueing, but eliminated under intralist cueing. In addition, recall was better under intralist than extralist cueing.

IJOL Results

The critical issue was the contrast between standard IJOLs and cue-only JOLs.

IJOL magnitude

We had predicted no impact of implicit interference for IJOLs, and expected no effects of cueing procedure on JOL magnitudes. The latter prediction was not fully borne out. A planned comparison of prediction magnitude for extralist cueing between the IJOLs (M = 70.13, SE = 3.15) and no-cue IJOLs (M = 77.74, SE = 2.68) barely missed significance, F(1, 55) = 3.39, p = .07, η_p² = .06, suggesting a possible effect of cueing procedure on JOL magnitudes. More critically, however, both types of IJOLs were insensitive to set size. For standard IJOLs, we found no reliable interaction between cue set size and cueing procedure, F(1, 52) = 2.25, p = .14, η_p² = .04. Likewise, no-cue IJOLs manifested no interaction, F < 1. Whether JOLs were elicited with presentation of the cue or not, IJOL magnitude did not vary with cue set size.

IJOL recall

As in Experiment 1, cue set size and cueing procedure for standard IJOLs did not interact in influencing recall, F(1, 52) = 1.19, p = .66, η_p² = .004. Most critically, the interaction between cue set size and cueing procedure was significant in the new no-cue IJOL condition, F(1, 63) = 33.76, p < .001, η_p² = .35. A planned comparison of probability of recall for extralist cueing between the IJOL (M = .75, SE = .03) and no-cue IJOL (M = .53, SE = .02) conditions was significant, F(1, 55) = 38.93, p < .001, η_p² = .41. Cue set size effects on recall were only obtained under extralist cueing when a cue was not used to elicit the IJOL.

Figure 2 shows a comparison of recall and prediction magnitude for the IJOL and no-cue IJOL conditions. In the standard IJOL condition, memory was similar regardless of cueing procedure. However, for no-cue IJOLs, the typical advantage for intralist over extralist cueing was obtained. Thus, as hypothesized, removing the IJOL cue reinstated implicit interference effects on recall without producing a similar effect on IJOLs themselves.

Figure 2.

Recall and IJOL magnitude for the standard and no-cue IJOLs in Experiment 2.

Probability of recall and IJOL prediction magnitude for the standard and no-cue IJOLs for each of the cue set size and cueing procedure conditions in Experiment 2.

Discussion

Experiment 2 demonstrated that cue-set-size effects in recall could be obtained for extralist cueing and eliminated under intralist cueing in the no-cue IJOL condition if the to-be-presented cue was not used to prompt IJOLs. In addition, in the no-cue IJOL condition, memory overall was worse for extralist than for intralist cueing, a finding that was not obtained in the standard IJOL condition. Apparently, when the backward association was eliminated by eliminating the cue at prediction in the no-cue IJOL condition, typical implicit interference effects were obtained in recall. In contrast, when the cue was presented to prompt standard IJOLs, they were not influenced by implicit interference. Similar to Experiment 1, standard IJOLs did not vary with cue set size; however, implicit interference effects were not obtained in recall in the standard IJOL condition. The findings demonstrated that the lack of cue-set-size effects in memory in the standard IJOL condition was due to the backward association between the target and cue created by using the cue for the IJOL prompt.

Although Experiment 2 succeeded in reinstating the implicit interference effect on recall, one can question whether the lack of cue-set-size effects on no-cue IJOLs is the strongest possible demonstration that IJOLs are insensitive to implicit interference effects. The cue whose set size generates implicit interference in this experiment was not present when the JOL was made in the no-cue IJOL condition. For the standard IJOL condition, the cue is present and can influence standard IJOLs, but its presence eliminates implicit interference effects at retrieval. The hypothesized pattern involves an effect on recall in one condition and no effect on judgments in another (in effect, the comparison is made between no-cue IJOL recall and standard IJOL judgments). A more compelling case for insensitivity of IJOLs to implicit interference would be to show that IJOLs are not influenced by implicit interference when the same generator of set size effects is available both at the time of the judgment and at recall. In Experiment 3, the materials were changed to create such a condition.

Experiment 3

Experiment 3 made the source of implicit interference present both at the time of the no-cue IJOL and at recall by varying the associative set size of the target, rather than the cue. The potential influence of many versus few associates is available at prediction because the target-set-size is manipulated and predictions are made without an accompanied cue. The no-cue IJOL condition should avoid the backward effects of cue presentation on reducing the effective associative set (as in Experiment 2) whereas the stimulus with varying associative set size that generates implicit interference is still in immediate memory while making the IJOL. Target-set-size effects in memory are similar to cue-set-size effects; recall is better for small- than for large-set-size targets under extralist cueing and similar for the two under intralist cueing (e.g., Nelson et al., 1992). In addition, both DJOLs and FOKs have been shown to track target-set-size effects in memory (e.g., Schreiber, 1998).

Thus, the critical prediction is that the no-cue IJOL condition will generate implicit interference effects on recall without the IJOLs themselves manifesting the effect. The standard IJOL condition was also included in the experiment to test, as expected, whether backward intralist cueing effects would also occur with target set size. For comparison purposes, the DJOL and FOK conditions that generated judgments sensitive to implicit interference in the previous experiments were also included.

Design and Participants

The design was a 2 × 2 × 4 mixed model factorial design. Target set size (small or large) was manipulated within subjects. Cueing procedure (extralist or intralist) and Prediction Type (IJOL, DJOL, FOK, or no-cue IJOL) were manipulated between subjects. Participants were 249 undergraduate students at Mississippi State University who participated for course credit.

Materials

A new list of 44 related cue and target word pairs were created using the University of South Florida Word Association Norms (Nelson et al., 1990). Half of the word pairs had small-set-size targets (3 to 8 associates, M = 6.64, SD = 1.53) and half had large-set-size targets (15 to 25 associates, M = 19.82, SD = 3.14). All other factors equated on the cue-set-size list were also equated on the target-set-size list. ListChecker Pro 1.2 (Eakin, in press) ensured that each cue was associatively related only to its intended target and not to any other target or cue on the list.

Procedure

The general procedures and those described in Experiment 2 for the no-cue IJOLs were also used in Experiment 3.

Results

Dependent measures and statistical procedures were identical to the previous experiments.

DJOL and FOK Results

The pattern of results from Experiments 1 and 2 were obtained in Experiment 3 for DJOLs and FOKs. Target set size impacted both judgment types and they tracked the effects in recall, whether implicit interference effects were obtained under extralist cueing or eliminated under intralist cueing. Recall was better under intralist than extralist cueing.

IJOL Results

The findings for the standard IJOLs replicated those from Experiments 1 and 2. There was no reliable interaction between target set size and cueing procedure for the judgments, F(1, 66) = 2.58, p > .10. However, the probability of recall also did not interact with target set size and cueing procedure, F < 1. There was no difference in recall overall between extralist and intralist cueing. Analogous to the first two experiments, a backward intralist effect was obtained by using the extralist cue to prompt the IJOL, eliminating target set size effects on recall that normally would be found under extralist cueing.

The critical data were generated by the no-cue IJOL condition (Figure 3 compares recall and prediction magnitude between the two IJOL conditions). As hypothesized, target set size and cueing procedure did not interact in affecting no-cue IJOLs, F(1, 53) = 1.91, p = .17. In contrast, target set size effects were obtained in recall; the interaction between target set size and cueing procedure was significant, F(1, 53) = 5.27, p = .03, η_p² = .09. The no-cue procedure avoided the backward intralist effect, so that target set size effects were obtained under extralist cueing and were eliminated under intralist cueing, as in previous research by Nelson and colleagues. In addition, the typical advantage for intralist versus extralist cueing was obtained in the no-cue IJOL condition.

Figure 3.

Recall and IJOL magnitude for the standard and no-cue IJOLs in Experiment 3.

Probability of recall and IJOL prediction magnitude for the standard and no-cue IJOLs for each of the cue set size and cueing procedure conditions in Experiment 3.

Discussion

As expected, DJOLs and FOKs tracked the impact of target set size on memory. The recall effects were smaller than previously found for cue set size in Experiments 1 and 2, which is frequently found (e.g., Nelson et al., 1992). Regardless, they were reliable and systematic. Thus, metamemory predictions that are arguably influenced by target accessibility were impacted by implicit interference effects based on target set size.

In contrast, no-cue IJOLs were not influenced by target set size; predictions were similar for small- and large-set-size targets under both extralist and intralist cueing. Yet it was still the case that a pattern of implicit interference effects were obtained for extralist cued recall in the no-cue IJOL condition. Recall was better for small- than large-set-size targets under extralist cueing, whereas target-set-size effects were eliminated under intralist cueing. Although implicit interference impacted memory, metamemory judgments made at the time of encoding did not reflect this impact.

General Discussion

In the three experiments, metamemory predictions that were potentially influenced by retrieval – DJOLs and FOKs – were shown to track implicit interference effects in memory. In all three experiments, implicit interference effects were obtained in recall for both DJOLs and FOKs under extralist cueing and eliminated for both under intralist cueing. In addition, memory was better for intralist than extralist cueing overall. In all three experiments, standard IJOLs made at encoding failed to track implicit interference effects that are typically obtained at recall, and those effects were not obtained in recall for IJOLs. However, Experiments 2 and 3 established that the method used to measure IJOLs caused the lack of implicit interference effects in memory for that condition. Use of the extralist cue to prompt the IJOL resulted in a backward intralist cueing procedure, thereby eliminating the typical cue-set-size effects obtained under extralist cueing.

In Experiments 2 and 3, implicit interference did occur for extralist cued recall, but no-cue IJOLs did not predict this interference. Although the qualia that will result in implicit interference are present when the cue (Exp. 2) or target (Exp. 3) is presented in the standard IJOL condition, the judgments did not vary with associative set size, regardless of cueing procedure. These outcomes corroborated the hypothesis motivating this study, namely, that IJOLs are insensitive to implicit interference effects during retrieval, whereas DJOLs and FOKs are sensitive to those effects.

This study adds to our understanding of the qualitative differences between metacognitive judgments arising at study versus those that occur during retention or test. IJOLs at the initial study opportunity can be influenced by a number of different sources of information, such as stimulus characteristics (e.g., Koriat, 1997), encoding fluency (e.g., Hertzog, Dunlosky, Kidder, & Robinson, 2003), retrieval fluency of aspects of the cue, and so on (e.g., Benjamin, Bjork, & Schwartz, 1998; Castel, 2008; Finn & Metcalfe, 2008; Koriat & Bjork, 2006; Kornell & Bjork, 2009). However, when people make IJOLs, apparently they are not influenced by implicit retrieval interference or the variables that produce that interference. Even though, in principle, a variable like cue set-size or target set-size is computable from observed characteristics during study, observers apparently do not do so. Hence, we conclude that IJOLs are not influenced by variables that lead to implicit interference when a retrieval search for target information is generated by a request for a metacognitive judgment. Phenomena that arise only during retrieval access, including implicit interference effects, do not influence IJOLs. Only judgments that can be influenced by qualia generated by target feature accessibility are influenced by implicit interference. Admittedly, the present research demonstrates the dissociation of implicit interference effects on IJOLs and DJOLs without directly demonstrating the sources of information people actually used to make IJOLs in this experimental context.

Although our study was not designed to evaluate alternative theories of FOKs, we note in passing that the target-set-size based implicit interference effects on FOKs in Experiment 3 seem more consistent with an accessibility account of FOKs (Koriat, 1993; 1997; Koriat & Levy-Sadot, 2001) than with a cue familiarity account of FOKs (e.g., Metcalfe et al., 1993), similar to the source of influence on DJOLs under conditions of explicit retroactive interference (Eakin, 2005).

Our results also are relevant to arguments about the basis for the delayed-JOL effect. The findings from Experiment 1 demonstrated a DJOL effect (Nelson & Dunlosky, 1991) in that metamemory was more accurate when predictions were made after a delay than when they were made immediately. Our implicit interference effects on IJOLs versus DJOLs support the argument that an important difference between IJOLs and DJOLs involves the influence of retrieval-based access to information about the target (e.g., Nelson et al., 2004). When JOLs are delayed, implicit interference influences both DJOLs as it did FOKs.

In summary, when people make DJOLs and FOKs, their predictions are influenced by the implicit interference produced by a larger number of associates for large-set-size cues and targets than for those with a small associative set -- unless the set is reduced by the presence of a related cue or target during intralist cueing. Conversely, IJOLs are not influenced by stimulus properties that lead to implicit interference at retrieval even under extralist cueing. The information people access when making IJOLs is not based on the same PIER2-type sampling of the target set. Nor, apparently do people infer the effect of varying cue set size or target set size when making their IJOLs.