Ironic Effects of Drawing Attention to Story Errors

Abstract

Readers learn errors embedded in fictional stories and use them to answer later general knowledge questions (Marsh, Meade, & Roediger, 2003). Suggestibility is robust and occurs even when story errors contradict well-known facts. The current study evaluated whether suggestibility is linked to participants’ inability to judge story content as correct versus incorrect. Specifically, participants read stories containing correct and misleading information about the world; some information was familiar (making error discovery possible), while some was more obscure. To improve participants’ monitoring ability, we highlighted (in red font) a subset of story phrases requiring evaluation; readers no longer needed to find factual information. Rather, they simply needed to evaluate its correctness. Readers were more likely to answer questions with story errors if they were highlighted in red font, even if they contradicted well-known facts. Though highlighting to-be-evaluated information freed cognitive resources for monitoring, an ironic effect occurred: Drawing attention to specific errors increased rather than decreased later suggestibility. Failure to monitor for errors, not failure to identify the information requiring evaluation, leads to suggestibility.

The goal of fictional movies, books, and short stories is normally to entertain, not to teach people about the world. But because stories are often set in familiar time periods and places, they can also be sources of information. Fictional stories, however, are not necessarily accurate, and thus can also be sources of misinformation. For example, the famous book A Wrinkle in Time (L’Engle, 2007) refers to New York City as the capital of New York, even though it has been Albany since 1797. Numerous studies suggest that reading such errors affects performance on later tasks. For example, readers are slower to reject false statements such as “Mental illnesses are contagious” after reading them in a story (Gerrig & Prentice, 1991). Similarly, participants rate statements like “Aerobic exercise weakens your heart and lungs” as more true after reading them in stories (Prentice, Gerrig, & Bailis, 1997; Wheeler, Green, & Brock, 1999). Because readers do not scrutinize fiction as carefully while reading, stories may influence readers more than the same text labeled as fact (Green, Garst, Brock, & Chung, 2006).

In related studies, participants read stories containing factual errors and later reproduce the errors as answers on a general knowledge test (e.g., Marsh, 2004; Marsh, Meade, & Roediger, 2003). Specifically, participants read stories containing both correct and misleading factual references. For example, a story about an art thief correctly references “the capital of Kentucky, Frankfort” or erroneously references “the capital of Kentucky, Louisville.” Later, participants take a general knowledge test that includes questions corresponding to story information, such as “What is the capital of Kentucky?” Readers benefit from reading correct answers in the story: they answer more questions correctly than if they had not read the story (Marsh & Fazio, 2006). But reading misinformation has negative consequences. First, reading misinformation reduces the proportion of questions answered correctly below baseline. Specifically, reading misinformation interferes with participants’ ability to answer questions they should have been able to answer correctly; suggestibility is not limited to instances where participants do not know the correct answer. Second, reading misinformation leads participants to answer more test questions with the story errors, as compared to when participants have not read the story. Importantly, readers are not immune to suggestibility even when story information contradicts well-known facts. For instance, when participants read stories containing obvious contradictions to general knowledge (e.g., St. Petersburg is the capital of Russia), they still reproduce the misinformation on a later test, albeit at a lower rate than when the misinformation references more obscure facts (e.g. Marsh & Fazio, 2006).

Reproduction of story errors is robust; manipulations that reduce suggestibility in other false memory paradigms do not reduce suggestibility in the story paradigm. For instance, warnings given before encoding help eyewitnesses avoid misinformation embedded in post-event narratives (Green, Flynn, & Loftus, 1982). Similarly, pre-encoding warnings reduce false memories for words; for example, learning about the Deese-Roediger-McDermott paradigm helps participants avoid falsely recalling “sleep” after listening to “bed, rest, tired…” (e.g. McCabe & Smith, 2002). However, general warnings have not helped readers avoid reproducing story errors, even when given before story reading. In one study, readers warned that authors often take liberties with facts became more conservative on the final test (Marsh & Fazio, 2006). Although they were less likely to answer test questions with misinformation from the stories, they also made fewer errors on questions unrelated to the stories they read. Thus, warning participants did not help them selectively avoid story errors, even when story information contradicted well-known facts.

Other attempts to increase monitoring for errors have backfired, increasing rather than decreasing reproduction of misinformation. For example, suggestibility increased when readers re-read the stories, even though this gave them a second chance to notice the errors (Marsh, et al., 2003). A similar ironic effect occurred when story presentation was slowed: participants became more suggestible (Fazio & Marsh, 2008). Although slowly reading a post-event narrative helps mock eyewitnesses catch an inconsistency between the narrative and what they witnessed (Tousignant, Hall, & Loftus, 1986), the same slow presentation does not reduce suggestibility in the fiction paradigm.

Why aren’t readers able to take advantage of manipulations aimed to increase monitoring, especially when they should have the knowledge required to notice contradictions to well-known facts? Most of these interventions still place a large burden on the participant. Even when a story is presented more slowly or re-read, the participant still needs to find factual references before they can be evaluated, while also continuing to follow the plot of the story. In this study, we examined whether suggestibility decreases if the to-be-evaluated factual references were easily identified in the story, thus making it clearer to participants what needed to be evaluated for truthfulness. Specifically, parts of the story were highlighted in red font (including both correct facts and misinformation). The reader did not need to locate factual references, but she still needed to evaluate their correctness.

As with prior research, the stories contained references to both familiar and obscure knowledge. If highlighting story information improves monitoring, participants should easily detect errors that contradict their stored knowledge; however, the manipulation might not help participants evaluate more obscure information. Thus it is possible that improved monitoring will only help participants detect errors that clearly contradict their general knowledge and have no effect on references to more obscure knowledge. If highlighting helps readers detect errors, it should reduce reproduction of the errors on the final general knowledge test. When eyewitnesses noticed contradictions between something they saw and something they read, they became less suggestible (Loftus, 1979). Importantly, reading obvious contradictions led participants to be more resistant to other, less obvious contradictions. Thus, if readers notice more obvious errors in the stories, they might read the other highlighted text more carefully and notice more obscure errors as well.

A second possibility is that readers will attend to the highlighted factual references, but will fail to evaluate them. Thus, highlighting may have no effect if readers are not able to critically compare what they read to their stored knowledge. Leading participants to carefully evaluate information for errors is difficult. In fact, even participants warned that they would encounter simple errors in general knowledge questions (e.g. How many animals of each kind did Moses take on the Ark?”) often fail to detect them (Erickson & Mattson, 1981; Bottoms, Eslick, & Marsh, 2010). Similarly, when participants detect story errors (via key press), they notice fewer than is predicted based on norms (Marsh & Fazio, 2006). Thus, highlighting may not influence general knowledge test performance if readers fail to notice that the highlighted facts contradict stored knowledge. Furthermore, highlighting information might have an ironic effect, boosting suggestibility. Highlighting should signal the reader to attend more closely to the text (e.g., Lorch, 1989). To the extent that they do this and fail to catch errors, it may boost memory for highlighted text.

In short, red font should draw readers’ attention to the text. The critical question is whether this increased attention will translate into careful evaluation of the highlighted text. Fewer errors will be produced on the final general knowledge test if highlighting successfully leads participants to catch the errors while reading. In contrast, there are two possibilities if highlighting does not improve evaluation of the text. Highlighting might have no effect on later performance, or it might increase suggestibility, if participants have improved memory for the highlighted text.

To preview, we examined whether highlighting factual references improved participants’ ability to effectively evaluate their truthfulness. Participants were warned that the stories might contain factual errors and then read stories containing facts and misinformation. Critically, some text requiring evaluation appeared in a visually distinctive red font, and the rest appeared in black font. Additionally, some text referenced well-known facts and some referenced more obscure knowledge. Readers should have the requisite knowledge to detect obvious contradictions to their stored knowledge, whereas references to more obscure knowledge may represent new learning. Of interest was whether readers used story information to answer later general knowledge questions, and more importantly whether suggestibility depended on the appearance of the font. The key question was whether readers avoided reproducing story errors that contradicted well-known facts, given that they no longer had to locate the factual references and only had to monitor them for correctness.

Method

Participants

One hundred and sixty Duke University undergraduates participated in exchange for partial credit towards a course requirement.

Design

This experiment had a 2 (highlighting: yes, no) × 2 (question difficulty: easy, hard) × 3 (fact frame: correct, not-read, misleading) within-subjects design.

Materials

Four short fictional stories were adapted from Marsh (2004). Each story had a separate theme (art, hunting, a planetarium, or a science fair) and traditional story components: plot, dialogue, and conflict. Each participant read two stories; the other two stories served as the not-read baseline (this was counterbalanced across participants). Story order was determined beforehand and held constant across participants.

Sixteen facts were adapted from the Nelson and Narens (1980) norms and embedded in each story, yielding a total of 64 factual references. As in prior research (Marsh et al., 2003; Marsh & Fazio, 2006), half of the references corresponded to easy questions (correctly answered by 68% in Nelson and Narens’ norming study), whereas the other half corresponded to hard questions (answered correctly by only 15%). Each story presented half of the references (four easy, four hard) in a correct frame and half in a misleading frame (i.e., an incorrect but plausible reference). For example, one version of a story referred to “…the largest ocean, the Pacific” (correct frame), and another version read “…the largest ocean, the Atlantic” (misleading frame). Two versions of each story were needed to counterbalance fact framing across participants.

In one of the stories participants read, all of the words appeared in black Times New Roman font on a white background. The majority of the sentences in the other story also appeared in black font, but critically 32 sentences (or phrases) were highlighted in red font. The critical factual references were embedded in 16 of these red sentences, while the remaining red sentences served as fillers containing non-critical information. Overall, 7.1 words were highlighted per critical sentence (on average, 37% of the sentence) and 15% of the total words in the highlighted story appeared in red font. The story containing highlighted font was counterbalanced across participants and was read first or second equally often.

The general knowledge test consisted of 69 cued recall questions. Thirty-two questions were based on factual references from the read stories (16 per story). Another 32 questions were from the not-read stories (these served as a baseline measure). Five unrelated questions served as easy fillers.

To measure how engaging readers found the stories, we used 9 questions based on Green and Brock’s (2000) transportation scale. Items were rated on a 1 (not at all) to 7 (very much) point scale. A sample item is: “While I was reading the narrative, I could easily picture the events in it taking place.”

Procedure

After giving informed consent, participants learned the experiment consisted of three reasoning tasks: reading comprehension, reasoning, and a general world knowledge test. The entire experiment was computerized using Direct RT (Jarvis, 2006a) and MediaLab software (Jarvis, 2006b).

In the reading comprehension phase, participants read two fictional stories. Before reading, they were warned that “…authors of fiction often take liberties with certain facts or ideas to make a story flow better or to be more entertaining; therefore some of the information you will read may be incorrect.” Participants were also told that the study was investigating ways to draw readers’ attention to certain text: “…some of the words in one of the stories will appear in red font. When the font color changes, you should be sure to pay attention to those words specifically.” This instruction slide provided an example of the font change.

Participants read each story at their own pace, pressing the space bar to advance. Each sentence was presented individually, and participants were not allowed to go back after advancing. To ensure participants were reading carefully, six catch trials (per story) were inserted after non-critical sentences; on these trials, participants were instructed to summarize the last sentence that they read. After reading the first story, participants completed the transportation scale. They then read the second story and completed a transportation scale for that story.

During the reasoning phase (a filler task), participants completed Sudoku puzzles for 5 minutes.

Finally, participants completed the general knowledge test; they were instructed to respond with “I don’t know” rather than guess on these questions. Questions were presented individually in random order. After the test, participants were debriefed. To ensure that nobody left the experiment believing misinformation, participants rated how surprising they found the corrected versions of facts that were previously framed in a misleading format.

Results

Significance was determined at the p < .05 level, unless otherwise noted.

Catch Trial Performance

To ensure participants read the stories carefully, we examined the catch trials. Two independent coders rated the accuracy of each summary, and a third coder resolved discrepancies. Ratings were made on a 3-point scale (0 = completely inaccurate; 1 = included portions of the sentence, but missed main ideas; 2 = accurately summarized). Ratings were averaged across participants, and eight participants scoring two standard deviations below the mean (M = 1.77, SD = .29) were excluded. Thus, the analyses include data from 152 participants.

Transportation Ratings

Transportation ratings were reverse scored, if necessary, and averaged into a mean score for each participant. Participants rated the first (M = 4.10) and second stories (M = 4.16) as equally engaging, t<1. Additionally, readers found the two stories equally engaging, regardless of whether stories contained highlighted text (Ms of 4.09 and 4.18, t(151) = 1.34, SEM = .07, p = .18).

Final General Knowledge Test Performance

Two independent coders classified each answer on the general knowledge test as correct, target misinformation, another wrong answer, or “don’t know” (Cohen’s kappa=.98); a third coder resolved discrepancies. Returning to our example about the largest ocean, “Pacific” would be scored as correct, “Atlantic” as the target misinformation, and “Indian” as another wrong answer.

Correct answers

To investigate how highlighting, question difficulty, and fact framing affected correct answers on the general knowledge test, we computed a 2 (highlighting: yes, no) × 2 (question difficulty: easy, hard) × 3 (fact frame: correct, not-read, misleading) ANOVA on the proportion of questions answered correctly on the general knowledge test. The relevant data are shown in the top panel of Table 1. As expected, there was a main effect of question difficulty: more easy questions were answered correctly (M = .63) than hard questions (M = .30), F(1, 151) = 738.56, MSE = .07, η_p² = .83.

Table 1.

Proportion of General Knowledge Questions Answered Correctly (Top Panel) or with Misinformation (Bottom Panel), as a Function of Prior Story Highlighting, Question Difficulty, and Fact Framing.

Proportion Correct Answers
	Non-highlighted Story			Highlighted Story
	Correct	Not-read	Misleading	Correct	Not-read	Misleading
Easy	.78 (.02)	.62 (.02)	.50(.02)	.79 (.02)	.62 (.02)	.51 (.02)
Hard	.46 (.03)	.24 (.02)	.19 (.02)	.49 (.03)	.26 (.02)	.19 (.02)
M (SE)	62 (.02)	43 (.02)	35 (.02)	64 (.02)	44 (.02)	35 (.02)
Proportion Misinformation Answers
	Non-highlighted Story			Highlighted Story
	Correct	Not-read	Misleading	Correct	Not-read	Misleading
Easy	.04 (.01)	.07 (.01)	.26 (.02)	.03 (.01)	.05 (.01)	.28 (.02)
Hard	.02 (.01)	.07 (.01)	.26 (.02)	.04 (.01)	.06 (.01)	.31 (.02)
M (SE)	03 (.01)	07 (.01)	.26 (.01)	03(.01)	06 (.01)	29 (.02)

Note. Standard Error is in parentheses.

More importantly, a main effect of fact framing emerged, F(2, 302) = 237.56, MSE = .05, η_p² = .61. Participants correctly answered more general knowledge questions after reading the correct answers in the story (M = .63) than if they had not read the story (M = .43), t(151) = 15.27, SEM = .01. Furthermore, reading a misleading reference resulted in fewer correct answers on the final test (M = .35), as compared to the baseline, t(151) = 8.05, SEM = .01. Additionally, participants showed greater benefits of story reading for hard questions and more costs for easy questions, as reflected in an interaction between fact frame and question difficulty, F(2, 302) = 6.12, MSE = .04, η_p² = .04. Reading correct, but obscure, facts in the story boosted performance above baseline (M = .23) more than reading well-known facts (M = .16), t(151) = 3.29, SEM = .02. In contrast, reading misinformation dropped performance below baseline more for easy questions (M = .12) than for hard questions (M = .06), t(151) = 3.23, SEM = .02.

There was no main effect of highlighting, F < 1, indicating references to red and black facts were equally likely to be reproduced on the final test. More importantly, no interactions emerged for highlighting, all Fs < 1. The benefits of reading correctly framed facts were similar regardless of whether those facts were highlighted. Similarly, the costs of reading misinformation did not depend upon highlighted text.

Misinformation answers

To examine whether highlighting, question difficulty, and fact framing affected misinformation answers on the general knowledge test, we computed a 2 (highlighting: yes, no) × 2 (question difficulty: easy, hard) × 3 (fact frame: correct, not-read, misleading) ANOVA on the proportion of general knowledge questions answered with misinformation.

As shown in the bottom panel of Table 1, participants were much more likely to answer a general knowledge question with misinformation if they had read it in the story (M = .28) than if they had not read the story at all (M = .06), t(151) = 16.76, SEM = .01. In addition, participants rarely answered with misinformation (M = .03) after reading the correct answer in a story, and this reduction below baseline (M = .06) was significant, t(151) = 6.46, SEM = .01. These effects resulted in a main effect for fact framing, F(2, 302) = 280.83, MSE = .04, η_p² = .65. After reading misinformation in the stories, misinformation production was robust for both easy (M = .27) and hard questions (M = .29); the interaction between fact framing and ease was not significant, F(2, 302) = 1.34, MSE = .02, η_p² = .01, p > .26.

Most important were the effects of highlighting. The main effect of highlighting was not significant, F(1, 151) = 1.17, MSE = .02, η_p² = .01, p > .28, but critically the interaction between fact frame and highlighting was significant, F(2, 302) = 4.78, MSE = .02, η_p² = .03. Participants were more likely to reproduce misinformation if it was highlighted in the story, (M = .29) than if it was not (M = .26), t(151) = 2.04, SEM = .02. That is, reading highlighted misinformation increased misinformation production almost .24 above the baseline, whereas reading non-highlighted misinformation increased it .19, t(151) = 2.55, SEM = .02. This effect of highlighting occurred for both easy and hard questions; the three-way interaction between font color, fact frame, and ease was not significant, F < 1. Highlighting increased readers’ reproduction of story errors, even when they contradicted well-known facts.

Discussion

These results replicate prior work showing that reading stories affects performance on a later general knowledge test: participants benefited from reading correct information and showed costs from reading misinformation (e.g., Marsh et al., 2003; Marsh & Fazio, 2006; Fazio & Marsh, 2008). However, there was no indication that the highlighting helped readers reject errors. Instead, highlighting had an ironic effect, increasing suggestibility, even when the misinformation contradicted facts the readers likely knew. Though this difference is relatively small, it is critical that highlighting did not reduce suggestibility, as would have been expected if highlighting improved monitoring for errors.

The ironic effect of highlighting is consistent with prior research showing that suggestibility increases when stories are re-read (Marsh et al., 2003) or presented more slowly (Fazio & Marsh, 2008). Compared to prior manipulations, highlighting factual references did remove one burden from the reader: she no longer had to locate the to-be-evaluated information within the stories. However, highlighting did not remove the need to judge factual references as true or false, and our data suggest that readers encoded all of the information rather than monitoring it for correctness. Critically, question difficulty had little effect on the results, even though readers should have been able to reject misinformation that contradicted well-known facts. This failure to detect errors is consistent with other studies in which participants neglect their general knowledge and fail to notice false information (e.g., Erickson & Mattson, 1981; Marsh & Fazio, 2006).

Highlighting affected the learning of misinformation embedded in the stories, but not the benefits of reading correct answers, suggesting story costs and benefits may depend upon different processes. When correct facts are read in the stories, they likely activate pre-existing representations. In contrast, encoding misinformation involves forming a new association between two concepts, as supported by the low level of misinformation production when the stories were not read. Neuropsychological test data from older adults support this distinction, with benefits of story reading mapping onto preserved semantic memory and misinformation production mapping onto preserved episodic memory (Marsh, Balota, & Roediger, 2005). Returning to our study, highlighted text selectively influenced misinformation production because it drew attention to the new associations that needed to be encoded. Pre-existing associations, however, were automatically activated during story reading regardless of font color.

To confirm that highlighting increased the memorability of factual references, we completed a follow-up study in which 16 new participants read the same stories used in the main experiment. Critically, after reading the stories, participants completed an episodic memory test for story information, rather than a general knowledge test. The cued recall test required participants to complete story sentences with the critical factual references previously presented (e.g.,” I liked to imagine paddling around the largest ocean, the ________” should be completed with “Pacific” or “Atlantic,” depending on which version of the story a participant read). Participants were warned not to use their general knowledge to complete the sentences and rather to rely on their memory for what they read in the stories.

To determine whether participants had better memory for highlighted or non-highlighted information, we computed a 2 (fact frame: misleading, correct) × 2 (highlighting: yes, no) × 2 (question difficulty: easy, hard) ANOVA on the proportion of sentences correctly completed with story references. Highlighted references were more memorable (M= .54) than non-highlighted references (M=. 45), F(1,15)=5.84, MSE=.04, η_p²=.28. No higher order interactions emerged for highlighting. Thus, both correct facts and misinformation were more memorable when highlighted. At first, this result may seem contradictory to our finding that highlighting affected misinformation production on the general knowledge test (and had no effect on correct answers). As described above, the sentence completion task required participants to retrieve their episodic memory for what they read in the story (for both correct facts and misinformation). However, answering general knowledge questions required both semantic and episodic associations (Marsh et al., 2005). Reading correct facts in the stories strengthened participants’ semantic memory for those facts, whereas reading misinformation resulted in new episodic memories. While highlighting improved memory for both correct and misleading facts, this increased memorability only affected performance on tests requiring retrieval from episodic memory. Both the sentence completion task, and the retrieval of misinformation on the general knowledge test relied on episodic memory. In contrast, because the retrieval of correct answers on the general knowledge test was linked to strengthened semantic associations, and not episodic memory, highlighting did not influence the production of correct answers on that test. That is, highlighting affects performance when the memory task depends on episodic retrieval.

We close by noting the educational implications of our results. Teachers want to incorporate fiction into the curriculum because students find it engaging (e.g., Smith, 1993). However, because students easily learn errors from fiction, teachers have two choices. They either need to select material that contains no factual inaccuracies, or they need to be careful about how the material is processed. Pointing out what information needs to be evaluated will not suffice; our results suggest that such highlighting will only increase suggestibility (and certainly will not decrease suggestibility). Rather, we expect that students will benefit from knowing each of the specific errors they will encounter. For instance, when participants read an accurate text and watch a film clip that contains an error, suggestibility is only reduced when viewers are warned about the specific discrepancy they will encounter (Butler, Zaromb, Lyle, & Roediger, 2009). Simply telling viewers films may be inaccurate does nothing to reduce suggestibility. Together, these results suggest that care must be taken when incorporating fiction into the classroom. Materials aimed at making learning more engaging might backfire and lead to the learning of incorrect information.