Patient Knowledge and Recall of Health Information Following Exposure to “Facts and Myths” Message Format Variations

Abstract

Objective

To assess if exposure to varying “facts and myths” message formats affected participant knowledge and recall accuracy of information related to influenza vaccination.

Methods

Consenting patients (N=125) were randomized to receive one of four influenza related messages (Facts Only; Facts and Myths; Facts, Myths, and Refutations; or CDC Control), mailed one week prior to a scheduled physician visit. Knowledge was measured using 15 true/false items at pretest and posttest; recall accuracy was assessed using eight items at posttest.

Results

All participants’ knowledge scores increased significantly (p <0.05); those exposed to the CDC Control message had a higher posttest knowledge score (adjusted mean=11.18) than those in the Facts Only condition (adjusted mean 9.61, p=<0.02). Participants accurately recalled a mean of 4.49 statements (sd=1.98). ANOVA demonstrated significant differences in recall accuracy by condition [F(3, 83)=7.74, p<.001, η² = 0.22].

Conclusion

Messages that include facts, myths, and evidence to counteract myths appear to be effective in increasing participants’ knowledge. We found no evidence that presenting both facts and myths is counterproductive to recall accuracy.

Practice Implications

Use of messages containing facts and myths may engage the reader and lead to knowledge gain. Recall accuracy is not assured by merely presenting factual information.

1. INTRODUCTION

Each year, influenza affects three to five million people worldwide, causing between 250,000 and 500,000 deaths per year.(1) In the United States, influenza and influenza-related illnesses cause approximately 36,000 deaths annually, 90% of which occur among those ages 65 and older.(2-5) Influenza vaccination is the most effective prevention technique.(1) In the United States, vaccination rates among those 50 – 64 and those 65 and older fall far short of the related Healthy People 2020 goals of 80% and 90%: Current vaccination rates among these populations are approximately 42.7% and 64.9%, respectively.(6-8)

Numerous strategies have been attempted to increase vaccination rates, including patient education, increased provider recommendation, standing orders, client reminder and recall systems, as well as interventions designed to address a particular target population (e.g., older adults, individuals of different race and ethnicity, health care providers, pregnant women, etc.).(8) Patient interventions have included printed educational materials, such as the Centers for Disease Control and Prevention's (CDC) “Flu Vaccine Facts & Myths,”(9) which attempts to refute common myths with factual information. Indeed, health messages using facts and myths organizational patterns abound; such a format appears to be a logical and appealing method to reinforce accurate information and refute false information.(9-10)

However, recent research called into question the efficacy of such an approach, specifically when the target audience includes older adults.(11, 12) This research demonstrates that older adults often misremember myths as facts, a result that may be due to an “illusion of truth effect,” occurring when individuals previously exposed to a claim have forgotten the context, or stated veracity of the claim.(11-13) Concerns are that the repetition of false information, even if repeated in an attempt to discredit that information, serves to increase one's familiarity with the information. In an unpublished study, Skurnik, Yoon and Schwarz, presented participants with a previous version of the CDC flyer (which included statements identified as being “true” or “false”) and provided others with a version of the flyer that contained only facts (12). They discovered that errors made when assessed immediately after participants read the flyers were both few and random, whereas after a delay of 30 minutes a larger proportion of myths were identified as being true than were facts identified as being false. Increased familiarity and fluency, then, may lead to an acceptance of the false information. Specifically, individuals may be reacting more to the familiarity of the claim (i.e., it is something that they have heard before) than to the actual truth value of the claim because messages that are familiar tend to be easier for a receiver to process (12, 13). Such research cautions against the use of a facts and myths format, raising concerns that message receivers may errantly believe the myths to which they have been exposed are facts, particularly when assessed following a delay after presentation of the messages. This research suggests that a more prudent approach would be to present only factual information.(12)

Research from the discipline of communication has focused on assessing the types of messages that may be most persuasive to a given audience. For example, when one prepares for a debate, a common strategy is to consider opposing arguments in order to be prepared to address them effectively. Persuasion researchers have studied whether presenting only the position advocated (or, one side of the argument) is more or less persuasive to an audience than is acknowledging the existing counterarguments (i.e., recognizing multiple sides of an argument).(14-17)

Hence, messages that present only supporting arguments in favor of a specific position (e.g., facts only) have been labeled “one-sided messages,” whereas messages that present supporting arguments in favor of the specific position and also address opposing viewpoints (e.g., facts and myths) have been labeled “two-sided messages.”(14-17) Message sidedness research suggests that there are two distinct types of two-sided messages.(14) A two-sided non-refutational message acknowledges the existence of opposing arguments, but makes no direct attempt to refute the arguments (i.e., facts and myths), whereas a two-sided refutational message not only acknowledges opposing arguments, but also refutes the opposing arguments by attacking their plausibility or offering additional undermining evidence (i.e., facts, myths, and refutations).(14-16) Subsequent studies and multiple meta-analyses investigating the effects of these three message types demonstrated that a two-sided refutational message is more persuasive than a one-sided message, which in turn is more persuasive than a two-sided nonrefutational message.(14-16) Moreover, when a myth is explicitly labeled as such, it may serve as a warning that the information that follows constitutes misinformation, which may reduce the likelihood of message receivers inaccurately recalling the veracity of the information presented (18). Given the ubiquity of the facts and myths format in health education materials, our objective was to ascertain the effect of exposure to varying facts and myths message formats on participant knowledge about and accurate recall of information related to influenza and influenza vaccination.

2. METHODS

A randomized controlled trial was conducted from September 2009 – March 2010 among patients with scheduled visits at a large urban academic adult primary care internal medicine practice in Chicago, Illinois. As one part of the study, we report analyses related to the effect of message condition on participant knowledge and recall. All study procedures were approved by the Institutional Review Board of Northwestern University.

2.1. Recruitment and participants

The practice uses an electronic health record (EHR) system (EpicCare, Epic Systems Corporation, Verona, Wisconsin). Queries of the EHR identified eligible patients. Patients were eligible for recruitment if they were ≥50 years of age, had an upcoming return patient visit scheduled with a physician, had no record of influenza vaccination within the past 12 months, if their physician provided consent for us to contact the patient, and if, upon patient contact, the patient did not respond that she “definitely will” get the influenza vaccination within the next 12 months.

A limited manual chart review was performed on patients identified as eligible based on the EHR query to ensure accurate exclusion of participants for criteria not readily obtained by query (e.g., first ever appointment with physician). All eligible patients were mailed a recruitment letter, which included a phone number to call to opt out of study recruitment. One week after the mailing, a research assistant called patients who did not opt out to assess final study eligibility, administer a verbal consent, and enroll consenting patients into the study.

2.2. Design

We used a pretest-posttest design to assess the effect of each of the four messages (described below) on patients’ knowledge related to influenza and the influenza vaccination. We also assessed their recall accuracy of the messages at posttest. At pretest participants completed a structured interview over the phone. Participants were randomized to receive one of four messages; one message (a single page flyer) was sent to each participant's address using an overnight express shipping service. This use of a known express shipper allowed us to ensure that participants received the message one week prior to their scheduled visit, and also served as a heuristic to suggest that participants open the parcel immediately. The package included a brief note that thanked participants for their time during the telephone interview and asked them to read and review the included message, a $US 2 bill, and a note reminding participants to arrive early to their scheduled appointment. The posttest was completed as a face-to-face interviewer-administered survey immediately prior to the patient's scheduled appointment. Participants were compensated $US 20 upon completion of the posttest interview.

2.3. Intervention

The intervention consisted of one of four printed message flyers. These flyers were developed based on the Centers for Disease Control and Prevention's (CDC) “Flu Vaccine Facts & Myths,”(9) which was used as our control condition. The existing flyer presented six myths about influenza and the influenza vaccination, each of which was followed by a short paragraph presenting related facts. For some of the myths presented, both facts and refutations to the myths were presented; for others, facts and additional facts (that did not necessarily directly relate to the exact myth) were presented. We opted to retain five of the original six fact-myth pairs – we eliminated the sixth pair as it was determined not to be a widespread “myth.” We retained the five other myths from the original CDC flyer; altering the related fact statements when needed to present a direct corollary to the identified myth.

We created three additional messages: (1) a “Facts Only” version (one-sided facts only message), using only facts that related to the five myths identified in the original CDC message; (2) a two-sided non-refutational “Facts and Myths” version; and (3) a two-sided refutational “Facts, Myths, and Refutations” version. The four messages were prepared using Adobe Photoshop to ensure consistency in design, color balance, and print quality. Lexile analysis indicated the text of the final messages ranged from 750L to 940L, which compares to an eighth-grade level or lower.(19-22) We completed cognitive interviews with 30 individuals to conduct a manipulation check on three of the messages (CDC Control; Facts and Myths; Facts, Myths and Refutations), to assess the extent of the contradiction between the facts and myths. Based on the cognitive interviews, one of the Fact/Myth pairs was altered to ensure participants perceived contradiction. The messages were reviewed by medical professionals to ensure accuracy of the information, and printed on heavy white paper, using a blue and white color scheme with black accents. The “sender” of the message was altered so that all four message formats appeared to come from “IPC, the Influenza Prevention Committee” and a tag line at the bottom stating “For more information, talk to your healthcare provider.” Figure 1 provides an example of how one of the fact/myth pairs differed across the four conditions.

Figure 1.

Example of Message Conditions. The style of the messages was based upon the “Flu Vaccine Facts and Myths” flyer distributed by the Centers for Disease Control and Prevention(9).

2.4. Measurements and procedure

The telephone pretest included three screening items: ensure participant was 50 years or older, verify that participant had not yet received the influenza vaccination for the 2009 – 2010 influenza, and to screen out those individuals who reported that they “definitely will” get a flu shot within the next 12 months. Following screening items, interested and eligible participants completed a verbal informed consent.

2.4.1. Knowledge measurement

Interviewers asked participants to respond to a series of 15 knowledge items about influenza and the influenza vaccination. Items were presented as statements and participants were asked to respond whether the items were true, false, or if the participant was unsure (see Table 1). At posttest, participants were again asked to respond to the same 15 knowledge items.

TABLE 1.

Knowledge items

Knowledge item (accurate response)
The flu is a serious disease (T)
Most of the people who die from the flu are younger than 65 (F)
More people die from HIV/AIDS each year than from the flu (F)
The flu shot can give someone the flu (F)
The flu shot contains a live virus (F)
The flu shot can protect people against the flu (T)
The flu virus changes every year (T)
Getting the flu shot is a person's best protection against the flu (T)
The side effects of the flu shot are worse than the flu (F)
Someone might have a sore arm after getting the flu shot (T)
People my age are at risk for getting the flu (T)
Only people 65 and older should get the flu shot (F)
The flu is spread through the air (T)
October and November are the best times to get the flu shot (T)*
People can get the flu shot in December or later (T)

^*Note that as of the time of the study (September 2009 – March 2010), the CDC recommended that the best time to get the flu shot was in October or November.

2.4.2. Sociodemographics

At the conclusion of the telephone pretest, sociodemographic data, including participant age, gender, race/ethnicity, education, income, marital status, self-reported health status, and insurance status were collected via self-report.

2.4.3. Recall of message format

At posttest, participants were asked if they had received the message sent to them, and if so, had they read the message and how many times they had done so. Participants who reported reading the message were read a series of eight statements and asked to recall the format of the message, specifically: if each of the statements had been (1) presented as a fact, (2) presented as a myth, (3) presented, but I don't recall if it was a fact or a myth, or (4) not presented in the message. To reduce bias, we created paired items for seven of the recall items (see Table 2), such that some participants were asked to respond to a fact (e.g., The flu shot can protect you against the flu) or to a myth (e.g., The flu shot does not work). Participants randomly received one of the two pairs of statements during the interview, regardless of message condition.

TABLE 2.

Paired Message Recall Items

Fact	Myth
Influenza (also called “the flu”) is a serious disease.	The flu isn't a serious disease.
The flu shot cannot cause the flu.	The flu shot can cause the flu.
Getting the vaccine is your best protection against this disease.	The flu shot does not work.
The flu itself is much worse than the side effects of the flu shot.	The side effects are worse than the flu.
People of all ages should get a flu vaccine.	Only older people need a flu vaccine.
The flu will not be cured by antibiotics.	Antibiotics can cure the flu.
Everyone needs to be vaccinated against influenza.	People who have never had the flu don't need to be vaccinated against influenza.

Adults and children with conditions like asthma, diabetes, heart disease, and kidney diseases need to get a flu shot.

2.5. Analysis

A total knowledge score (range 0 – 15) was computed for each participant at pretest and posttest. Participants received a “point” for each correct answer; responses of “don't know/not sure” were counted incorrect. Pretest knowledge scores were analyzed using ANOVA. Posttest knowledge differences were assessed on the mean of the total knowledge scores for each of the message conditions using analysis of covariance, with pretest knowledge as a covariate and message format as the independent variable. Bonferroni post-hoc analyses were used to compare adjusted means.

For these analyses, we focused on recall accuracy, by summing the number of statements to which a participant provided the accurate response (possible range 0 – 8). If a participant provided an incorrect response (e.g., believing a statement that had been presented as a myth in the message to have been presented as a fact or vice versa, or believing a statement to have been presented when it had not), or responded that she did not recall if the statement had been presented as a fact or a myth, the participant's response was considered to be inaccurate. A mean of recall accuracy was computed for each message condition. Participants were required to complete at least five of the eight recall items in order to be included in the analysis. ANOVA was used to assess differences in recall across the four message conditions; Bonferroni post-hoc analyses were used to assess differences among the conditions.

3. RESULTS

Following the EHR query, receipt of physician consent, and a secondary chart review to assess eligibility, a total of 683 recruitment letters were mailed (see Figure 2). A total of 125 patients completed the pretest and were randomized to one of four message conditions; 105 (84.0%) completed the pretest and the posttest. Mean age was 63.4 (SD=9.5); 79.0% were female; 47.6% were African American, and 42.9% were white. Table 3 provides additional detail regarding participant characteristics.

Figure 2.

Participant flow diagram.

TABLE 3.

Participant Characteristics (n = 105)

Characteristic (N = 105)
Age, mean (SD)	63.4 (9.54)
Female, %	79.0
Race/Ethnicity, %
White	42.9
African American	47.9
Hispanic/Latino	2.9
Multiracial/Other	3.9
Education
<High school	8.6
High school graduate	19.0
Some college	27.6
≥ College	43.8
Household Income (USD), %
<$20,000 per year	19.0
$20,001 - $60,000 per year	27.6
$60,001 - $100,000 per year	16.2
>$100,000 per year	25.7
Marital status, %
Married	43.8
Divorced	17.1
Widowed or separated	18.1
Never married	19.0
A member of an unmarried couple	1.9
Self-reported health status, %
Excellent	13.3
Very Good	30.5
Good	30.5
Fair	19.0
Poor	2.9
% having health insurance	97.1

3.1. Participant knowledge

We report on the 105 participants who completed both pretest and posttest; the total achievable knowledge score was 15. Participant responses ranged from 2 – 13 correct answers at pretest; a oneway ANOVA indicated no significant differences in baseline knowledge [F (3, 101)=0.64, p=ns, η² = 0.02] (see Table 4). As the small sample size may have precluded identification of any significant knowledge variation at baseline, the posttest analyses were adjusted for pretest knowledge. At posttest, participant responses ranged from 4 – 14 correct answers. Using pretest knowledge as a covariate in univariate tests, analyses revealed a significant effect of condition on posttest knowledge score [F(3,100) = 3.76, p < 0.02, η² = 0.07]).

TABLE 4.

Total knowledge scores at pretest and postest, by message condition

Total Knowledge Score (out of 15)	Pretest M (SD)	Posttest M (SD)	Average Δ	p value
Full sample (N = 105)	8.82 (2.31)	10.33 (2.19)	1.51	<0.001
Facts Only Condition (N = 29)	8.97 (2.44)	9.69 (2.49)	0.72	<0.05
Facts and Myths Condition (N = 23)	8.39 (2.48)	9.86 (1.87)	1.48	<0.05
Facts, Myths, and Refutations Condition (N = 27)	9.22 (2.22)	10.70 (2.05)	1.48	<0.01
CDC Control Condition (N = 26)	8.62 (2.12)	11.08 (2.04)	2.46	<0.001

Results indicated that although all participants’ knowledge scores increased significantly following receipt of a message (p <0.05 for all), after using the Bonferroni adjustment for multiple comparisons, those exposed to the CDC Control message had a higher knowledge score at posttest (adjusted mean=11.18) than those in the Facts Only condition (adjusted mean 9.61, p<0.01); no other significant differences emerged.

3.2. Participant recall

At posttest, 103 participants reported receiving the mailing. Of those, 88 (85.4%) reported looking at the message. These participants were asked to complete recall items; 81 completed all eight recall items, six participants completed seven items, one completed five items and one completed zero recall items. All participants except the one who did not complete any recall items were included in the analyses. A missing response to a recall item was considered to be an incorrect answer. Overall, participants accurately recalled a mean of 4.49 (sd = 1.98) of eight statements. An ANOVA demonstrated significant differences in recall accuracy by condition [F(3, 83)=7.74, p<.001, η² = 0.22; see Figure 3]. Bonferroni post-hoc analysis revealed significant differences in recall accuracy between the Fact Only and Facts, Myths, and Refutations conditions (p< 0.001) and the Facts Only and CDC Control (p< 0.05) conditions; the difference in recall accuracy approached significance between the Facts Only and Facts and Myths conditions (p = 0.07). No other significant differences emerged between message conditions for recall accuracy.

Figure 3. Participant Recall (n = 87).

Participant recall accuracy of information presented; eight recall items were presented.

4. DISCUSSION AND CONCLUSION

4.1. Discussion

This study focused on assessing the effects on knowledge and recall accuracy of presenting patients with varying types of “facts and myths” message formats in a printed flyer about influenza and the influenza vaccination. Previous research by Skurnik and colleagues,(11,12) suggested that caution should be exhibited when using facts and myths message formats for educational purposes, particularly among older adults. More recent research by Lewandowsky, Ecker, Seifert, Schwarz and Cook (2012) also discusses necessary cautions when trying to “debunk” misinformation (18). The lesson from this body of research is to be wary when presenting anything except the factual information one wishes to convey.

The application of message-sidedness research (14-17) to printed health messages is novel; this study was designed to assess and test those predictions. In this study, regardless of the message format received, all participants increased their knowledge score from pretest to posttest, as expected. These results suggest that receipt of printed messages can increase one's knowledge, regardless of message format. As the CDC Control condition was more effective in increasing participant knowledge than the Facts Only format, it appears that presenting individuals with both facts and myths is not detrimental to their knowledge acquisition. As no other significant differences appeared, our findings also are not consistent with the predictions of the message-sided research, that a two-sided refutational message (Facts, Myths, and Refutations) will be more effective than a one-sided message (Facts Only), which will be more effective than a two-sided non-refutational message (Facts and Myths). It is possible that the presentation of facts and myths in one message may engage the receiver in a way that a Facts Only message cannot, a possibility bolstered by the fact that participants exposed to the Facts Only message format reported less knowledge gain than any of the other conditions (although only significantly different from the CDC Control Message in this sample).

Even at posttest, mean overall knowledge only reached 10.33 out of a possible 15 points; suggesting that there is still room to improve knowledge. It is possible that individuals do not need to be able to respond accurately to all knowledge items about influenza and influenza vaccination that are presented. Future explorations into the use of message formats should identify exactly what knowledge about influenza and influenza vaccination is most meaningful for decisions related to vaccination intention, and ultimately vaccination behavior.

When we seek to design public health education materials for a broad audience, it is critical to understand what information the receivers will retain after exposure to these messages. This study allowed us to assess recall accuracy among our participants, who each received a printed message a week prior to the posttest. We cannot ascertain exactly when each participant read the message, so we cannot state that our time horizon was exactly one week, but rather, up to one week. As participants in the Facts Only condition received a flyer that included five facts but no myth statements, they may have performed inferiorly on the recall task as they, unlike participants in the other three conditions, had never been exposed to any of the myths. However, these participants were provided the opportunity to choose the response option of “not presented;” the correct answer for each myth recall item they received. Results of previous studies may have been more affected by participant beliefs and attitudes as participants were asked to indicate if they believed various statements to be “true” or “false.” By specifically asking participants to state whether a statement had been presented as a fact, presented as a myth, presented with an unknown truth value (participant did not remember truth value) or not presented, our study more accurately tested a patient's accuracy of recall of the information presented. Our findings that both the Facts, Myths and Refutations condition and the CDC Control condition produced more accurate recall than the Facts Only condition suggest that presentation of myths are not counterproductive to accurate recall. However, our results related to recall accuracy are not exactly aligned with predictions made by message sidedness research. Although the versions of the messages that included facts, myths, and refutations produced greater recall in our participants, the Facts Only condition (one-sided message) resulted in the least accurate recall. These results appear to suggest that providing variation in what is presented (i.e., presenting both facts and myths in some format) may increase the accuracy of receiver recall. We acknowledge the accurate recall was relatively low overall. In part, this outcome may have resulted from our experimental materials. Lewandowsky et al. (2012) argue that to help overcome the misinformation effect, explicit warnings that information to be provided is inaccurate or misleading should be given prior to exposure to the misinformation.(18) Our materials did not include an explicit warning but instead relied upon refutation and/or labeling to serve as an implicit warning against misinformation. Although producing differences in accurate recall, our manipulations might have a greater effect had we also included a detailed warning.

This study has some limitations. The study was conducted at a single institution in the United States, which, in addition to the small sample size and strict eligibility criteria, limits the generalizability of these results. Study recruitment was severely affected by the emergence of the H1N1 influenza subtype, which was the most common cause of influenza in 2009. As the World Health Organization declared this new strain to be pandemic in June 2009,(23) the population's attention to influenza increased dramatically during the course of the study. Hence, a number of patients who had refused seasonal influenza vaccine in the past opted to accept the seasonal vaccination that year, reducing the number of eligible participants.

Participant knowledge regarding influenza and influenza vaccination is likely related to their attitudes; the analyses presented here do not include any effect of participants’ attitudes toward influenza and influenza vaccination. Our results related to recall may not be generalizable to other recall time-horizons; , we cannot assess how recall may change over time. As our recall horizon was of a longer duration than studies recommending against the use of both facts and myths (11,12), it may provide a more accurate representation of recall as individuals often are exposed to public health materials through the media days, weeks, or months prior to the opportunity to take action on that health issue (e.g., physician's appointment).

We conducted an intent-to-treat analysis for the knowledge items, using all 105 participants who responded to both the pretest and posttest knowledge items. Only 88 of those 105 participants reported both receiving and reading the printed messages, ergo some individuals in each of the four conditions who responded to the knowledge items were not exposed to the messages. It was our original intention for the primary outcome of our study to be the effect of the four messages on influenza vaccination behavior; however, our behavioral results as well as participant intention were severely constrained by our small sample size and are not reported here. Despite these limitations, our results strongly demonstrate that presentation of facts on their own resulted in less accurate recall and less knowledge gain than other conditions that included both facts and myths.

4.2. Conclusion

Receipt of a printed message increased knowledge among all participants, but the existing CDC Control message, which includes facts, myths, and evidence to counteract the myths was more effective in increasing knowledge among participants than was the Facts Only message. Further, participants receiving the Facts Only format demonstrated lower recall accuracy than all other message formats, suggesting that merely presenting individuals with factual information does not automatically result in increased knowledge nor assure recall accuracy. Understanding how printed messages affect patient knowledge, what information receivers will retain, as well as receivers’ recall of the veracity, or “truth value,” of information presented in messages is critical when designing educational materials. Our study demonstrates that when message receivers are provided additional evidence, they may be able to more accurately distinguish facts from myths. Presenting participants with alternatives that included facts and myths resulted in less than ideal recall, yet recall was significantly better than for a message presenting only facts.

4.3. Practice Implications

This study found no evidence to suggest that presentation of facts and myths in a message is counterproductive to recall accuracy or to one's ability to gain knowledge regarding a specific health topic. These results suggest that message designers, as well as practitioners, should not automatically shy away from identifying and addressing the numerous existing myths about preventive health services, such as wide-spread myths about influenza and the influenza vaccination. Recognizing and confronting these myths head-on may serve to increase both patients’ knowledge about a health topic, as well as their ability to accurately distinguish and remember both mythical and factual information when exposed to it at a later time.