The FDA’s Message Testing: Putting Health Literacy Advice into Practice

Abstract

Acting on health literacy principles in a large agency requires not only knowledge of the research base, but also creative work to implement the concepts in practice. Sound scientific advice needs practical development, shaped for the specific working environment. For example, the U.S. Food and Drug Administration (FDA) needs to test communications with intended audiences, but must work within constraints including:

• Widely varied audiences

• Significant time pressure

• Complying with multiple reviews designed to protect potential message testers

Testing messages with the intended audiences is a basic communications responsibility, not just an option. Yet, how to do the testing can be challenging. The FDA’s experience suggests two practical approaches for user testing:

• Internal message testing with a network of employee volunteers

• External message testing with consumer panelists

The report briefly explains how the FDA assesses some public communication internally and externally to attain insights about a target audience or a health message as well as discover how a communication might be modified to improve its usability by an intended audience. The report suggests internal or external message testing is superior to controls (no testing) and such testing can be accomplished by a large governmental agency embedded within a complex regulatory environment.

1. Introduction

The U.S. Food and Drug Administration’s (FDA) mission is to protect and promote the public’s health by regulating food, tobacco, human and veterinary drugs, biological products, and medical devices; a mission supported by communicating information about these products to lay audiences and medical professionals [1]. The FDA provides clinical and public health information, which is critical for consumers, patients, and health care professionals to make informed decisions about when and how to use FDA-regulated products.

However, the FDA’s biomedical information is most useful when it is understandable to its intended audience. Communication science, including its special fields of risk communication and health literacy, has developed a solid body of evidence about how to maximize clarity and minimize cognitive barriers for more understandable messages. In addition to following the principles of good communication during message creation, pretesting messages with intended audience members is critical. Testing a health communication message with the intended audience may uncover facets that are unclear or distracting.

The FDA’s Risk Communication Advisory Committee (RCAC), a panel of communication experts external to the agency, addressed this point in their first committee meeting in 2008, and in subsequent gatherings [2]. In turn, the FDA acted on the advisory committee recommendation to find a time-and cost-efficient way to provide some testing of communication materials before releasing them, which is illustrated in the examples below.

The intended audiences for the FDA’s health messages vary widely, from the U.S. general population (who purchase food, nonprescription drugs, or products for animals) to highly specialized health care professionals. As a result, the FDA’s messages must be designed for their intended audience and tested with members of the intended audience to ensure the messages are understandable and usable.

The FDA’s message testing protocols use in-depth interviews with a small sample of individuals. Then, a qualitative analysis is applied to identify themes in the responses, which are used to develop recommendations to improve messages. Each project is different, calling for specific interview guides. An individualized, semi-structured approach enables interviewers to explore responses in context and reveal what may be unclear or may not resonate, and why.

Prior research suggests “saturation,” or the point at which interview results become repetitive so more interviews reveal little new information, can occur with as few as twelve interviews when the intended audience is fairly homogeneous [3]. The FDA’s experience likewise confirms interviewing a small sample size can lead to saturation. Therefore, to the extent that messages testers and the intended audience share characteristics, the FDA’s small samples help obtain useful feedback to improve messages for intended audiences.

2. Findings And Results

As noted, while each message testing project is a unique quality improvement effort, the FDA’s testing repeatedly suggests where to provide changes to improve message clarity and audience usability. To obtain feedback, the FDA pursues both internal and external message testing strategies. An example of each strategy is provided immediately below.

Example 1: The first example comes from a project tested through the FDA’s Internal Message Testing Network, which is explained below. In this case, the FDA’s Office of External Affairs requested a review of two draft versions of a graphic designed to illustrate a Consumer Update article regarding over-the-counter (nonprescription) cryogenic wart removers [4]. The key message for consumers was the FDA received reports of some products catching fire at home -- without an identifiable ignition source. The Consumer Update’s call to action was intended to instruct readers on safer use, along with suggestions for alternative treatments. The graphic’s intent was to summarize its key message.

A need for audience testing quickly surfaced during the message’s development period when the project’s graphic designers debated where to place a flame, and whether to show the applicator that came with the product.

FDA employee volunteer message testers, then, provided feedback on items that were initially overlooked. The employee volunteer testers suggested:

• The terms “OTC” and “Cryogenic” were problematic in the graphic, even when defined in the text of the article

• The slash made the graphic seem disjointed

• The image on the left gave the wrong impression that consumers should spray the product directly onto the wart

• The image on the right looked too much like a cotton swab and people mistakenly thought more than one type of product was depicted

• An image of a wart needed to be included to better communicate the purpose of the product

Thanks to feedback from FDA employee volunteer testers, message designers redesigned the visual, creating a simpler and more direct image. They replaced the terms “OTC” and “Cryogenic” with frosty “Freeze Gone,” removed the slash, and kept the wart image.

Example 2: A second example is derived from an external consumer panel that received an FDA contract. The draft communication to be tested was an opioid analgesic patient counseling guide [5,7]. The guide outlined the risks of taking opioid analgesics and explained how to take them safely. The target audience for testing was mostly women, adults with lower health literacy, and persons with a range of opioid user experiences. The 15 participants recruited as testers included:

• Ten women and five men

• A diverse mix of races from regions all around the country

• Testers with a high school education or less

• Testers mostly aged 45-64

• Seven persons who were opioids users and eight people who had never taken opioids before.

All the participants found the counseling guide helpful and useful. The participants understood the guide’s core call to action, which encouraged consumers to talk to their healthcare provider about opioid use and abuse. The participants also made recommendations for clearer language and suggested:

• Deleting medical terms or replacing them with plainer language followed by medical terms in parentheses

• Providing a clearer explanation of the difference between “tolerance” and “addiction”

• Revising the description of naloxone to explain more clearly how to use it

• Consider brighter or more varied colors to highlight text related to dangers or risks.

Following this feedback, the creators of the opioid analgesic patient counseling guide simplified the language and used color more liberally. The FDA also assessed the draft guide using the U.S. Center for Disease Controls and Prevention’s (CDC) Clear Communication Index [6]. After the editing revisions above, the guide’s Index score improved ten points compared to the pre-testing version. Figure 2 above illustrates the before-and-after-versions of a section of the patient counseling guide [7].

Figure 2.

3. Background Information: The FDA’s Internal And External Message Testing Projects

3.1. Internal Message Testing

The FDA’s Internal Message Testing initiative began with a pilot project in 2009 as a proof of concept within the agency. The initial concept was to: develop an internal network of volunteers willing to devote up to one hour on message testing; conduct message testing interviews with small samples of those volunteers; and report recommendations to the message developer based on themes from interview discussions. The pilot results were encouraging: some FDA staffers were eager to volunteer as message testers in sufficient numbers to form the network; supervisors were willing to have employees take an hour to do message testing of a communication from diverse FDA divisions; and staff across the FDA reported they enjoyed an opportunity to contribute to mission-critical quality improvement.

Volunteers:

To participate, FDA staff volunteers first completed a form to provide basic demographic data including year of birth, gender, race, ethnicity, education level, professional training, years at the FDA, languages(s) spoken, and caretaking for children, elders, or animals. From the start, it was important to solicit new volunteers to replace those lost to turnover, to avoid overburdening FDA staff, and to prevent volunteers from becoming so experienced in message testing that they were less representative of the intended audience. Over time, the most effective internal recruitment tool became a simple and no-cost insertion of a request for volunteers within the FDA’s agency-wide daily news announcements.

Currently, the Internal Message Testing Network has about 900 volunteers. In terms of possible confounding variables among employee participants, the FDA is large enough (approximately 18,000 employees) that staffers in one office often are unfamiliar with the products and regulatory issues in some other areas of the agency. To avoid additional confounds, the volunteers most often sought for testing are those who recently have joined the FDA and do not have four years of college education. The latter demographic characteristics are similar to some of the FDA’s intended audiences.

In contrast, the FDA also targets messages to health care professionals, and this network includes volunteers who are pharmacists, nurses, and physicians.

Coordinators:

The Internal Message Testing Network’s coordinators, who are social scientists, have several tasks. They maintain the FDA’s network, recruit new volunteers, update information, and use the database to identify the proper participants for a project. They also conduct, analyze, and report findings from individual interviews.

Experience:

The FDA’s Internal Message Testing Network process assessed more than 60 projects from 2010 to 2018. While most assessments focused on safety communications or other web content, other examples included proposed label changes as well as guidance to industry, forms, pamphlets, and posters. The suggestions from the Internal Message Testing Network helped many of the FDA’s developers tailor messages towards their intended audience. Besides increased understandability, participant suggestions helped enhance graphics for clarity and visual appeal, and rearranged web content for user-friendliness.

Strengths:

Overall, the FDA’s no-cost Internal Message Testing Network has improved the speed, ease, and cost of assessments by facilitating a feedback channel between message creators and employee volunteer message testers.

Limitations:

The staff volunteers in the FDA’s Internal Message Testing Network program often are not ideally representative of a health message’s target audience. Compared to the general public, FDA staff members have more years of education as well as more knowledge about the agency and regulated products. Also, a comprehensive evaluation of the program’s effectiveness is challenging because post-implementation feedback is not always available. Yet, the frequency of requests within the agency for internal message testing suggest the process and its results improve health messages.

3.2. External Message Testing

Parallel to the Internal Message Testing Network, the FDA has explored external message testing options, including online user panels. Opt-in, incentivized consumer panels have emerged as an important strategy to collect consumer information and responses. In addition, reputable vendors who can provide access to large diverse panels enable the FDA to select participants based on geography, age, race, ethnicity, as well as other pertinent demographic characteristics representative of specific intended audiences.

In 2017, the FDA awarded a contract to conduct rapid message testing with small samples drawn from an external online panel of more than two million U.S. consumers. While each of these message testing projects fit the patterns outlined in Box 1, the larger pool of potential testers provided participants who were more representative of intended audiences. The FDA’s social scientists also worked with a contractor on the selection criteria for participants and developed subsequent interviews with panelists.

BOX 1: MESSAGE TESTING BASICS.

Who –

That is, who is the message for? The creator of the communication should have characteristics of the intended audience in mind. Message testing coordinators then identify from a database of volunteers approximately twelve testers who share characteristics with the intended audience.

When –

Testing a communication should occur when the draft is solid but not yet final. The message itself should be well-formed and approved so it will not become obsolete, but the overall communication should be subj ect to change in response to the testing results.

What –

The coordinators develop a semi-structured interview guide to ensure relevant feedback about the draft communication. They conduct in-depth, one-on-one interviews, then compile a report including quotes, findings, and recommendations.

How –

Interviews are typically 30 to 45 minutes long, can be face-to-face, online, or by telephone. Coordinators analyze and organize the responses to look for qualitative themes, using either a coding matrix or NVivo software. At least two coordinators do the analysis to ensure consistency.

Why –

Individual interviews allow probing for deeper understanding of comprehension and language by asking participants to paraphrase items, discuss thoughts or emotions that come to mind, and offer suggestions for improvement. Every message testing project has yielded suggestions for improvement, sometimes in surprising ways.

External message testing helps address the limitations of internal message testing noted above. A large panel of potential respondents often yields samples that resemble target audiences, which potentially improves the reliability of the findings.

The protocol also calls for FDA coordinators to compare the initial health communication with its post-evaluation final version, using the Centers for Disease Control’s Clear Communication Index (to facilitate consistent assessments) [6]. Since the CDC’s Clear Communication Index is based on health literacy principles, ‘before and after’ comparisons suggest the revised messages are easier to read or understand.

Experience:

Currently, the FDA contracts for ten message testing projects per year. As projected, the evaluations that utilize a larger external panel have facilitated the participation of: racial and ethnic minorities; participants with lower education and socioeconomic status; persons with specific chronic diseases; participants with roles such as a caregiver or primary grocery shopper; as well as specific age groups. Some recently evaluated initiatives included: several types of brochures; public service announcements to consumers and patients; and tools for health care professionals, such as patient-provider agreements and counseling guides.

Strengths:

The primary strength of an external panel approach has been improved access to diverse participants that more closely resemble specific intended audiences. External testing also has increased the FDA’s capacity to support message developers.

Limitations:

External testing with a vendor requires sustained funding. Since the participant/panelists are not FDA employees, the materials must be screened to be appropriate for the public. Additionally, while the goal is to connect with hard-to-reach populations online, many participants greatly prefer phone-only interviews. When participants prefer phone interviews, a contractor can send materials by overnight mail and then conduct an interview over the phone.

4. Discussion: The FDA’s Communications Responsibilities And Mission

The FDA’s public communication responsibilities cover a spectrum from proactively developed communications campaigns, such as ‘Real Cost’ from the Center for Tobacco Products, to reactive and time-sensitive announcements of new and emerging information [8]. The FDA’s intended audiences range from members of the general public to highly specialized health care practitioners. The science of risk communication provides principles to facilitate communication to each group. To make the state of the science and its practical implications more accessible, the FDA’s Risk Communication Advisory Committee (RCAC) collaborated on a report that collected scientific advice for government communicators on special topics, such as quantitative communication and communication across the life span [9]. Most notably, the RCAC panel’s report consistently recommended testing and evaluating communications to understand an intended audience’s perspective.

4.1. Addressing The Common Knowledge Effect

The critical importance of message testing with the intended audience is a recurrent theme within risk communication because it helps overcome communicators inadvertent, tacit assumptions that can interfere with the understanding of health messages. As Baruch Fischhoff (the panel’s chair) noted: “[P]eople exaggerate how well they understand others’ perspectives. This general tendency, perhaps familiar to most people in their everyday communication has many expressions. One is the common knowledge effect: people exaggerate how much of their knowledge is shared by others” [9, p. 42].

The FDA’s internal and external message testing projects directly address the common knowledge effect: the FDA seeks feedback on a draft communication from a sample of participants similar to the intended audience, and their comments show how message developers could improve the communication. Similarly, the suggested improvements also demonstrate where the draft reflects some assumptions about what should be clear or engaging (which sometimes needs modification). For instance, in example 1, message developers thought the terms ‘OTC’ and ‘Cryogenic’ would be clear once explained in the text, but message testing results showed their assumptions were incorrect.

4.2. Programmatic Constraints

The FDA’s initial message testing implementation has continued in spite of some practical barriers to its quick implementation. Institutionally, it is difficult to initiate optimal external message testing without sufficient budgetary support. Also, the FDA’s responsibilities to provide immediate information about an urgent public health risk, such as foodborne illness, can compress the timeline for message development.

Likewise, it can be time consuming to identify persons with lower health literacy and other specific audience characteristics to participate in message testing.

Also in the U.S., an array of regulations limit the capacity of federal agencies to quickly set up an external consumer testing panel. For example, U.S. laws require: securing the informed consent of participants; protecting the privacy of public participants; as well as the FDA’s required compliance with the U.S. federal government’s Paperwork Reduction Act (PRA).

The PRA seeks to shield the American public from the burden of repetitive federal government information collections. Since the PRA requires a federal agency to pursue public comment and obtain approval from the U.S. Office of Management and Budget before collecting information from ten or more persons, the process can add considerable time to an information collection project. Moreover, the FDA must comply with U.S. regulations for the protection of human subjects, which requires an Institutional Review Board review and approval of research [10]. Although these regulations are specific to the U.S. federal government; there may be similar rules in institutional or legal contexts in other nations.

To minimize the anticipated delays, the FDA works collaboratively and proactively with persons who supervise human subjects protection as well as federal PRA offices.

More positively, despite the delays, the FDA staff usually completes each external testing project within six weeks. In contrast, the internal message testing approach is generally classified as a quality improvement initiative (rather than human subjects research), which shortens the implementation process to about four weeks.

5. Conclusion

The authors suggest the message testing projects using the FDA’s Internal Message Testing Network or external panels consistently have yielded practical health communication feedback and prevented possible public message missteps. The authors suggest the FDA’s internal and external message testing especially has yielded contextual insights about a target audience or a message and sometimes has provided constructive feedback regarding unintentionally mistaken assumptions about an audience or a specific health topic. In summary, the FDA’s experience to date suggests internal or external health message testing is superior to no testing and assessments can be accomplished by a large governmental agency embedded within a complex regulatory environment.

The authors note that both internal and external message processes are examples of applied health literacy practices. In general, the authors recommend a draft communication for testing initially needs to be created within an enduring public communication framework of: a main message; language appropriate to a target audience; and a layout that reinforces the main message. Even then, message creators sometimes incorporate tacit assumptions, which is why empirical testing with members of an intended audience is strongly recommended. No matter how carefully a communication is created and crafted, it is only as good as the message that is actually received and understood by an intended audience, which suggests a foundational need for audience involvement and testing. Message testing also is an extension of the principles to make health information more understandable and involve intended audiences in the creation of health communications.

Finally, the authors suggest internal message testing may be feasible within agencies that are large enough to have employees who work in diverse health topic areas. In addition, the authors suggest it is helpful to develop a large volunteer network; online consumer panels are increasingly available if agencies invest funds in this potentially helpful resource.

For health literacy practitioners, the FDA’s experience strongly suggests testing is preferable to no testing, and some testing may be possible internally even without dedicated funding (given a workforce large enough to include members who are not directly familiar with the topic of a draft communication). That said, if some funding to access to large external panels is available, then, health communication processes will benefit by testing messages with participants more similar to the specific intended audience.

Figure 1.