User Testing of the NCI’s ASA24 Respondent Nutrition Report for total nutrient intake from foods, beverages, and dietary supplements: A qualitative study

Abstract

Background:

Approximately 75% of the U.S. population use dietary supplements to boost their nutrient intake beyond what they obtain from foods and drinks. Therefore, it is crucial to clearly communicate how dietary supplements contribute to overall nutrient intake, enabling consumers to make informed health decisions.

Objective:

The objective of this study was to test user comprehension of four aspects of nutrient intake displayed in the ASA24 Respondent Nutrition Report (Total, Target, Limit, and Warning) among adults with diverse numeracy levels.

Design:

Two-phase qualitative study using semi-structured interviews with dietary supplement users (Phase I conducted March 1-April 31, 2021, Phase II conducted June 1-July, 31 2021).

Participants/Setting:

A total of fifty dietary supplement users ≥19 years of age were recruited via a nationwide opt-in research panel for Phase I (30 participants) and II (20 participants). Participant interviews were conducted virtually.

Analyses Performed:

A deductive-to-inductive coding method was used for creation of the codebook; 20% of transcripts were double-coded with inter-rater reliability >90%. Summative content analysis assessed comprehension of reviewed concepts.

Results:

In Phase I, participants found the graphic format easier to understand than a table. Additionally, most participants believed the graphic would be easier for the general public to understand. Using participant feedback from phase I, the graphic was revised, which led to better understanding of key features of the graphic for participants in Phase II, especially for nutrients that fell below the target. These updates improved understandability among both higher and lower numeracy participants.

Conclusions:

Content for Total Nutrient Intake from Foods, Drinks, and Supplements for the ASA24’s Respondent Nutrition Report was developed, underwent usability testing, and demonstrated sufficient user comprehension. Future research can evaluate the effectiveness of the final graphic for behavior change with different populations or conduct future testing of the RNR, with a focus on individuals with lower numeracy scores.

INTRODUCTION

Nearly three-quarters of American adults in 2024 reported using dietary supplements¹ and a quarter of those over age 60 years report use of four or more types of supplements.² Dietary supplements are beneficial when nutrient needs exceed what can be obtained solely from foods, and they can be used to improve overall health or manage certain health conditions (e.g., iron supplements for people with anemia).³ Consumers, however, may not be aware of recommended daily intake values based on their age and sex or potential side effects that can occur with high doses of dietary supplements.³

Dietary Reference Intakes (DRIs) are used to identify nutrient intake needs for the U.S. population. DRIs include reference values specific to age and sex groups.⁴ However, DRI values are often intended for use by health professionals rather than consumers because they require a high level of nutrition expertise to understand their application.⁴ Many consumers struggle with understanding the increasing volume of nutrition-related content from online sources and in printed materials.⁵ This challenge is especially true for the estimated 62.7 million U.S. adults with lower numeracy skills,⁶ who may find it difficult to understand and apply DRI values.

Within the context of health literacy, numeracy describes a person’s ability to access, understand, compute, and communicate numerical data in health materials, visual displays of risk, and within patient-provider communications.⁷

Given the complexity of dietary reference intakes (e.g., recommended dietary allowance, adequate intake, tolerable upper intake level), applying a user-centered design approach⁸ is essential for developing nutrition education materials that are accurate, clear, and actionable. Obtaining feedback from intended audiences through user testing is a well-established strategy to improve comprehension of nutrition materials and ensure that messages achieve their intended effect.⁹ This process enables consumers to make informed food and drink choices and potentially improve their diet quality and health.¹⁰

The National Cancer Institute (NCI)’s Automated Self-Administered 24-Hour Dietary Assessment Tool (ASA24^®) is a free, automated, web-based tool for collecting food records or 24-hour dietary recalls. The ASA24 is a registered trademark of the U.S. Department of Health and Human Services.¹¹ The tool gives feedback via a Respondent Nutrition Report (RNR), which shows how respondent diets compare with country-specific dietary guidance.¹¹ The tool may be used for a variety of purposes, such as collection of 24-hour recalls from patients or nutrition students learning about dietary assessment methods.

In 2019 usability testing of the RNR, participants had difficulty identifying key information in a table on nutrient intake from food, drinks, and supplements. Specifically, participants found the 2019 table difficult to comprehend and were unable to compute calculations, such as total nutrient intake (Lerman, Kaefer, Subar, Herrick & Wambogo, 2020, unpublished data). Consequently, the entire table was not included in the revised RNR that was released in September 2020 (Lerman, Kaefer, Subar, Herrick & Wambogo, 2020, unpublished data).

Based on the results from 2019, this two-phase qualitative study aimed to test user understanding of a simplified numeric table and a newly designed graphic depicting total nutrient intake from foods, drinks, and dietary supplements for inclusion in the NCI’s ASA24 RNR for participants who report supplement use. The primary objective of this study was to test user understanding, among participants with a diverse range of numeracy, on the four aspects of nutrient intake (1) Total: the concept of total nutrient intake from foods, drinks, and supplements; 2) Target: how total nutrient intake compares to national nutritional recommendations; 3) Limit: when the maximum daily nutrient targets are met; and 4) Warning: a “warning” feature used to indicate excessive intakes that could pose potential health risks. The second objective was to evaluate which version, the table or the figure, participants of diverse numeracy scores better understood and subjectively preferred. Finally, a third objective was to examine participants’ intention to change behavior.

MATERIALS AND METHODS

Study Overview

For this study, two phases of user testing (Phase I: March 1-April 31, 2021 and Phase II: June 1-July 31, 2021) were conducted with separate participant groups to determine how to best display and clearly communicate total nutrient intake as part of the ASA24 RNR.

The NCI’s ASA24 is a fully automated, web-based, self-administered 24-hour dietary assessment tool. The U.S version is available in English and Spanish and is publicly available at no cost to researchers, clinicians, and educators.¹¹ The tool allows for collection of multiple dietary recalls or records in both small and large-scale nutrition research.

The RNR provides feedback to participants on how their reported intake compares to U.S. dietary guidelines for calories, food groups, and nutrient intakes.¹² For RNRs generated by the U.S. version of ASA24, the calorie and food group recommendations come from the Dietary Guidelines for Americans and the nutrient requirements from the Institute of Medicine, National Academy of Sciences.^13–15 It consists of five major sections: Total Calorie Consumption, Daily Food Group Recommendations, Nutrients and Foods to Limit, Nutrient Intake from Foods and Drinks, and Vitamins, Minerals, and Nutrient Intake from Foods and Drinks.¹²

This study focused on the “Nutrient Intake from Foods and Drinks” section of the ASA24 RNR. At the time of user testing, the RNR only included nutrient intake from foods and drinks. As a result, the displayed values did not represent total nutrient intakes for individuals who consumed dietary supplements.

Participants and Recruitment

A convenience sample of 50 dietary supplement users were recruited via a nationwide opt-in research panel for Phase I (n=30) and new participants were recruited for Phase II (n=20). Participants were balanced for gender identity, age, race/Hispanic origin, and subjective numeracy.^16–18 Both Phases I and II used virtual data collection methods due to the COVID-19 pandemic. Therefore, eligibility required reliable access to a computer or laptop with internet access. Participants had to be aged 19 years or older and had to use any type of a dietary supplement. Participants were ineligible if they were federal employees; worked in fields related to nutrition, graphic design, or web development; did not use the internet to find health information; or were not fluent in English.

Participants were contacted via email about the study by Research America,¹⁹ a market research recruitment company. Participants were asked to complete a web-based screening survey to determine eligibility. During screening, self-report data were collected on subjective numeracy and other demographics, including age (19–30, 31–50, 51–70), race (American Indian or Alaska Native, Asian, Black or African American, Native Hawaiian or other Pacific Islander, Caucasian/White, other), Hispanic origin (yes, no), gender identity (female, male, transgender, gender non-conforming, other), and education (High school diploma/GED or less, some college or college degree, some graduate school or graduate school degree). Recruitment efforts aimed to balance participants by subjective numeracy, age, and gender identity. Dietary supplement use was required and determined by an affirmative response to binary (yes/no) questions about vitamin, mineral, or other dietary supplement use in the past 30 days.²⁰

Research America rescreened participants via telephone for eligibility to ensure they could participate (i.e., need for a computer and internet). In Phase I, 30 eligible participants gave oral consent, and 25 completed the virtual interview. Five participants who gave oral consent were unable to complete the interview process; therefore, Research America recruited another 5 participants for a total of 30, whom all gave oral consent and completed the virtual interview. In Phase II, 20 eligible participants gave oral consent, and all 20 participants completed the virtual interview. A systematic review on assessing sample sizes in qualitative research found that 9–17 interviews is enough to reach saturation.²¹ However, the sample size was increased because this study had an eye-tracking component (data not reported in this paper). Participants in Phase I were assigned identifiers 01–30, and participants in Phase II were assigned identifiers 101–120.

A three-item version of the Subjective Numeracy Scale (SNS) was used to determine participant numeracy levels. Scores can range from 3–18, with higher scores representing higher subjective numeracy.^16–18 A cut-off score of 16 and under (approximately 50% of Phase I sample) denotes lower subjective numeracy, compared to a higher subjective numeracy score of 17–18. For example, 47% of the sample in Phase I had a score of 17–18, so they were considered to have higher subjective numeracy. Zikmund and colleagues¹⁸ examined subjective numeracy in terciles; however, in this study, it was switched to approximately 50% due to a smaller sample size and the term “lower numeracy” was used to represent anyone with a score ≤16.

Participants were compensated with a $75 Amazon gift card following completion of the qualitative interview. This study was considered exempt for human subjects research from the National Institutes of Health Institutional Review Board [IRB#−000219].

Figure Development

This study applied the first two stages of the Stages of the Health Communications Process⁹ developed by the National Cancer Institute: (1) Planning and Strategy Development, (2) Developing and Pretesting Concepts, Messages, and Materials. The iterative and non-linear nature of the process gave the research team the flexibility to create and revise a newly designed graphic through two phases of user testing. This approach ensured that the graphic was systematically developed and refined with input from both experts and intended users prior to broader implementation.

In Stage 1, Planning and Strategy Development, internal meetings were held with eight experts at the National Cancer Institute specializing in the fields of nutrition, design, health and risk communication, and health literacy. These meetings reviewed unpublished data from 2019 user testing of a table displaying total nutrient intake from foods, drinks, and supplements, which participants found difficult to understand. Guided by this feedback, the group discussed the development of a newly designed figure aimed at improving the clarity and usability of this information. Initial design ideas for the figure were reviewed and discussed in detail, with feedback provided to the research team to guide the creation of an initial prototype for pretesting in Stage 2.

Stage 2, Developing and Pretesting Concepts, Messages, and Materials, began with the creation and iterative refinement of an initial prototype of the newly designed graphic, developed with input from the research team and NCI experts. The nutrients included in the table and newly designed figure were based on what was already included in the ASA24 RNR, ensuring consistency with the existing tool. The refined version of the figure was approved by NCI prior to use in user testing.

Phase I of the study constituted the first round of pretesting within Stage 2. Semi-structured interviews were conducted to assess comprehension, usability, and visual appeal of both the table and the newly designed graphic among a diverse sample of participants. Participant feedback was compiled, analyzed, and discussed by the research team to identify areas for improvement. While participants found the graphic easier to understand compared to the table format, they offered specific suggestions for further refinement. These suggestions were carefully considered for inclusion based on their alignment with project objectives and health literacy best practices,²² and revisions were subsequently incorporated into the graphic in preparation for Phase II.

Phase II constituted the second round of pretesting within Stage 2. The revised graphic was tested with a new group of participants to evaluate the effectiveness of the refinements made following Phase I. Participants reviewed the updated figure and provided feedback on its clarity, usability, and interpretability, and final refinements were made based on their responses. Together, Phases I and II reflect the iterative, user-centered pretesting process central to Stage 2 of the NCI Health Communications Process model, resulting in a refined graphic ready for broader evaluation in future research.

Data Collection: Usability Testing Procedures

Phase I

A market research recruitment company, Research America,¹⁹ emailed participants a weblink to a Qualtrics²³ page that described the study. Participants were then asked to join two members of the research team in a recorded videoconference Zoom session for an average 30-minute semi-structured qualitative interview with a trained interviewer. Interviewers introduced themselves, described the study, and confirmed consent before starting the interview. One member of the research team led the interview and the other took notes. Each interviewer (H.P., L.O., C.R., M.Z.) completed advanced training on how to design an interview guide, conduct a qualitative interview, code and analyze qualitative data using NVivo.²⁴ All interviewers were female; two have PhDs and were Cancer Prevention Post-doctoral Fellows at the time the study was conducted (H.P, L.O.). The other two were a Cancer Training Award Fellow and Health Communication Fellow with an MPH degree C.R., M.Z.). The interview guide was created by the lead author based on data from previous user testing of the ASA24 RNR and was pilot tested for clarity and understanding prior to starting the study (Supplementary Figure 1). Pilot testing included a review of the interview guide by a qualitative research expert and the completion of mock interviews with NCI fellows who self-identified as supplement users to assess question clarity and overall interview flow from the participant’s perspective.

To protect their privacy, participants were required to keep their cameras off during the session and used a participant ID instead of their name. Participants in Phase I were asked to compare the table and graphic, discuss comprehension of features in the table and graphic (i.e., target, limit), and total nutrient intake as well as format, missing information, preference, and behavioral intentions. Phase I participants were able to see both the table and newly designed figure.

Phase II

All steps in Phase II were the same as Phase I, except for the differences noted below and in the interview guide (Supplementary Figure 2). Participants in Phase II were asked to provide general feedback on the complete ASA24 report (which included the newly revised graphic), discuss comprehension of features for the graphic (i.e., total nutrient intake), total nutrient intake, and provide information on whether the graphic would change their intake behaviors. Participants in Phase II did not get to see the table from Phase I. Participants in Phase II were able to comment on updates to the graphic from participant suggestions in Phase I, which is a form of member checking.

Data Analysis

Data from both study phases were processed and analyzed using the same methodology. Upon completion of an interview, the audio recording was transcribed verbatim using NVivo software 14 (Release 1.2 for Windows).²⁴ After transcription, each interviewer reviewed their own transcripts for accuracy by listening to the audio while reading the transcript to find and correct errors. The clean transcripts were then uploaded to NVivo for analysis.

The lead author, H.P, used a deductive-to-inductive coding method to create the codebook. The interview guide was developed with questions that had objectively correct or incorrect answers. Based on these questions, we created a codebook using deductive a priori codes (e.g., limit, target, recommended range), each reflecting a correct or incorrect participant response,²⁵ allowing the research team to systematically code participant responses as correct or incorrect based on established nutrition literature.

During the analysis, additional codes emerged that were not anticipated. These new insights were coded inductively,²⁵ which allowed for the identification of new information relevant to the study objectives. The deductive-to-inductive coding approach is consistent with methods described in qualitative research, which combines question-driven a priori coding with emergent information coding.²⁵ To finalize the codebook, the lead author coded a randomly selected interview from each moderator (n=4). The study team (H.P., L.O., C.R., M.Z.) then met to review the initial coding, discuss discrepancies, and refine the codebook.

Interviews were then divided among the study team for individual coding. Data were analyzed on a rolling basis using an iterative approach. The codebook functioned as a living document, evolving through constant comparison, with new codes added or refined as needed until thematic saturation was reached. To ensure rigor, this study incorporated strategies aligned with Lincoln and Guba’s criteria for qualitative research.²⁶ Credibility was established through double-coding 20% of transcripts (Phase I n=6; Phase II n=4) to compute inter-rater reliability, which measures consistency and agreement between multiple coders.²⁷ Inter-rater reliability scores below 90% triggered team discussions to resolve coding discrepancies until a minimum agreement level of 90% was achieved. Furthermore, dependability was accomplished through maintaining an audit trail of coding decisions, codebook revisions, and analytic reflections throughout the research process.

Summative content analysis was used to identify patterns in the dataset by integrating interpretive strategies to explain the frequency of which words, phrases, or information appeared in participants’ responses.²⁸ Specifically, this approach focused participant understanding of 8 key features presented in both the table and graphic: 1) Below the target; 2) Target; 3) Limit; 4) Over the limit; 5) Total nutrient intake; 6) Warning symbol; 7) Intake from Foods and Drinks; and 8) Intake from supplements. Interview questions focused on participants’ understanding of these key features. Each question had a predefined correct answer, developed by the research team. For example, “What do you think the warning symbol means?” The correct answer needed to include a key phrase similar to the following: “Amounts that exceed the upper limit / go over the limit / caution/dangerous or adverse health effects”). Analysis involved searching for the presence of these key terms or phrases in participant responses. The number of correct responses per key feature were summed to generate a final score, based on the number of participants in each phase.

Results

Participant Characteristics

Participants in Phases I and II had comparable distributions of gender identity, race, education, and age (Table 1). In Phase I, 16 of 30 participants were female (53.3%), 15 of 30 (50.0%) were white, 7 of 30 (23.3%) had a high school diploma/GED or less, 12 of 30 (40.0%) were between 31 and 50 years old, and 16 of 30 (53.3%) had a Subjective Numeracy Score (SNS) of 16 or lower. In Phase II, 11 of 20 participants were female (55.0%), 10 of 20 (50.0%) were white, 6 of 20 (30.0%) had a high school diploma/GED or less, 8 of 20 (40.0%) were aged 31–50, and 12 of 20 (60.0%) had an SNS of 16 or lower (Table 1).

Table 1.

Demographic and Sample Characteristics of 50 Adult Participants in Two Phases of User Testing on Communicating Total Nutrient Intake from Foods, Drinks, and Supplements

Characteristics	Phase I (n=30)	Phase II (n=20)	Total (n=50)
	n (%)
Racea
AI/ANb	1 (3.3%)	1 (5.0%)	2 (4.0%)
Asian	3 (10.0%)	2 (10.0%)	5 (10.0%)
Black	6 (20.0%)	4 (20.0%)	10 (20.0%)
White	15 (50.0%)	10 (50.0%)	25 (50.0%)
Hispanic Origin	8 (26.7%)	4 (20.0%)	12 (24.0%)
Gender Identity
Female	16 (53.3%)	11 (55.0%)	27 (54.0%)
Male	14 (46.7%)	9 (45.0%)	23 (46.0%)
Gender non-conforming	0 (0%)	0 (0%)	0 (0%)
Transgender	0 (0%)	0 (0%)	0 (0%)
Education
High School/GED or Less	7 (23.3%)	6 (30.0%)	13 (26.0%)
Some College / College Degree	13 (43.3%)	8 (40.0%)	21 (42.0%)
Graduate Degree	10 (33.3%)	6 (30.0%)	16 (32.0%)
Age Range (years)
19–30	7 (23.3%)	6 (30.0%)	13 (26.0%)
31–50	12 (40.0%)	8 (40.0%)	20 (40.0%)
51–70	11 (36.7%)	6 (30.0%)	17 (34.0%)
Subjective Numeracy Scale (SNS)c
Median Score (Range)	15.5 (7–18)	16 (6–18)	16 (6–18)
6 – 10 (n, %)	3 (10.0%)	1 (5.0%)	4 (8.0%)
11 – 16 (n, %)	13 (43.3%)	11 (55.0%)	24 (48.0%)
17 – 18 (n, %)	14 (46.7%)	8 (40.0%)	22 (44.0%)

^aPercentages do not sum to 100% as participants could select more than one racial category, and some participants identified with Hispanic ethnicity only and did not select a racial category

^bAI/AN = American Indian/Alaskan Native

^cSNS is Subjective Numeracy Scale Range 3–18, Study Range 6–18¹⁶; high numeracy is 17–18 and lower is ≤16

Phase I Results

This section summarizes Phase I participants’ understanding of key features presented from a table and graphic on nutrient intake from food, drinks, and dietary supplements. We first present information on whether participants found the table or graphic format easier to understand for themselves and for the general public. Then we describe overall response accuracy across the key features, such as Target, Limit, and Total Nutrient Intake.

Easier to Understand

When asked whether the table or graphic was easier to understand, 19 of 30 (63.3%) participants in Phase I stated they comprehended the graphic better than the table (Figure 3). Approximately 56 percent of participants (n=9) with lower numeracy found the graphic easier to understand compared to the table format (Table 2).

Table 2.

Percent of Phase I user testing participants who found the Graphic or the Table easier to understand, for themselves and the general public, by participant subjective numeracy score^a

Easier to understand	High Numeracya (n=14)	Lower Numeracy (n=16)	Total (n=30)
	n (%)
Graphic
Participant	10 (71.4%)	9 (56.3%)	19 (63.3%)
Public	12 (85.7%)	11 (68.8%)	23 (76.7%)
Table
Participant	4 (28.6%)	7 (43.7%)	11 (36.7%)
Public	2 (14.3%)	5 (31.2%)	7 (23.3%)

^aSNS is Subjective Numeracy Scale: Range 3–18, Study Range 6–18;¹⁶ high numeracy is 17–18 and lower numeracy is ≤16

Participants were also asked which, the figure or the table, the general public would best understand; 23 of 30 (76.7%) participants agreed that the public would best understand the graphic, which included 68.8% of participants (n=11) with lower numeracy. Participant 11 (SNS=15) stated “in today’s society, people want things quickly and so they look at the graphic first… Like, for me, I would look at the graphics first, but I would also go back to complete the numbers on the table.” One participant found the table easier to understand simply because they were more familiar with it, though they also understood the graphic. Participant 24 (SNS =14) stated “Actually, the table format is something that I’m more used to. So, because I’m used to it, that would make it a little easier for me. But it’s not easy for me to not understand the other. I’m just more used to the table format, that’s all.” While 11 of 30 (36.7%) participants preferred the table format for themselves, only 7 of 30 (23.3%) stated it would be easier for the public to understand. More than 75.0% of participants believed it would be easier for the general public to understand the graphic.

Several comments were made on how the numeric information on the table was helpful and should be added to the graphic. For example, participant 23 (SNS=17) stated: “I feel if you really want to educate us, I think you need to provide what the food would be, intake supplements would be, total intake and the limit. So, I feel like a mix of both would be beneficial to really have a basis and foundation of what’s important versus just kind of have a cereal box.”

Total Nutrient Intake and Recommended Range

Participants in Phase I were asked to calculate the correct number for total nutrient intake from iron in the table as well as what they get when the combine the blue bar (nutrients from supplements) and green bar (nutrients from food and drinks) in the graphic (Table 3). While most participants correctly answered how to calculate total nutrient intake from iron (n=25, 83.3%; 68.8% SNS ≤16), only 18 of 30 (60.0%; 43.8% SNS ≤16) were able to answer what combining the blue and green bar represented in the graphic (answer: total nutrient intake) (Table 3). Additionally, many participants had trouble understanding which nutrients were outside of the recommended range, such as nutrients below the target value. Although 25 out of 30 participants (83.3%; 81.3% SNS ≤16) got the “over the limit” answer correct, only 15 of 30 participants (50%; 56.3% SNS ≤16) knew how to interpret which nutrient was below the target. One participant (P06, SNS=18) who incorrectly answered the “below the target” question stated: “I think, um, the vitamin C, is like over the limit. I think the other ones are, let me see, I think the magnesium is the only one that doesn’t have a limit, but all the other ones are like below the limit.” While the participant correctly answered, “over the limit”, “below the target” was incorrect (answer: Vitamin E).

Table 3.

Number and Percentage of Participants with Correct Responses for Each Graphic Feature in User Testing Phases I and II, by Subjective Numeracy Score

	Phase I (n=30)		Phase II (n=20)		Total (n=50)
Features	High Numeracya (n=14)	Lower Numeracy (n=16)	High Numeracy (n=8)	Lower Numeracy (n=12)	High Numeracy (n=22)	Lower Numeracy (n=28)
	n (%) of Participants with Correct Responses
Below the Target (Vitamin E)	6 (42.9%)	9 (56.3%)	8 (100%)	9 (75.0%)	14 (63.6%)	18 (64.3%)
Blue Bar + Green Bar (Total Nutrient Intake)	11 (78.6%)	7 (43.8%)	7 (87.5%)	12 (100%)	18 (81.8%)	19 (67.9%)
Blue Bar (Nutrient from Supplement)	14 (100%)	14 (87.5%)	8 (100%)	12 (100%)	22 (100%)	26 (92.9%)
Calcium Intake (Intake from Supplement)	-b	-	6 (75.0%)	9 (75.0%)	6 (75.0%)c	9 (75.0%)
Green Bar (Nutrient from Food)	13 (92.9%)	15 (93.8%)	8 (100%)	12 (100%)	21 (95.5%)	27 (96.4%)
Grey Bar (Intake Remaining to Reach Limit)	-	-	7 (87.5%)	11 (91.7%)	7 (87.5%)	11 (91.7%)
Iron Intake (Intake from Food)	-	-	7 (87.5%)	11 (91.7%)	7 (87.5%)	11 (91.7%)
Limit	14 (100%)	13 (81.3%)	8 (100%)	10 (83.3%)	22 (100%)	23 (82.1%)
Limit (Magnesium)	-	-	2 (25.0%)	1 (8.3%)	2 (25.0%)	1 (8.3%)
Over the Limit	12 (85.7%)	13 (81.3%)	8 (100%)	9 (75.0%)	20 (90.9%)	22 (78.6%)
Target	13 (92.9%)	16 (100%)	8 (100%)	10 (83.3%)	21 (95.5%)	26 (92.9%)
Total Intake Table (Iron)	14 (100%)	11 (68.8%)	-	-	14 (100%)	11 (68.8%)
Total Intake Graphic (Vitamin E)	-	-	7 (87.5%)	10 (83.3%)	7 (87.5%)	10 (83.3%)
Warning Symbol (When Over the Limit)	12 (85.7%)	15 (93.8%)	7 (87.5%)	11 (91.7%)	19 (86.4%)	26 (92.9%)

^aSNS is Subjective Numeracy Scale: Range 3–18, Study Range 6–18;¹⁶ high numeracy is 17–18 and lower numeracy is ≤16

^bA dash (−) represents data that was not collected in Phase I or Phase II.

^cFor features assessed in Phase I or II only (e.g., Calcium Intake from Supplement), percentages in the Total column reflect the participant total from that phase.

Target, Limit, Warning Symbol, and Bar Colors

Participants in Phase I were asked to define Target, the recommended daily nutrient intake based on age and sex (Table 3). Most participants, 29 out of 30, (96.7%; 100% SNS ≤16) were able to accurately define the target. The concepts of the limit (highest daily nutrient intake safe to consume) and the warning symbol feature (an amount that exceeds the upper limit) were also well understood, with 27 out of 30 participants (90.0%) demonstrating correct understanding. Of the 16 participants with lower numeracy, 13 correctly answered limit (81.3%), while 15 correctly interpreted the warning symbol (93.8%). Additionally, more than 93.0% of participants correctly understood the green (nutrient from food) and blue bars (nutrient from supplement) in the graphic. Of those with lower numeracy, 93.8% understood the green bar (n=15) and 87.5% understood the blue bar (n=14).

Participant Phase I Feedback

All participants in Phase I offered suggestions to improve the graphic. Participant feedback was reviewed and discussed before making final decisions on what would be implemented or changed in the graphic for Phase II testing. For example, participants were confused about the white (“blank”) bar and wanted a color or design added, so lines were added within the bar and the description (intake remaining before the limit) placed in the legend. Participants also wanted a clearer, bolder target line, so the revised graphic has one continuous bolded line representing the target across all nutrients. Participants also requested modifications to the layout for the graphic (i.e., alphabetical order of nutrients and bolding of key features). Additionally, participants asked for other nutrients, such as potassium and vitamin B12, which were discussed with the nutrition experts on the team for inclusion in the updated graphic. Most of all, participants wanted to have the numbers from the table in Phase I added to the graphic, creating a combination of both the table and graphic. Based on Phase I results, adjustments were made to the graphic for Phase II to enhance the clarity of the most easily understood representation (Figure 4).

Phase II Results

This section summarizes Phase 2 participants’ understanding of key features presented from an updated graphic on nutrient intake from food, drinks, and dietary supplements (Figure 4). First, we describe the response accuracy for key features, such as the recommended range and bar colors (Table 3). Second, we summarize results on behavioral intentions, followed by participant feedback to improve the graphic.

Total Nutrient Intake and Recommended Range

Participants in Phase I had difficulty stating what the combination of the blue (nutrient from supplement) and green (nutrient from food) bars in the graphic meant. With the addition of numeric information that included total intake in the graphic, 19 of 20 (95.0%) participants in Phase II correctly answered what the combination of the blue and green bar created (total nutrient intake) with all lower numeracy participants correctly answering this question, compared to 7 of 16 (43.8%) in Phase I. Furthermore, Phase II participants (n=20) were asked to identify intake from food for iron and intake from supplements for calcium. Most participants (n=18, 90.0%) correctly answered intake from food for iron (91.7% SNS ≤16); however, there was poorer understanding for total intake from supplemental calcium (n=15, 75.0%; 75.0% SNS ≤16). Participants were asked in Phase II to calculate total intake of Vitamin E, of which 17 (85.0%; 83.3% SNS ≤16) were able to correctly identify.

After incorporating the edits for the graphic, compared to 50% Phase I participants, 17 of 20 (85.0%) participants in Phase II understood the meaning of below the target of the recommended range when examining the revised graphic. All participants with higher numeracy correctly answered which nutrients were below the target and participants with lower numeracy improved from 56.3% in Phase I to 75.0% in Phase II. Most participants in Phase II (n = 17, 85.0%) correctly identified the nutrient that exceeded the recommended limit. However, among those with lower numeracy, understanding declined slightly in Phase II (n = 9, 75.0%) compared to Phase I (n = 13, 81.3%). In Phase II, an asterisk was added for magnesium to denote a limit from supplements only. Participants were asked when magnesium would be over the limit; only 3 of 20 (15%) participants in Phase II could correctly identify the reason demonstrating a need to change how this information is presented.

Target, Limit, Warning Symbol, and Bar Colors

The number of correct participant responses in Phase II were similar to those in Phase I for understanding the target, limit, and warning symbol. Most Phase II participants (n=18, 90.0%) understood the limit, target, and the warning symbol. An improvement was found among lower numeracy participants for limit (81.3% to 83.3%), yet there were decreases for target (100% to 83.3%) and warning symbol (93.8% to 91.7%). Additionally, all participants correctly understood what the green and blue bars in the graphic meant, noting that the colors and legend were especially helpful. In Phase II, grey stripes were added to the bars to indicate the remaining amount of a nutrient that could be safely consumed before reaching the limit. Most participants (n=18, 90.0%) correctly interpreted this feature, including 91.7% of participants with lower numeracy. However, two participants demonstrated confusion, for example, interpreting it as “anything less than the limit” (P116, SNS=14) or “how much intake you could take, whether it’s food and drinks or from supplements” (P113, SNS=18).

Behavioral Intentions

Participants in Phases I and II were asked what they would do next if they had a concern or question about the information presented in the graphic. Most participants, regardless of numeracy level, gave similar responses: Participant 102 (SNS=13) shared “I’d do some research, all of those links, poke around online and then talk to my doctor about it.” Similarly, Participant 03 (SNS=13) shared they would “click on the link, see if I could get a little bit more clarity. And, like I said, if I can’t, then I will have to ask the experts.” Besides clicking the link and speaking with their doctor, participants also discussed searching for information online using tools like a consumer medical information website and an internet search engine; for example, participant 10 (SNS=17) shared “I would consult a physician, and I would do my own Google research.”

Participants were also asked how they would respond if, such as in the study graphic, their Vitamin E was too low and Vitamin C too high. Responses for Vitamin E varied between “The page shows me I should take more vitamin E, but given that it’s so minor, I probably wouldn’t really change anything because it doesn’t seem very significant to me” (P109, SNS=18) to “I would just increase it. I’m not really concerned with whether it comes from food or supplements” (P111, SNS=17). Similar answers were also found in Phase I: “I would definitely increase my Vitamin E intake because I’m not missing the one hundred percent of the target, so, I would probably consider just seeing if I can incorporate more foods that are high with Vitamin E or rich with Vitamin E that way before seeing supplements” (P08, SNS=17).

Some discussed how they would use an internet search engine to find information on what to eat to increase Vitamin E intake. For example, participant 105 (SNS=16) shared “I would just probably do a Google search and just do my own research that way.” Similarly, participant 30 (SNS=9) shared “I have no idea what vitamin E is that I recall. So maybe have an example, like the vitamin E is fish, it’s kiwi. It’s this kind of meat because it will remind me of what vitamin E is. Then I have to Google it.” Participants were very focused on decreasing their intake for Vitamin C, with 100% of participants stating they would lower the dose of or stop a Vitamin C supplement, speak with their doctor, or lower it through their food intake due to the graphic showing that it is mostly via food.

Participant suggestions on revised graphic

Given that participants in Phase II overall better understood the graphic compared to participants in Phase I, there were fewer suggestions offered for the final graphic. The final version of the graphic was updated based on participant feedback as well as programming considerations required for coding and displaying the graphic in an online environment. Participants liked the addition of the numerical information and color coding, added from Phase I. Participants wanted to know how much intake was needed to reach a nutrient target; therefore, a pattern was added to the legend to support understanding of lower numeracy participants. An additional suggestion included moving “(No Limit)” to the end of the line with the limit numbers to make it easier for participants to find the information. The last suggestion was to include an “Intake exceeds target” to the graphic, which was added to the legend.

Discussion

To our knowledge, this study is among the first to apply usability testing methods to optimize communication about total nutrient intake from foods, drinks, and supplements. Additionally, concepts such as when a nutrient reaches a limit, a warning sign when over the limit, and daily target of a nutrient were tested. User testing allowed for the development of a graphic that provides accurate nutritional data on total nutrient intake by foods, drinks, and supplements and is easy to understand by people with a variety of numeracy levels. The final version of the graphic, published in November 2023 as a new section called Nutrient Intake from Foods, Drinks, and Supplements, incorporated final participant suggestions alongside necessary programming adjustments (Figure 5).¹² For example, there was difficulty displaying the limit at the correct scale for each nutrient; therefore, the limit line was removed. Also, programmers ensured 508 compliance standards with regards to color contrast and font legibility.

After completing Phase I of user testing, this study found the graphic easier for participants to understand and was the preferred format. Various scientific fields have called for the integration of pictorial and graph-based material to accompany - or in some circumstances, replace - more complicated text and numerical health information.²⁹ Participant preferences for graphic presentations of nutrient information in this study are consistent with prior research showing that pictorial educational materials are often easier to process and recall, particularly among individuals with lower health literacy or numeracy.³⁰ McGuire’s Information Processing Theory³¹ helps explain this preference by showing that the way a message is presented, such as including visuals, is an important factor that can affect how much attention people pay, how well they understand and remember the information, and whether they act on it.³²

In this study, 11 participants preferred the table format for themselves but only seven of them stated it would be easier to understand by the general public. Information on whether or how these participants considered themselves different from the general public was not obtained. Despite not obtaining this information, valuable data were gathered about the need to include the numerical data from the table in the graphic. Integrating numerical data into the graphic substantially improved user comprehension of total nutrient intake (blue + green bar), from 60.0% to 95.0%. This result suggests that a hybrid approach—combining visual and numeric information—may enhance participant understanding. Similar findings have been reported by Hamstra and colleagues,³³ who observed that adding numbers to graphics improved knowledge among participants with lower numeracy skills.

Research shows that patients with higher numeracy skills are more likely to correctly interpret nutrition information, such as food labels, and have better health outcomes.³⁴ However, given that an estimated 62% of U.S. adults aged 16 to 65 years have lower numeracy skills,³⁵ it is critical to test nutrition materials with this population to ensure the information is understandable across skill levels. In this study, participants in Phase II with lower numeracy demonstrated greater understanding on 5 of the 8 graphic features compared to lower numeracy participants in Phase I. Participants with higher numeracy showed improved or perfect understanding across all 8 features after revisions, suggesting that the final graphic was accessible to individuals across the numeracy spectrum. Final changes to the graphic included edits to the target (thicker target line; pattern added to represent intake needed to reach target), warning symbol (standardized placement and design for consistency), and over the limit (patterned color added for intake exceeds limit as well as for the warning symbol). These revisions were implemented in the final graphic to improve participant understanding, particularly among those with lower numeracy.

Participant feedback in this study highlights a strong preference, across both lower and higher numeracy levels, for simple, easy-to-understand visuals that also include numeric information to help interpret nutrient intake from food, drinks, and supplements. Participants in this study appreciated how the graphic allowed them to quickly assess whether their intake met, fell short of, or exceeded target values. This finding is consistent with previous research; for example, a systematic review by Galmarini, Marciano, and Schulz³⁶ found that interventions incorporating visual health materials improved comprehension, especially among individuals with lower health literacy skills. Furthermore, several participants indicated that viewing the graphic motivated them to seek additional information. For example, participants noted they would research how to reach a target value when intake was below the recommended level. This finding aligns with prior work by Pratt and colleagues³⁷, who found that graphically presenting nutrition information not only enhanced consumer understanding but also influenced behavior change by making nutritional decision-making more accessible. This feedback reinforces the value of hybrid graphics that combine both pictorial and numeric data to support informed decision-making across individuals with varying levels of health literacy.

Strengths and Limitations

A strength of this study is the heterogeneity of the sample, which was intentionally recruited to reflect diversity across these characteristics (age, gender identity, race, and subjective numeracy) to capture diverse perspectives. Another strength of this research is the ability to obtain participant feedback in real time during the interview, then update the graphic as recommended to improve understandability. Although participants in Phase II were able to reflect on and respond to changes suggested by participants in Phase I—serving as a form of iterative feedback and informal member checking—the research team was unable to conduct formal member checking with the same participants after data analysis, due to time and budget constraints. Due to less variation in subjective numeracy scores, participants were divided into two groups (high and lower subjective literacy) instead of terciles based on their subjective numeracy scores. Additionally, data in the table and graphic were hypothetical and not actual participant data. While this research may not be generalizable to non-supplement users, the goal was to user test specifically with supplement users who would use this portion of the RNR.

Conclusions

User testing in this two-phase study led to the development of a new graphic “Nutrient Intake from Food, Drinks, and Supplements” (Figure 5) with improved comprehension. After updating the graphic based on participant suggestions, there was better comprehension of the graphic providing evidence for the efficacy of user testing of nutrition materials. A new graphic that is easier to comprehend is more likely to enhance the end-user’s ability to make informed decisions regarding their health. Further, participants noted feeling inspired to seek out additional information on dietary supplements after seeing their hypothetical total nutrient intake results, which can potentially lead to better informed consumers of dietary supplements. Future research could explore the effectiveness of the revised graphic among different populations or assess whether it influences behavior change among supplement users.

Supplementary Material

Supp.files

Supplementary Figure I. Semi-structured qualitative interview guide used during Phase I of user testing to assess comprehension of total nutrient intake from foods, beverages, and dietary supplements

Supplementary Figure II. Semi-structured qualitative interview guide used during Phase II of user testing to assess comprehension of total nutrient intake from foods, beverages, and dietary supplements

1

Figure III. Original table and newly designed figure on nutrient intake, used during Phase I user testing to assess comprehension of intake from foods, drinks, and supplements

Figure IV. Revised figure on nutrient intake, used during Phase II user testing to assess comprehension of intake from foods, drinks, and supplements

Figure V. Published figure on nutrient intake from foods, drinks, and supplements from the National Cancer Institute ASA24 Respondent Nutrition Report, revised after two rounds of user testing

Research Snapshot.

Research Question:

How well do supplement users understand total nutrient intake from foods, drinks, and dietary supplements as presented in the Respondent Nutrition Report of the NCI’s Automated Self-Administered 24-Hour Dietary Assessment Tool, and does comprehension differ between a table and a newly designed figure format?

Key Findings:

This qualitative study included a total of 50 dietary supplement user participants across two phases (Phase I: n=30; Phase II: n=20). Phase I user testing determined that a graphic representation of total nutrient intake from foods, drinks, and dietary supplements was easier to understand than a table. After incorporating graphic edits in Phase II, understanding of the graphic features improved among both lower- and higher-numeracy participants.