Table 3.

Summary of themes, their implications, and revision decisions resulting from cognitive interviews with patients (n=12) concerning the Nutrition Literacy Assessment Instrument.

Original Content	Issues Uncovered by CIs	Potential Action Taken	Implication	Revision
Nutrition & Health, n=7^a
Prose Text	1. ‘Shelf-stable’ is an uncommon term, though all participants who noted unfamiliarity provided examples of foods that meet the definition 2. ‘nutrient-dense’ and ‘energy-dense’ were unfamiliar terms to most participants. All were able to give correct examples of foods in each category. Two were unable to answer questions about the terms correctly.	1. “packaged or prepackage foods” were suggested replacement terms 2. More examples could be given to explain the terms. Alternatively, different terms could be used, such as ‘nutrient-rich’ or ‘nutrient-poor’.	1. Even though the term is unfamiliar, it was interpreted correctly. Many fresh foods come in packages and are low in sodium, so either of the suggested terms could also be misleading. 2. Strong readers use clues from the text to interpret and apply unfamiliar terms. These clues were sufficient for the majority of the sample to answer questions, which suggests these are good terms to include for testing literacy.	1. No revision made. 2. No revision made.
‘An example of an energy-dense beverage is _____.’	Five of seven chose incorrect answers. Three chose ‘diet-soda’, acknowledging the belief that diet soda also has calories. One chose ‘unsweetened tea’ and another chose ‘black coffee.’ The latter two had difficulty with the term ‘energy dense’, and chose these options because they are healthy.	An alternative answer option to ‘diet soda’ could be used.	Incorrect answers reflect inaccurate understanding of the calorie contents of lemonade and diet soda. This may be an actual nutrition literacy problem, reflected in increasing intakes of sugar-sweetened beverages. Thus, this question could be a discriminating question.	No revision made
Energy Sources in Food, n=6^a
All questions	Most questions were answered correctly by the entire sample. Two questions were missed by 1 or more individuals, but no consistency in logic for incorrect answers	Add more distractor options	If the entire sample answers questions correctly, the questions are more likely to be non-discriminating. Adding slightly wrong answer options may increase item difficulty.	Distractor options added
Household Food Measurement, n=10^a
Instructions	Instructions begin with, “We all have different nutrition needs. Sometimes we eat food in the right amounts and sometimes we choose smaller or larger portions than might be best to achieve a healthy diet…” Two participants felt these statements personalized the portions and answered questions based upon the portions they serve themselves rather than what they thought the portions should be as intended.	These sentences were removed and tested with two more participants. Both answered some questions based on what portions they think they should eat and other questions based on actual portions consumed. One said, “…if people aren’t familiar with recommended portion sizes, they’re just gonna compare to how much they usually serve themselves…”	If people do not know recommended portion sizes, their only reference is their own experience, causing them to sometimes use the amounts they consume as their comparison. More instruction about the discrepancy between recommended portions and actual consumption is needed. However, answering based on actual consumption habits reflects behavior, which may be the action upon one’s nutrition literacy.	Instructions read, “Sometimes we eat food in the right amounts as advised by nutrition experts…For each food in question, choose what you think is the right portion size. This portion may or may not be the amount you usually eat. The portion amounts given in the question are also shown in pictures.”
“Pictured at right is one 5-ounce chicken breast.”	9/10 participants who answered this question chose incorrectly. Most identified this portion as ‘about right.’	Delete question or modify	A different format for the question, similar to the format for the hamburger patty question, requiring selection of the right portion may be more effective.	“Using the photos above, choose the right portion for chicken.” [options are 3 ounces, 5 ounces, 10 ounces]
Household Food Measurement, n=6^a
“Pictured at left are 2 cups of pasta noodles.”	Participants suggested adding “cooked” before “pasta” to be clearer. Some incorrectly identified the portion as ‘about right’ because it appeared that was all that would be eaten at a meal.	Add “cooked”; Add side items to photo	While 2 cups of pasta is more than a recommended serving, eating 2 cups of pasta if nothing else is consumed is not excessive. People could use good logic and arrive at an incorrect answer.	“The spaghetti and meat sauce pictured at left includes two (2) cups of cooked pasta and 1 cup of meat sauce.” A glass of water and a side of garlic bread also pictured.
“Pictured at right is ½ cup of uncooked carrots.”	Most (7/9) incorrectly selected that ½ cup of uncooked carrots is ‘about right’ for a portion.	Delete question, keep unchanged, or modify	There is no consensus among guidelines as to what a serving of vegetables should be. Some say ½ cup, others 1 cup. ½ cup cooked is traditional standard. If item is removed, no vegetables are represented.	No revision made.
“Which portion of peanut butter as pictured above is equal to the portion for one serving according to a food label for peanut butter?”	Participants responded awkwardly to this question, some explaining the answer to this question was too obvious.	Delete question or modify	The question tests one’s ability to read the serving size on a food label, which is accomplished by another section of the instrument.	Question deleted.
Food Label & Numeracy, n=6^a
“If your doctor has advised you to limit your total fat intake to 60 grams per day, what percentage of your day’s intake have you eaten in one serving of macaroni and cheese?”	Question was missed by 4/6 participants. Most were unwilling to calculate a % and guessed “18%”	Delete question, keep unchanged, or modify	Calculating a % may be too advanced for most people, but the question may also be a distinguishing question. More data is needed on item performance before deletion.	No revision made.
Food Groups, n=3^a
Butter	Might also be considered dairy, “…when you churn it…’cause I thought it was made with milk…”	Keep unchanged	Butter is a significant source of saturated fat and little else. This may be a true nutrition literacy issue.	No revision made.
Lemonade	Incorrectly identified as a fruit, with logic that “if homemade, then it’s a fruit; if store-bought, then it’s an added sugar”	Keep unchanged	Whether store-bought or homemade, lemonade is an added sugar. This may be a true nutrition literacy issue.	No revision made.
All	No foods were unfamiliar to any participant	Keep unchanged	Familiarity with all foods is important to ensure that incorrect answers are due to problems with nutrition literacy.	No revision made.
Consumer Skills, n=6^a
‘Which green bean option is lowest in sodium content?’	Only question this section answered correctly by all participants	Delete, keep unchanged, or modify	May be a non-discriminating question, but not enough data from pilot testing to remove at this point. Sodium intake via processed foods is an important nutrition concept to include.	No revision made.
‘Which section on a food label provides the best information for choosing a whole grain food?’	Answered incorrectly by 3 participants because they felt it was a nuisance to read, ‘…the label is just full of all kinds of words I don’t even understand.’	‘best’ could be interpreted by some as ‘easiest’	Even if interpreted as ‘easiest’, those with stronger nutrition literacy may read ingredients lists more and be more comfortable. In pilot testing, ‘best’ was most often interpreted as ‘most reliable.’ Item testing is needed to determine action.	No revision made.
Hard Copy version	Formatting of this section was problematic for the 1 participant who used the hard copy of the instrument. He automatically disregarded option C (choices are equal) for each item, stating the “C” options did not seem to be “parallel” with the other answer options. The electronic version presented no similar problems.	Reformat to same presentation as electronic version, which gives all 3 answer option with a reference picture below.	Because no picture was associated with option C, it did not capture attention. Using the pictures as a reference below answer options removes the separation of this answer from the others	Paper version formatted according to electronic format.

A similar version of the NLit was previously tested via cognitive interviewing of 18 breast cancer patients with subsequent pilot testing [20]. The revisions determined from the pilot were incorporated into the present NLit along with minor modifications for the general chronic disease population. Thus, the varying number of patients interviewed here reflects the proportion of revisions to the domain between the two cognitive interview samples.