Abstract
Introduction
Identification of low levels of health literacy is important for effective communication between providers and clients. Assessment instruments for general health literacy are inadequate for use in nutrition education encounters because they do not identify nutrition literacy. The primary objective of this 2-part study was to assess content validity for the Nutrition Literacy Assessment Instrument (NLAI).
Methods
This study included a 35-item online survey of registered dietitians (134 of whom answered all questions) and a pilot study in which 5 registered dietitians used the NLAI among 26 clients during nutrition education consultations. To assess agreement with the NLAI by survey participants, we used the following scale: “necessary” (70% agreement), “adequate” (80% agreement), or “good” (90% agreement); comments were analyzed by using content analysis. For the pilot, we made comparisons between subjective assessments, the Rapid Estimate of Adult Literacy in Medicine (REALM), and the NLAI. Registered dietitians also completed a postpilot–study survey.
Results
For the online survey, we found good agreement (average, 89.7%) for including each section of the NLAI. All sections accomplished their purpose (average, 81.5%). For the pilot, REALM and NLAI correlation (r = 0.38) was not significant; the subjective assessment of clients by dietitians and NLAI lacked agreement 44% of the time, and registered dietitians provided instruction on deficient knowledge and skills identified by the NLAI 90% of the time.
Conclusion
The NLAI is a content-valid measure of nutrition literacy. Additional validation of the NLAI is important because an objective instrument is needed for identifying nutrition literacy, a construct that appears to be different from health literacy.
Introduction
Health literacy is “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” (1). Assessment instruments (2–4) provide guidance for choosing patient health information according to individual health literacy needs.
Nutrition is important for disease prevention, but increasing rates of chronic diseases suggest a need for nutrition education. However, nutrition information is complex and may require high levels of cognitive skills (5). Most registered dietitians do not assess health literacy, perhaps because current instruments do not assess nutrition literacy. Nutrition literacy requires knowledge of nutrition principles and skill in food-related tasks (6).
The Rapid Estimate of Adult Literacy in Medicine (REALM) is a valid and reliable instrument that predicts reading ability of health care–related words (2). Two instruments relate to nutrition literacy assessment: the Newest Vital Sign (7) and the Nutrition Literacy Scale (8). The food label used in the Newest Vital Sign purports to measure numeracy, but this instrument does not assess nutrition knowledge. The Nutrition Literacy Scale attempts to measure comprehension of nutritional information, but it is unclear whether it provides any measure beyond print literacy, and other uses have not been described in the literature.
Consequently, we developed the Nutrition Literacy Assessment Instrument (NLAI). Literature review and expert interviews (6) were the basis for including the following domains: appreciation of relationships between nutrition and health, knowledge of macronutrients, food measurement skill, numeracy and label reading, and skill in grouping like foods.
The objectives of this 2-part study were to 1) establish content validity of the NLAI and identify attitudes toward needs and constructs for a nutrition literacy instrument (survey) and 2) test the usability of the instrument in nutrition education settings and determine its impact on teaching delivery from the registered dietitian’s perspective (pilot).
Methods
Survey: critique of instrument
A 35-item survey was designed to assess content validity for the NLAI and attitudes among dietitians toward nutrition literacy. Content validity reflects the relevance of the assessment instrument to its targeted construct (9) and usually requires expert review (10). Content validity should be determined before construct validity, which is the accuracy with which the assessment measures the targeted construct. The survey had 7 sections; the first section addressed whether survey participants found the instrument’s algorithm useful and understandable. For the next 5 sections of the survey (nutrition and health, macronutrients, household food measurement, food labels and numeracy, and food groups), participants were asked whether the instrument accomplished its purpose, whether the questions were appropriate in difficulty, whether anything important had been left out, and whether the section was important to include. For the section on nutrition and health, a question on length replaced a question on whether anything was left out. Each question of the survey allowed participants to provide additional comment. The seventh section addressed health literacy, and because another research question was whether dietitians would prefer to use the REALM instead of the NLAI, we asked 2 questions about the REALM.
All methods were approved by the University of Illinois institutional review board (IRB). Two dietetic practice groups (each with approximately 6,400 members) of the Academy of Nutrition and Dietetics (formerly the American Dietetic Association) were selected as participants in the survey because their practice areas involve nutrition education. Methods were approved by the dietetic practice groups and the Academy of Nutrition and Dietetics. All data were collected in December 2011.
The survey recruitment invitation was distributed by an e-mail that included a link to the survey, which was designed by using online survey software (SurveyGizmo, Boulder, Colorado). Data were automatically saved as an Excel (Microsoft Corp, Redmond, Washington) spreadsheet. Of 385 potential participants, 377 (98%) consented to the survey, 178 (46%) answered some questions, and 134 (35%) completed the entire survey. Of the 243 who consented but did not finish, 211 left the survey immediately after consent; this drop-off might be explained by the temporary web-browser incompatibilities on the day the survey was released.
For statistical analyses, we used PASW Statistics for Windows, version 18.0 (SPSS Inc, Chicago, Illinois). To evaluate the degree of agreement with the NLAI by survey participants, we compared our data with the following scale: agreement at or above 70% is necessary, agreement at or above 80% is adequate, and at or above 90% is good (11). Additionally, we analyzed comments by using content analysis, which involves identifying coherent and important examples, themes, and patterns (12). Two researchers analyzed the comments separately to develop a list of keywords and codes and assigned comments to categories. Then each reviewed the results of the other; they agreed on assignment of comments to categories for 363 of 377 (96%) comments and discussed differences until consensus was reached. Overarching themes were developed from the codes that had the highest frequency of similar response.
Pilot study
Five registered dietitians were recruited from personal contacts for the pilot study; because they were viewed as coinvestigators, they completed human subjects training before participation. They were recruited from 4 participating sites: a regional bariatric surgical facility, 2 metropolitan outpatient clinics, and a private practice. The pilot study was implemented in the metropolitan areas of Chicago and Urbana-Champaign, Illinois. All methods received IRB approval, and all data were collected between May 2011 and May 2012.
Participants in the pilot study were trained by the first author (H.G.) on health literacy so they would know how to interpret and act upon the results of the NLAI. Health care providers who receive training on health literacy have greater intentions of identifying patients with low levels of health literacy and of checking for patient understanding of information provided (13) and more often use methods and educational materials designed for people who have low levels of health literacy (14). The training included 3 online educational modules, a packet of research materials, and a telephone conversation with the researchers to clarify the research process.
Adult clients (aged >18) were recruited through a flyer by clinic personnel using a convenience sample approach at selected outpatient clinics in the dietitians’ normal clinic area. We targeted 10 completed surveys per participating dietitian within 1 month. If the dietitian determined the client was not suitable because of cognitive impairment or an inability to read, the patient did not participate. A prescreening instrument was required by the IRB at 1 site to ensure competency for participation.
After consent forms were completed, the dietitian completed a subjective assessment (a short form to indicate whether the dietitian’s initial impression, based on general observation, of the client’s nutrition literacy was adequate, marginal, or inadequate); read the instructions for the REALM and administered the interview; gave the NLAI to the client to complete, and recorded the time required to complete the NLAI. The REALM is scored by totaling the number of words (of 66) read correctly, yielding a predicted age-level reading. On the basis of the number of correct answers, each section of the NLAI is scored as “suggests high likelihood of inadequate nutrition literacy,” “suggests marginal nutrition literacy,” or “suggests adequate nutrition literacy.” The dietitian then proceeded with the scheduled nutrition education and completed a postpilot–study survey. We used nonparametric correlation tests (Spearman ρ) to evaluate the relationship between the NLAI and the REALM.
Results
Survey: critique of instrument
We found that the algorithm achieved adequate agreement (81.5%) on the basis of yes answers to “Is the algorithm easy to understand and follow?” (93.1%) and “Is this section important to include?” (83.1%) and no answers to “Are there decisions that are missing?” (68.1%) (Table 1). Almost one-third (29.5%) of participants believed decisions had been left out of the algorithm; the most common themes identified as being left out were “language/cognitive barriers” (14 comments), “readiness to learn” (5 comments) and “ability to purchase and prepare food” (4 comments).
Table 1. Results of Survey Among Registered Dietitians on Nutrition Literacy Assessment Instrument (NLAI), 2011.
| Question | No. of Responses | n (%) | n (%) | n (%) |
|---|---|---|---|---|
| Algorithm | Yes | No | Neithera | |
| Does the algorithm accomplish its purpose? | 178 | 153 (85.9)b | 16 (8.9) | 9 (5.0) |
| Is the algorithm easy to understand and follow? | 176 | 164 (93.1)c | 9 (5.1) | 3 (1.7) |
| Is this section important to include? | 166 | 138 (83.1)b | 22 (13.2) | 6 (3.6) |
| Are there decisions that are missing? | 165 | 48 (29.5) | 111 (68.1) | 6 (2.5) |
| Nutrition and health | Yes | No | Neithera | |
| Does this section accomplish the purpose of measuring reading comprehension? | 157 | 131 (83.4)b | 22 (14.0) | 4 (2.6) |
| Does this section accomplish the purpose of identifying client’s understanding of relationship between nutrition and health? | 157 | 133 (84.7)b | 16 (10.2) | 8 (5.1) |
| Is this section important to include in the instrument? | 157 | 127 (80.9)b | 23 (14.6) | 7 (4.5) |
| Yes | No, Too Short | No, Too Long | ||
| Is the passage appropriate in length? | 157 | 105 (66.9) | 3 (1.9) | 49 (31.2) |
| Yes | No, Too Easy | No, Too Hard | ||
| Are the questions appropriate in difficulty? | 155 | 109 (70.3) | 15 (9.7) | 31 (20.0) |
| Macronutrients | Yes | No | Neithera | |
| Does this section accomplish the purpose of identifying knowledge of macronutrients? | 149 | 133 (89.3)b | 14 (9.4) | 2 (1.3) |
| Is this section important to include in the instrument? | 147 | 128 (87.1)b | 17 (11.6) | 2 (1.4) |
| Has anything been left out of this section that you feel is important? | 148 | 32 (21.6) | 114 (77.0) | 2 (1.4) |
| Yes | No, Too Easy | No, Too Hard | ||
| Are the questions appropriate in difficulty? | 146 | 101 (69.2) | 1 (0.7) | 44 (30.1) |
| Household food measurement | Yes | No | Neithera | |
| Does this section accomplish the purpose of identifying ability to estimate portion size? | 145 | 122 (84.1)b | 22 (15.2) | 1 (0.7) |
| Is this section important to include in the instrument? | 147 | 140 (95.2)c | 6 (4.1) | 1 (0.7) |
| Has anything been left out of this section that you feel is important? | 148 | 60 (40.5) | 88 (59.5) | NA |
| Yes | No, Too Easy | No, Too Hard | ||
| Are the questions appropriate in difficulty? | 147 | 143 (97.3)c | 0 | 4 (2.7) |
| Food label and numeracy | Yes | No | Neithera | |
| Does this section accomplish the purpose of identifying skill with use of food labels? | 146 | 139 (95.2)c | 6 (4.1) | 1 (0.7) |
| Is this section important to include in the instrument? | 145 | 137 (94.5)c | 8 (5.5) | 0 |
| Has anything been left out of this section that you feel is important? | 142 | 26 (18.3) | 103 (72.5) | 13 (9.2) |
| Yes | No, Too Easy | No, Too Hard | ||
| Are the questions appropriate in difficulty? | 146 | 107 (73.3) | 0 | 39 (26.7) |
| Food groups | Yes | No | Neithera | |
| Does this section accomplish the purpose of identifying ability to group foods? | 141 | 139 (98.6)c | 2 (1.4) | 0 |
| Is this activity appropriate in difficulty? | 139 | 135 (97.1)c | 4 (2.9) | 0 |
| Is this section important to include in the instrument? | 140 | 127 (90.7)c | 13 (9.3) | 0 |
| Has anything been left out of this section that you feel is important? | 139 | 23 (16.5) | 112 (80.6)b | 4 (2.9) |
| Health literacy | ||||
| Would you prefer to use the REALM instead of the NLAI to assess nutrition literacy? | 139 | 20 (14.4) | 111 (79.9) | 8 (5.8) |
| Are there any sections of the REALM that you feel would be beneficial to include on a new assessment tool? | 134 | 34 (25.4) | 98 (73.1) | 2 (1.5) |
| Do you use an instrument to assess health literacy in your clients? | 140 | 10 (7.1) | 130 (92.9) | NA |
| Would you use an instrument if there was one available? | 134 | 98 (73.1) | 28 (20.9) | 8 (6.0) |
| Is health literacy an issue that you feel is important? | 138 | 133 (96.4) | 5 (3.6) | NA |
| Is an assessment of nutrition literacy important enough to nutrition education to take the time for an assessment? | 139 | 112 (80.6) | 25 (18.0) | 2 (1.4) |
Abbreviation: REALM, Rapid Estimate of Adult Literacy in Medicine; NA, not applicable.
Category added by the researchers after the survey was administered for participants who did not choose yes or no but commented in a way that was not distinguishable as yes or no.
Answer had adequate agreement (>80%).
Answer had good agreement (>90%).
For yes answers to the question, “Is this section important to include in the instrument?” the sections achieving adequate agreement were nutrition and health (80.9%) and macronutrients (87.1%), whereas the sections achieving good agreement were household food measurement (95.2%), food label and numeracy (94.5%), and food groups (90.7%). With an overall score of 89.7%, these 5 sections achieved good agreement for importance.
Combining yes answers to whether the section accomplished its purpose and whether the section was appropriate in difficulty or in length and no answers to whether anything had been left out, we found that 5 sections achieved agreement. The sections on nutrition and health (76.3%) and macronutrients (78.5%) achieved the minimum required for agreement; the sections on household food measurement (80.3%) and food label and numeracy (80.3%) achieved adequate agreement; and food groups (92.1%) achieved good agreement.
For the section on nutrition and health, 15 participants commented that the reading level was too high, 13 commented that the section was too long or wordy, and 9 commented that the concepts were too advanced. For the section on macronutrients, 10 participants commented that the section was too difficult or “encourages guessing.” For the section on household food measurement, 19 participants commented that visual references in pictures are needed for better size estimation; 11 noted issues with the use of the word “portions”; 8 suggested modifying the “milk image”; and 39 made suggestions for including various alternative foods.
The section on food label and numeracy achieved a high score for importance (94.5%) and purpose (95.2%), but it did not score as well for the questions on whether anything had been left out (only 72.5% said no) and difficulty (26.7% said section was too hard). Fifteen participants commented that 1 question, which requires computation of percentages, is too hard, and 8 commented the section overall is too difficult.
Most participants agreed the section on food groups is important (90.7%), achieved its purpose (98.6%), and left nothing important out (80.6%). Six participants noted a need for an “others” category, which might include foods such as added sugars or oils, and 4 noted that a “combination food” (foods that are composed of more than 1 food group, such as tacos or lasagna) should be added as food categories.
Most (79.9%) participants did not prefer to use the REALM instead of the NLAI to assess nutrition literacy. Whereas 92.9% of participants indicated they do not use health literacy assessment instruments, 73.1% agreed that if an instrument designed for nutrition was available, they would use it in their practice. Almost all participants (96.4%) agreed that health literacy is important, and most (80.6%) agreed that an assessment of nutrition literacy is worth the time it would require. Of the 98 participants who indicated they would use an instrument if one was available, 10 indicated yes, if time is available. Of those who said they would not used such an instrument, 5 participants indicated there is not enough time to use an instrument during nutrition education encounters, and 4 participants said they prefer an interactive approach.
Pilot study
Twenty-five clients completed the REALM, and 26 clients completed the NLAI (Table 2). Most (n = 23) scored greater than 61 on the REALM (predictive of a reading level above 9th grade), but of those, 7 clients scored at marginal nutrition literacy and 1 client scored at inadequate nutrition literacy for at least 1 area of the NLAI. The Spearman ρ correlation between the REALM and NLAI was not significant (r = 0.38; P = .06). Additionally, the subjective assessment and the NLAI indicated lack of agreement at a rate of 44%. For the 10 clients who received less than adequate scores on the NLAI, the dietitian provided instruction on the deficient knowledge or skill area 90% of the time. All 5 dietitians indicated that the time needed to complete the assessment is “about right” (Table 3).
Table 2. REALM and NLAI Scores for Nutrition Education Clients Participating in Pilot Study on Nutrition Literacy, 2011–2012.
| Category | No. of Clients Evaluated | Mean (SD) | Range |
|---|---|---|---|
| REALM scorea | 25 | 63.9 (3.8) | 50–66 |
| Time it took to complete NLAI, min | 20 | 8.4 (3.0) | 4–15 |
| NLAI scoreb, % correct | 26 | 86.8 (9.6) | 58.3–100.0 |
| Readability of educational materials, grade levelc | 23 | 7.0 (1.1) | 3.0–7.6 |
Abbreviations: REALM, Rapid Estimate of Adult Literacy in Medicine; NLAI, Nutrition Literacy Assessment Instrument.
Based on maximum score of 66.
Although the maximum possible score on the NLAI is 40, some clients were not instructed to complete the entire instrument; thus, we converted scores to percentage correct.
Readability assessed by Flesh-Kincaid Grade Level in Microsoft Word 2010 (Microsoft Corp, Redmond, Washington).
Table 3. Postpilot Survey Among Registered Dietitians (n = 5) on Nutrition Literacy Assessment Instrument, 2011–2012.
| Question | No. of Responses |
|---|---|
| The time needed to complete the assessment is . . . | |
| About right | 5 |
| Too long | 0 |
| Was the content of the assessment applicable to the needs of the client? | |
| Yes | 5 |
| No | 0 |
| Does the instrument adequately separate clients into different levels of understanding? | |
| Yes | 3 |
| No | 1 |
| Not applicable | 1 |
| Comment | “I felt that with the very few clients that I tested that sometimes reading level and understanding don’t necessarily coincide. One lady did well on the test but had a more limited reading ability.” |
| Please rank the difficulty experienced by clients in completing the assessment. | |
| Very difficult | 0 |
| Difficult | 0 |
| Appropriate | 4 |
| Too easy | 1 |
| Do you have any suggestions for improvements? | “It was perceived that some of the clients did not want to take the time to complete the assessment, and others may have been too intimidated to participate. My only suggestion would be to provide an incentive for participation. Possibly a handout or booklet on improving health literacy.” |
Discussion
The NLAI was valid as tested within this study, and dietitians preferred the NLAI over the REALM. However, nutrition literacy is a complex concept, and responses to some questions deserve further investigation after the instrument is refined. In addition, the methods used to evaluate nutrition literacy need to reflect this complexity. “Dimensionality . . . the number of theoretically and empirically distinct subcomponents of the broader construct” (15) is important to the design of nutrition instruments because gaining an understanding of one’s nutrition literacy is dependent upon the complex interplay of knowledge of related subtopics as reflected in the NLAI.
Several sections of the NLAI require further investigation. For instance, the responses using the cloze method for the section on nutrition and health indicated that the section accomplished its purpose but may be too long and difficult. Although others have used simpler and shorter assessment techniques (15), these techniques may not capture the essence of the questions’ rationale. Diet and health is complex, and requires a higher order of integrated conceptualization than perhaps multiple response items can divulge. However, the cloze method has been used in assessing the understanding of other health-related topics, such as prostate cancer (16), pharmacy instructions (17), and cardiovascular disease risk (18). Therefore, while the cloze method is used in health arenas other than nutrition, dietitians may be unfamiliar with, uncomfortable with, or lack knowledge about the applicability of this method.
Our approach to the section on macronutrients is similar to the approach used for the 23-item Nutrition Knowledge test (15), which asks participants to identify the nutrient content of foods. Ideally, an instrument should stratify participants into categories of nutrition literacy, requiring questions with varying levels of difficulty. Deciphering which questions to include or exclude according to difficulty is necessary and is determined by measures of construct validity. Because the section on macronutrients has not been widely evaluated with clients, the range in difficulty of questions or appropriate foods to include cannot be determined at this time.
Food photographs (like the photographs used for the food portions section of the NLAI) are a useful aid for estimating food portions (19) and have been studied with varying degrees of success for use in reporting portion sizes in food consumption studies (20–22). It is not known, however, whether food photographs labeled by common household measurements (such as those in the NLAI) improve accuracy of estimation (23). Our questions did not ask participants to estimate the amount seen in photographs because the amounts were identified in the questions. Rather, participants were asked to identify whether the stated amount for a given food is the “right” portion. In our study, the photographs served as a visual cue for proportionality, but they may not be necessary if knowledge of common food measurements is strong.
The section on food label and numeracy was adapted from the Newest Vital Sign, a reliable and valid measure of health literacy. Because it uses a nutrition label as the text reference, it is not on its own a measure of nutrition literacy but rather a measure of the ability to read food labels. Food labeling is an area with which the public has struggled — to the point where front-of-package labeling and healthfulness scores have been suggested (24). Another food label literacy tool has been developed for children; it uses 10 items to determine children’s ability to make healthful choices based on the Nutrition Facts label. Each item presents 2 labels and asks which choice is the more healthful (25). Future developments of the NLAI could consider similar items for an adult version.
Conceptually, food groups have been central to nutrition education since 1916 (26), so we were not surprised that our survey participants indicated food groups are important to evaluating nutrition literacy. Several participants commented that combination foods should be addressed in the NLAI; however, the US Department of Agriculture’s MyPlate food guidance graphic does not incorporate combination foods, nor did previous food guides. We question whether the general public recognizes that some foods can be classified into more than 1 food group. In addition, the concept of food groups may be different for the client than for the professional. For instance, a study among low-income African American women found the names of food groups may differ between the women and the professionals, as well as foods attributed to the food groups (27).
This study had several limitations. Dietitians were unable to recruit 10 clients within 1 month at any of the participating sites. Through personal communication, the dietitians provided the following reasons for this inability: summer vacations (for dietitians and their clients), lack of interest among potential clients in spending additional time required for assessment, and lack of incentives for potential clients. Time was extended for 1 location to increase participation, but because of the extended initial IRB review process, and because multiple IRBs were involved, others did not want to engage in an amendment process to extend the time for data gathering.
We did not identify whether the dietitians participating in the pilot study had educational materials targeting the spectrum of health literacy levels that were available for use in educational encounters. Lack of variability in educational materials makes it difficult to derive relationship between use of the NLAI and adjustment in teaching methods made by the dietitian as a result of the assessment. Additionally, although written materials are commonly used in nutrition education and were included in data assessment, oral instruction is an important method of education as well and is not investigated here.
Although most participants did not prefer the REALM over the NLAI, and most indicated they would use an assessment tool if it were available, we did not specifically ask whether the participants would use the NLAI. We interpreted the positive responses to the content of the NLAI and the minimal number of suggestions for additions to mean that the NLAI would serve their needs and they would use it. Further testing of the applicability in the clinical setting is warranted.
Another potential limitation is that we did not gather demographic information from our pilot participants. We did not compare age, socioeconomic status, race/ethnicity, or educational attainment of our sample with the characteristics of samples used to develop other instruments.
The lack of agreement between the subjective assessment and the NLAI in the pilot study suggests a need for more objective measures for determining nutrition literacy. Although the Newest Vital Sign has received some attention in nutrition literature (28,29) as an instrument that can reliably and quickly assess health literacy, and its use of the Nutrition Facts label may be preferable to other health literacy instruments for nutrition practice, our survey indicates that dietitians believe nutrition literacy requires skills and knowledge beyond an ability to read food labels. Additionally, testing of the Newest Vital Sign in an elderly African American population indicates its practicality may be limited (30).
Registered dietitians participating in our online survey found the NLAI to be content valid as a measure of nutrition literacy. Although participants gave numerous suggestions for instrument improvement, the reader should exercise caution in overemphasizing any suggestion because most participants agreed with the approach and methodologies of the NLAI. Future research should evaluate use of the NLAI in clinical settings.
Acknowledgments
Funding for the online survey was provided by the Illinois AMVETS. The authors thank Kathy Keim, PhD, RD, LD, of Rush University Medical Center for her assistance with this research. At the time of the study, Dr Gibbs was a graduate student at the University of Illinois, Urbana, Illinois.
Footnotes
The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors' affiliated institutions.
Suggested citation for this article: Gibbs H, Chapman-Novakofski K. Establishing Content Validity for the Nutrition Literacy Assessment Instrument. Prev Chronic Dis 2013;10:120267. DOI: http://dx.doi.org/10.5888/pcd10.120267.
References
- 1. Neilson-Bowman L. Health literacy: a prescription to end confusion. Washington (DC): National Academies Press; 2004. 346 p. [PubMed] [Google Scholar]
- 2. Davis TC, Long SW, Jackson RH, Mayeaux EJ, George RB, Murphy PW, et al. Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med 1993;25(6):391–5. [PubMed] [Google Scholar]
- 3. Parker RM, Baker DW, Williams MV, Nurss JR. The test of functional health literacy in adults: a new instrument for measuring patients’ literacy skills. J Gen Intern Med 1995;10(10):537–41. 10.1007/BF02640361 [DOI] [PubMed] [Google Scholar]
- 4. Baker DW, Williams MV, Parker RM, Gazmararian JA, Nurss JR. Development of a brief test to measure functional health literacy. Patient Educ Couns 1999;38(1):33–42. 10.1016/S0738-3991(98)00116-5 [DOI] [PubMed] [Google Scholar]
- 5. King L, Watson WL, Chapman K, Kelly B, Louie JC, Hughes C, et al. Do we provide meaningful guidance for healthful eating? An investigation into consumers’ interpretation of frequency consumption terms. J Nutr Educ Behav 2012;44(5):459–63. 10.1016/j.jneb.2011.12.004 [DOI] [PubMed] [Google Scholar]
- 6. Gibbs HD, Chapman-Novakofski K. Exploring nutrition literacy: attention to assessment and the skills clients need. Health 2012;4(3):120–4 10.4236/health.2012.43019 [DOI] [Google Scholar]
- 7. Weiss BD, Mays MZ, Martz W, Castro KM, DeWalt DA, Pignone MP, et al. Quick assessment of literacy in primary care: the newest vital sign. Ann Fam Med 2005;3(6):514–22. 10.1370/afm.405 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Diamond JJ. Development of a reliable and construct valid measure of nutritional literacy in adults. Nutr J 2007;6:5. 10.1186/1475-2891-6-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Haynes SN, Richard DCS, Kubany ES. Content validity in psychological assessment: a functional approach to concepts and methods. Psychol Assess 1995;7(3):238–47. 10.1037/1040-3590.7.3.238 [DOI] [Google Scholar]
- 10. DeVellis RF. Scale development: theory and applications. 3rd edition. Los Angeles (CA): Sage Publications, 2012. [Google Scholar]
- 11. House AE, House BJ, Campbell MB. Measures of interobserver agreement: calculation formulas and distribution effects. J Behav Assess 1981;3(1):37–57. 10.1007/BF01321350 [DOI] [Google Scholar]
- 12. Patton MQ. How to use qualitative methods in evaluation. Newbury Park (CA): Sage Publications, 1987. [Google Scholar]
- 13. Mackert M, Ball J, Lopez N. Health literacy awareness training for healthcare workers: improving knowledge and intentions to use clear communication techniques. Patient Educ Couns 2011;85(3):e225–8. 10.1016/j.pec.2011.02.022 [DOI] [PubMed] [Google Scholar]
- 14. Schlichting JA, Quinn MT, Heuer LJ, Schaefer CT, Drum ML, Chin MH. Provider perceptions of limited health literacy in community health centers. Patient Educ Couns 2007;69(1–3):114–20. 10.1016/j.pec.2007.08.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Sapp SG, Jensen HH. Reliability and validity of nutrition knowledge and diet-health awareness tests developed from the 1989–1991 Diet and Health Knowledge Surveys. J Nutr Educ 1997;29(2):63–72. 10.1016/S0022-3182(97)70157-2 [DOI] [Google Scholar]
- 16. Friedman DB, Corwin SJ, Dominick GM, Rose ID. African American men’s understanding and perceptions about prostate cancer: why multiple dimensions of health literacy are important in cancer communication. J Community Health 2009;34(5):449–60. 10.1007/s10900-009-9167-3 [DOI] [PubMed] [Google Scholar]
- 17. Miller MJ, DeWitt JE, McCleeary EM, O’Keefe KJ. Application of the cloze procedure to evaluate comprehension and demonstrate rewriting of pharmacy educational materials. Ann Pharmacother 2009;43(4):650–7. 10.1345/aph.1L642 [DOI] [PubMed] [Google Scholar]
- 18. Martin LT, Schonlau M, Haas A, Derose KP, Rudd R, Loucks E, et al. Literacy skills and calculated 10-year risk of coronary heart disease. J Gen Intern Med 2011;26(1):45–50. 10.1007/s11606-010-1488-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Ovaskainen ML, Paturi M, Reinivuo H, Hannila ML, Sinkko H, Lehtisalo J, et al. Accuracy in the estimation of food servings against the portions in food photographs. Eur J Clin Nutr 2008;62(5):674–81. 10.1038/sj.ejcn.1602758 [DOI] [PubMed] [Google Scholar]
- 20. Nelson M. Jaraldsdottir J. Food photographs: practical guidelines I. Design and analysis of studies to validate portion size estimates. Public Health Nutr 1998;1(4):219–30. 10.1079/PHN19980038 [DOI] [PubMed] [Google Scholar]
- 21. De Keyzer W, Huybrechts I, De Maeyer M, Ocké M, Slimani N, van ’t Veer P, et al. Food photographs in nutritional surveillance: errors in portion size estimation using drawings of bread and photographs of margarine and beverages consumption. Br J Nutr 2011;105(7):1073–83. 10.1017/S0007114510004551 [DOI] [PubMed] [Google Scholar]
- 22. Foster E, Matthews JNS, Nelson M, Harris JM, Mathers JC, Adamson AJ. Accuracy of estimates of food portion size using food photographs — the importance of using age-appropriate tools. Public Health Nutr 2006;9(4):509–14. [DOI] [PubMed] [Google Scholar]
- 23. Subar AF, Crafts J, Zimmerman TP, Wilson M, Mittle B, Islam NG, et al. Assessment of the accuracy of portion size reports using computer-based food photographs aids in the development of an automated self-administered 24-hour recall. J Am Diet Assoc 2010;110(1):55–64. 10.1016/j.jada.2009.10.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Schor D, Maniscalco S, Tuttle MM, Alligood S, Kapsak WR. Nutrition facts you can’t miss: the evolution of front-of-pack labeling: providing consumers with tools to help select foods and beverages to encourage more healthful diets. Nutr Today 2010;45(1):22–32. 10.1097/NT.0b013e3181cb4561 [DOI] [Google Scholar]
- 25. Reynolds JS, Treu JA, Njike V, Walker J, Smithe E, Katz CS, et al. The validation of a food label literacy questionnaire for elementary school children. J Nutr Educ Behav 2012;44(3):262–6. 10.1016/j.jneb.2011.09.006 [DOI] [PubMed] [Google Scholar]
- 26. Welsh S, Davis C, Shaw A. A brief history of food guides in the United States. Nutr Today 1992;27(6):6–11. 10.1097/00017285-199211000-00004 [DOI] [Google Scholar]
- 27. Lynch EB, Holmes S. Food group categories of low-income African American women. J Nutr Educ Behav 2011;43(3):157–64. 10.1016/j.jneb.2010.02.011 [DOI] [PubMed] [Google Scholar]
- 28. Zoellner J, Connell C, Bounds W, Crook L, Yadrick K. Nutrition literacy status and preferred nutrition communication channels among adults in the Lower Mississippi Delta. Prev Chronic Dis 2009;6(4):A128. [PMC free article] [PubMed] [Google Scholar]
- 29. Carbone ET, Zoellner JM. Nutrition and health literacy: a systematic review to inform nutrition research and practice. J Acad Nutr Diet 2012;112(2):254–65. 10.1016/j.jada.2011.08.042 [DOI] [PubMed] [Google Scholar]
- 30. Patel PJ, Joel S, Rovena G, Pedireddy S, Saad S, Rachmale R, et al. Testing the utility of the newest vital sign (NVS) health literacy assessment tool in older African-American patients. Patient Educ Couns 2011;85(3):505–7. 10.1016/j.pec.2011.03.014 [DOI] [PubMed] [Google Scholar]