Abstract
Background and objective
The aim of this study was to develop a questionnaire for assessing nutritional knowledge among overweight adults. The questionnaire should reveal knowledge about current dietary recommendations, sources of nutrients, everyday food choices, and conditions related to overweight.
Design
The first draft of the nutrition knowledge questionnaire (113 items) was based on literature review. To ensure content validity and expert-assessed face validity, an expert panel examined the questionnaire. Thereafter, the questionnaire was tested for user friendliness and ambiguity by five students. The questionnaire was pilot tested in a group of obese adults, similar to the target group. The results were analyzed for item difficulty and internal consistency and comments made by respondents were taken into account. Two student groups, differing in nutritional expertise, answered the questionnaire on two occasions to test construct validity and test–retest reliability. After the retest, a total overview of the questionnaire was made by the expert panel. The final questionnaire consisted of 91 items.
Subjects
The pilot study was conducted in obese adults waiting for a gastric bypass operation (n=33). Construct validity (n=34) and test–retest reliability (n=27) was tested in two student groups: public health nutrition students and construction students.
Results
Results from the pilot study showed that internal consistency of the three first sections together was 0.84, measured by Cronbach's α. Test of construct validity showed that public health nutrition students scored significantly better than construction students (p<0.001 for all sections), and test–retest reliability for all sections together was 0.82 (Pearson's r).
Conclusion
The knowledge questionnaire had reasonable content-, face-, and construct validities and overall good reliability. The questionnaire can be a useful tool for measuring nutrition knowledge among obese adults.
Keywords: nutrition knowledge, questionnaire, obesity, adults, dietary recommendations
Obesity is a global challenge and one of the main risk factors for disease and early death (1). The condition increases the risk for developing diabetes type 2, cardiovascular disease, high-blood pressure, and several cancers (1–4). Rates of obesity are on the rise in both developed and developing countries (5). This is caused by an imbalance between energy intake and expenditure, due to increased intakes of energy-dense foods and reduced levels of physical activity in parts of the population (1, 4). Modifications of these behaviours are therefore needed in order to halt the increase in obesity. One of the suggested strategies for behavior modification is to enable people to make informed choices and take effective action (6).
Knowledge is recognized to be one of the components for change of food habits (7, 8) but research is, however, inconclusive on how important nutrition knowledge is for changing dietary behavior (8–14). One of the explanations for the inconclusive findings could be that knowledge has been poorly measured (15, 16).
Knowledge can be difficult to assess and good instruments for valid and reliable measurements are required (17, 18). Kline had defined a set of criteria for constructing a valid and reliable test that can be used for measuring psychological attributes, including nutrition knowledge (19). Parmenter and Wardle (20) focused on psychometric measures when developing their questionnaire to measure general nutrition knowledge among adults. Furthermore, they have developed a guide for evaluation and design of nutrition knowledge measures (15). Psychometric methods have been used by several in development of existing nutrition knowledge questionnaires (9, 11, 18, 20–23).
The aim of the present study was to develop a nutrition knowledge questionnaire with satisfactory reliability and validity for use among obese adults. The questionnaire should assess the level of general nutrition knowledge among obese adults about current dietary recommendations, sources of nutrients, everyday food choices, and assess the knowledge of different topics related to obesity. This instrument can be used in a survey setting but also as a starting point for tailoring educational programs in smaller groups and clinical practice. Since the nutrition knowledge can be variable among patients and group participants, mapping of preknowledge can make the teaching more relevant, focused, and effective.
Methods and results
The questionnaire development process consisted of four steps (Fig. 1): (1) preparation of scope and structure; (2) development of questionnaire items; (3) pilot study for further development of the questionnaire; and (4) test and retest by student groups for construct validity assessment and reproducibility.
Fig. 1.
Summary of steps followed in the development of the nutrition knowledge questionnaire for obese adults (figure inspired by Whati et al. 22).
The development and evaluation of the questionnaire was conducted in 2006–2007. Norwegian Social Science Data Services approved the study and The Regional Committees for Medical and Health Research Ethics approved the pilot study. Informed consent was obtained from all participants.
Step 1 – Preparation of scope and structure
A literature review of nutrition knowledge linked to weight reduction, dietary behavior, general dietary recommendations, and nutrition knowledge questionnaires was performed to define the scope of the questionnaire. In addition, telephone calls and personal meetings were held with professionals working in the area of dietetics and obesity to get in-depth information of the different aspects the questionnaire should cover. Based on this information, it was decided to divide the questionnaire into four main sections assessing knowledge about: (1) official dietary recommendations; (2) nutrient content in food items, with a main focus on macronutrients; (3) healthy food choices; and (4) meal pattern, energy intake, food labeling, and the food plate model. The first three sections were related to items about general nutrition knowledge. The fourth section was specifically related to obesity (such as calorific requirements, sugar contents of soft drinks, and skills of label reading). As a fifth section, demographic questions were included at the end of the questionnaire (age, gender, educational level, nutrition-related qualifications, occupation, height, and weight).
Step 2 – Development of questionnaire items
Based on the literature review and the four chosen categories, an item pool of 273 items was generated, mainly based on six existing nutrition knowledge questionnaires (9, (11, 16, 20, 23), and AKH de Soysa & S Severinsson, personal communication 2006).
Content validity refers to how representative the items are in covering the subject matter. The items should, as far as possible, sample the whole domain of the construct (16, 22). The items in the present knowledge questionnaire should cover essential aspects of general nutrition knowledge and conditions related to obesity and be phrased simply and unambiguously (20).
Face validity refers to how relevant and appropriate items appear to the respondents (15). Persons who have expertise with the target group or representatives of the target group itself are generally considered to be good judges of face validity (21, 24).
To ensure content validity and face validity, the item pool was evaluated by an expert panel of four dieticians who had experience with weight reduction groups. The experts received the item pool of 273 items and were requested to evaluate them for accuracy, appropriateness, and relevance related to measuring nutrition knowledge among obese. The expert panel selected 113 items from the item pool, and these items became the first questionnaire draft. The knowledge items were divided into three main types of answering options: agree/disagree (/not sure), multiple –choice, and more/less of different food items. These 113 items were then subjected to a second evaluation by the expert panel. The experts were in the second round asked to select items for adequate coverage of the knowledge area, interpretability of the food choices listed, and structure of the different questionnaire sections. Other complementary items were also asked for. In addition, one expert of questionnaire development was consulted to improve structure and layout.
After this review of the first draft, the answer category ‘not sure’ was included for items in sections for dietary recommendations and sources of nutrients. The expert panel review led to changes in 23 items, and the second draft of the questionnaire consisted of 98 items. Change of items included editing of items, removing, and adding new items. The second draft was tested for user friendliness and item ambiguity in a prepilot conducted by five students who had completed a course in general nutrition. Each questionnaire item was orally asked and the students discussed the answers. The discussion was taped. The prepilot led to changes in five items. After a new consultation with the experts, a third draft was finalized with 98 items.
Step 3 – Pilot study in target group for further development of the questionnaire
A pilot study was conducted to test whether the questionnaire was appropriate in a group of obese adults (n=33, BMI>35, age between 21 and 60 years). The respondents, similar to the target group, participated in a nutrition course prior to gastric bypass operation. Data were analyzed using SPSS version 14.0 for Windows (SPSS Inc., Chicago, IL, USA). The results were analyzed quantitatively for item difficulty and internal consistency and qualitatively by looking at the respondents’ comments on, e.g. the format of the questionnaire, the interpretability of the item, lack of important items, and time used for filling in the questionnaire.
Item difficulty refers to the proportion of respondents answering items correctly. Individual items should not be so easy that almost everyone completes them, nor so difficult that very few complete them (19, 20). The item analyses from the pilot study showed for example that all (100%) agreed that they should eat more vegetables and less sugar (94%). Few had the knowledge that olive oil (39%) and vegetable margarine (36%) contained a lot of fat. This was in accordance with the result that only 9% knew that it was the same amount of calories in butter and margarine. Only 36% disagreed on the statement that ‘oil contains less fat than vegetable margarine.’ Few (27%) also disagreed that dark chocolate contained less fat than milk chocolate. These results were concurrent with the comments from the participants in the prepilot test.
Kline (19) claimed that items are not useful if they are answered correctly by fewer than 20% or more than 80% of respondents (20). The limits of 30%–90% was chosen for this study to let less knowledgeable respondents be able to answer more items as a motivation factor for completing the questionnaire. Based on the pilot study, 22 items were answered correctly by more than 90% and five items by fewer than 30% of the respondents. Of the 27 items that did not meet the item difficulty criteria, 19 items were nevertheless retained to assess essential aspects of nutrition knowledge not covered elsewhere in the questionnaire.
Internal consistency refers to the homogeneity of the questionnaire. Individual items within each section or subsections should be well correlated to the total score of all sections (20, 25). Internal consistency was measured using Cronbach's α. Cronbach's α values range from 0 to 1, and a score of 0.7 or higher is generally acceptable (20, 25). All variables were recoded into 1 or 0, where 1=correct, 0=wrong, or 0=not sure. Cronbach's α was calculated for each of the three first sections (dietary recommendations, sources of nutrients, and healthy food choices) and for the three sections combined (Table 1). Cronbach's α varied from 0.51 in the section for healthy food choices to 0.84 for the three first sections combined (Table 1). The fourth section (factors that relate to obesity reduction) consisted of different types of items, as meal frequency, counting of calories, food labeling, and one would not assume any internal consistency between these items.
Table 1.
Internal consistency in pilot study of obese adults (n=33)
A further analysis of items in sections for dietary recommendations and healthy food choices was conducted because these sections did not reach the score of 0.7. This analysis showed for example that the items ‘How many servings of fruit and vegetables should we eat every day?’ (correct answer 67%) and ‘What would be best to drink when you are thirsty? Skimmed milk, orange juice, water or low calorie soft drink’ (correct answer 100%), would give their sections a higher Cronbach's α if deleted. Totally, there were 13 items that produced a higher α value for the total section when deleted; however, only three were removed. The rest was retained for the sake of content validity (22).
Respondents’ comments were made from 25 of the 33 respondents. Time used for filling in the questionnaire was between 10 and 45 min. Several of the participants commented that items regarding carbohydrates and proteins were difficult to answer. One remarked that ‘it is a bit tricky when the food options seem to be exactly the same.’ However, 10 respondents answered ‘no’ to the question ‘did you think that some of the items were difficult?’ The respondents were also asked to comment on positive aspects of the questionnaire. Representative remarks were ‘I think that the most important issues are covered’ and ‘we get an eye-opener about how little we care about diet, which is so important.’
The results from the measures of item difficulty and internal consistency from the pilot study and comments from the evaluation of the questionnaire were all considered before further changes were done. After consultations with the expert panel, the fourth draft of the questionnaire was finalized. Between the third and fourth draft, 18 items were changed and the fourth draft consisted of 92 items.
Step 4 – Test and retest study
A test–retest study was conducted with two groups of students; one group consisted of second year bachelor students in public health nutrition (n=16) and the other group consisted of construction students (n=18). The number of students who completed both test and retest was 27.
Construct validity refers to whether the combination of items in a specific construct provides a good measure (15, 22). The construct validity can be assessed in two groups, one which is known to have good nutrition knowledge, e.g. public health nutrition students and the other not e.g. construction students. The former group should score significantly higher, which was ascertained statistically using an independent sample's t-test (11, 20, 21). Table 2 shows the differences in knowledge between the public health nutrition and the construction students based on the first questionnaire test. The public health nutrition students had a significantly higher mean score in each section of the nutrition knowledge questionnaire (p<0.05) and for all sections together (p <0 .001) (Table 2).
Table 2.
Differences in knowledge scores between public health nutrition students and construction students in test (n=34)
| Public health nutrition students (n=16) | Construction students (n=18) | |||
|---|---|---|---|---|
| Knowledge section (max score) | Mean | SD | Mean | SD |
| 1. Dietary recommendations (11) | 10.1* | 1.5 | 9.0 | 1.4 |
| 2. Sources of nutrients (52) | 43.8*** | 4.0 | 34.3 | 5.1 |
| 3. Healthy food choices (19) | 16.1** | 2.2 | 13.6 | 2.1 |
| 4. Obesity reduction factors (10) | 8.2*** | 1.1 | 6.9 | 1.1 |
| Total (92) | 78.2*** | 5.4 | 63.8 | 7.4 |
Independent samples t-test, two-sided.
p<0.05
p<0.01
p<0.001.
The test–retest reproducibility of a questionnaire is the extent to which it produces the same results, when applied repeatedly in the same situation with the same persons (26). The time lag between the measures should be long enough for precise answers to be forgotten and short enough to minimize real change of knowledge (20). Pearson's correlation was used to measure the reliability between test and retest. It has been suggested that Pearson's correlation between scores of two tests should be at least 0.7 (15, 19). The time between the test and the retest was 3 and 4 weeks for the public health nutrition students and the construction students, respectively. Table 3 shows Pearson's correlation between the students who completed both the test and the retest. The correlation coefficients varied within sections from 0.53 (obesity reduction factors) to 0.85 (sources of nutrients). For all sections together, the correlation coefficient was 0.82. All correlations were significant for each section in the questionnaire (p<0.01) and for all sections together (p<0.001). After the retest, a total overview of the questionnaire was made by the expert panel. This resulted in three changes, and the final questionnaire consisted of 91 items (web link to questionnaire in Norwegian and a version translated to English: http://hist.no/content/33064/).
Table 3.
Test–retest reliability for public health nutrition students (n=13) and construction students (n=14)
| Test–retest reliability | |
|---|---|
| Knowledge section (max score) | (Pearson's r) |
| 1. Dietary recommendations (11) | 0.56** |
| 2. Sources of nutrients (52) | 0.85*** |
| 3. Healthy food choices (19) | 0.64*** |
| 4. Obesity reduction factors (10) | 0.53** |
| Total (92) | 0.82*** |
p<0.01
p<0.001
Discussion
In the present study, special attention was given to the development and testing of a nutrition knowledge questionnaire focusing on general dietary recommendations (27) and food choices to be used among obese adults.
Securing high-content validity was prioritized during the whole development process. The initial item pool consisted of items that already had been through a process considering reliability and validity (9, 11, 20, 23). New items were added to the questionnaire if the experts considered that certain aspects were not covered. Every draft was reviewed by the expert panel to select the best items in terms of clarity of the items, interpretability, and accuracy of the knowledge area being measured.
Separating the questionnaire into four main sections gives the possibility to assess both general and more specific nutrition knowledge related to obesity. The fourth section with specific items will also make the questionnaire more user friendly for other purposes. It is thereby easier for other users of the questionnaire to change the questions and direct the fourth section to for example diet–disease associations, fruit and vegetable intake, or diet recommendations for more specific groups.
It has been stated that without face validity, it is questionable whether the measure has content validity (15). In the current study, considerable effort was invested to ensure face validity. This was done both by taping and analyzing the discussion of all items and answers in the prepilot study,asking the respondents in the pilot study to comment on the content and design of the questionnaire, and noting misunderstandings according to comprehension (28).
Content validity and face validity must be assessed qualitatively, whereas construct validity can be measured statistically. The public health nutrition students (with extensive nutritional knowledge) scored higher on all sections of the questionnaire compared to the construction students.These significant differences signal that the questionnaire measures what it is supposed to measure, namely different dimensions of nutrition knowledge (18, 20, 23).
The sample size of both the pilot study (n=33) and test–retest (n=27) were relatively small. The reliability of the test depends on the sample size and test length (19). The reliability coefficient increases as the number of respondents and number of items increase. Separating the knowledge items of the questionnaire into smaller sections may have had an impact on the test results both for the internal consistency (Table 1), and the test–retest reliability of the questionnaire (Table 3). Measuring internal consistency for the pilot study, the sections for dietary recommendations (11 items) and healthy food choices (21 items), did not meet the score of 0.7. The internal consistency measure was highest for the sources of nutrients section (55 items), and the overall Cronbach's α of 0.84 for these three sections together was high. Similar results are found by others (9, 20, 21, 25). The test–retest reliability of the questionnaire was significant within all sections. The test–retest reliability for the sections dietary recommendations (11 items), healthy food choices (19 items), and obesity reduction factors (10 items) did not reach the correlation score of 0.7 (Pearson's r). The correlation scores for all knowledge sections together (92 items) were 0.82 (Pearson's r), indicating that the questionnaire, seen as a whole, measures nutrition knowledge consistently over time from one testing occasion to another. This corresponds with what others have found for their knowledge questionnaires (18, 20, 25).
Items, which are very intricate or very simple, may not differentiate between actual knowledge of individuals (22). In this study, several items that did not have consistency with the rest of the questionnaire and items that did not meet difficulty criteria were retained for the sake of content validity and only few were removed. All these items were presented theand if it is possible expert panel before adjustments were made. One item the panel chose to remove was: ‘What would be best to drink when you are thirsty?’ because the item was not regarded as essential for mapping of knowledge. The expert panel chose on the other hand, to keep the items about eating more vegetables, eating less sugar and servings of fruit and vegetables because all the items were closely related to the dietary recommendations from the government. Another illustration is that less than 30% of the respondents answered correctly to the statement: ‘Dark chocolate contain less fat than milk chocolate.’ A reason for the misunderstanding could be the strong promotion of dark chocolate as a healthy product in the media. The item was on these grounds regarded as important for mapping this aspect of nutrition knowledge.
This focus on content validity, by retaining items that did not meet the criteria for item difficulty and consistency, may influence the consistency of the questionnaire and hence the statistical result. However, the expert panel found it important to keep these items to be able to test essential aspects of nutrition (15, 16).
Conclusions and implications
This questionnaire was designed to assess nutrition knowledge among obese adults. The knowledge questionnaire had reasonable content-, face-, and construct validities and overall good reliability. The questionnaire should provide a useful tool for measuring nutrition knowledge among obese adults. We believe that this tool also could be used among other groups of adults with minor adjustments.
Acknowledgements
We acknowledge with great appreciation the contribution from the expert panel members Ann Kristin Hjelle de Soysa, Sara Severinsson, Britt Olufsen and Sigrun Henjum.
Conflicts of interest and funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
References
- 1.International Obesity Taskforce. Promoting healthy diets and physical activity: a European dimension for the prevention of overweight. obesity and cronic diseases. 2006. http://www.iaso.org/publications/iotfreports/ [cited 18 February 2011]
- 2.American Diabetes Association. Nutrition recommendations and interventions for diabetes: a position statment of the American Diabetes Association. Diabetes Care. 2008;31(Suppl 1):61–78. doi: 10.2337/dc08-S061. [DOI] [PubMed] [Google Scholar]
- 3.Goldstein LB, Adams R, Alberts MJ, Appel LJ, Brass LM, Bushnell CD, et al. Primary prevention of ischemic stroke. Circulation. 2006;113:873–923. doi: 10.1161/01.STR.0000223048.70103.F1. [DOI] [PubMed] [Google Scholar]
- 4.World Cancer Research Fund/American Institute for Cancer Research. Food, nutrition, physical activity and the prevention of cancer: a global perspective. Washington, DC: AICR; 2007. [Google Scholar]
- 5.Kelly T, Wang W, Chen C-S, Reynolds K, He J. Global burden of obesity in and projections 2005 to 2030. Int J Obes. 2008;32:1431–7. doi: 10.1038/ijo.2008.102. [DOI] [PubMed] [Google Scholar]
- 6.WHO. WHO Technical Report Series, No. 916. Geneva: World Health Organization; 2003. Diet, nutrition and the prevention of chronic diseases. Joint WHO/FAO Expert Consultation. http://www.who.int/dietphysicalactivity/publications/trs916/en/ [cited 18 February 2011] [PubMed] [Google Scholar]
- 7.Worsley A. Nutrition knowledge and food consumption: can nutrition knowledge change food behaviour? Asia Pac J Clin Nutr. 2002;11(Suppl):579–85. doi: 10.1046/j.1440-6047.11.supp3.7.x. [DOI] [PubMed] [Google Scholar]
- 8.Kearney JM, McElhone S. Percieved barriers in trying to eat healthier – result of a pan-EU consumer attitudinal survey. Br J Nutr. 1999;81(Suppl 2):133–7. doi: 10.1017/s0007114599000987. [DOI] [PubMed] [Google Scholar]
- 9.Boulanger PM, Perez-Escamilla R, Himmelgreen D, Segura-Millan MS, Haldeman L. Determinants of nutrition knowledge among low-income, Latino caretakers in Hartford, Conn. J Am Diet Assoc. 2002;102:978–81. doi: 10.1016/s0002-8223(02)90223-3. [DOI] [PubMed] [Google Scholar]
- 10.Burns CM, Richman R, Caterson ID. Nutrition knowledge in the obese and overweight. Int J Obes. 1987;11:485–92. [PubMed] [Google Scholar]
- 11.Klohe-Lehman DM, Freeland-Graves J, Anderson ER, McDowell T, Clarke KK, Hanss-Nuss H, et al. Nutrition knowledge is associated with greater weight loss in obese and overweight low-income mothers. J Am Diet Assoc. 2006;106:65–75. doi: 10.1016/j.jada.2005.09.047. [DOI] [PubMed] [Google Scholar]
- 12.Wing RR, Jeffery RW. Food provision as a strategy to promote weight loss. Obes Res. 2001;9(Suppl 4):S271–5. doi: 10.1038/oby.2001.130. [DOI] [PubMed] [Google Scholar]
- 13.Axelson ML, Federline TL, Brinberg D. A meta-analysis of food-related and nutrition-related research. J Nutr Educ. 1985;17:51–4. [Google Scholar]
- 14.Hendrie G, Cox D, Coveney J. Nutrition knowledge as a predictor of nutrient intake and diet quality. J Nutr Educ Behav. 2008;40(4):50–1. [Google Scholar]
- 15.Parmenter K, Wardle J. Evaluation and design of nutrition knowledge measures. J Nutr Educ. 2000;32:269–77. [Google Scholar]
- 16.Sapp SG, Jensen HH. Reliability and validity of nutrition knowledge and diet-health awareness tests developed from the 1989–1991 diet and health knowledge surveys. J Nutr Educ. 1997;29:63–72. [Google Scholar]
- 17.Wardle J, Parmenter K, Waller J. Nutrition knowledge and food intake. Appetite. 2000;34:269–75. doi: 10.1006/appe.1999.0311. [DOI] [PubMed] [Google Scholar]
- 18.Steenhuis IH, Brug J, van Assema P, Imbos T. The validation of a test to measure knowledge about the fat content of food products. Nutr Health. 1996;10:331–9. doi: 10.1177/026010609601000404. [DOI] [PubMed] [Google Scholar]
- 19.Kline P. London: Routledge; 1993. The handbook of psychological testing. [Google Scholar]
- 20.Parmenter K, Wardle J. Development of a general nutrition knowledge questionnaire for adults. Eur J Clin Nutr. 1999;53:298–308. doi: 10.1038/sj.ejcn.1600726. [DOI] [PubMed] [Google Scholar]
- 21.Steyn NP, Labadarios D, Nel JH, Robertson H-L. Development and validation of a questionnaire to test knowledge and practices of dietitians regarding dietary supplements. Nutrition. 2005;21:51–8. doi: 10.1016/j.nut.2004.09.008. [DOI] [PubMed] [Google Scholar]
- 22.Whati LH, Senekal M, Steyn NP, Nel JH, Lombard C, Norris S. Development of a reliable and valid nutritional knowledge questionnaire for urban South African adolescents. Nutrition. 2005;21:76–85. doi: 10.1016/j.nut.2004.09.011. [DOI] [PubMed] [Google Scholar]
- 23.Hawkes A, Nowak M. A validated nutrition knowledge questionnaire for cardiac patients. Aust J Nutr Diet. 1998;55:21–4. [Google Scholar]
- 24.Talmage H, Rasher S. Validity and reliability issues in measurement instrumentation. J Nutr Educ. 1981;13:83–5. [Google Scholar]
- 25.Hendrie GA, Cox DN, Coveney J. Validation of the general nutrition knowledge questionnaire in an Australian community sample. NutrDiet. 2008;65:72–7. [Google Scholar]
- 26.Klaver W, Burema J, Staveren WA, Knuiman JT. Definitions of terms. In: Cameron ME, Staveren WA, editors. Manual on methodology for food consumption studies. Oxford: Oxford University Press; 1988. pp. 11–23. [Google Scholar]
- 27.Nordic Nutrition Recommendations. 4th ed. Copenhagen: Norden; 2004. Integrating nutrition and physical activity; p. 436. [Google Scholar]
- 28.Anderson AS, Bell A, Adamson A, Moynihan P. A questionnaire assessment of nutrition knowledge-validity and reliability issues. Public Health Nutr. 2002;5:497–503. doi: 10.1079/PHNPHN2001307. [DOI] [PubMed] [Google Scholar]