Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Dec 7.
Published in final edited form as: J Am Diet Assoc. 2010 Jan;110(1):80–90. doi: 10.1016/j.jada.2009.10.002

Using Qualitative Methods to Improve Questionnaires for Spanish Speakers: Assessing Face Validity of a Food Behavior Checklist

JINAN C BANNA 1, LUZ E VERA BECERRA 1, LUCIA L KAISER 1, MARILYN S TOWNSEND 1
PMCID: PMC4671629  NIHMSID: NIHMS741180  PMID: 20102831

Abstract

Development of outcome measures relevant to health nutrition behaviors requires a rigorous process of testing and revision. Whereas researchers often report performance of quantitative data collection to assess questionnaire validity and reliability, qualitative testing procedures are often overlooked. This report outlines a procedure for assessing face validity of a Spanish-language dietary assessment tool. Reviewing the literature produced no rigorously validated Spanish-language food behavior assessment tools for the US Department of Agriculture’s food assistance and education programs. In response to this need, this study evaluated the face validity of a Spanish-language food behavior checklist adapted from a 16-item English version of a food behavior checklist shown to be valid and reliable for limited-resource English speakers. The English version was translated using rigorous methods involving initial translation by one party and creation of five possible versions. Photos were modified based on client input and new photos were taken as necessary. A sample of low-income, Spanish-speaking women completed cognitive interviews (n=20). Spanish translation experts (n=7) fluent in both languages and familiar with both cultures made minor modifications but essentially approved client preferences. The resulting checklist generated a readability score of 93, indicating low reading difficulty. The Spanish-language checklist has adequate face validity in the target population and is ready for further validation using convergent measures. At the conclusion of testing, this instrument may be used to evaluate nutrition education interventions in California. These qualitative procedures provide a framework for designing evaluation tools for low-literate audiences participating in the US Department of Agriculture food assistance and education programs.

Production of valid and reliable evaluation and survey tools involves a complex process of ensuring adequate content, comprehension by the target audience, correlation with other measures evaluating the same constructs, and consistency over time (1). Although many studies report assessment of validity and reliability using quantitative techniques, relatively few discuss validation procedures using qualitative measures. Even fewer report collection of qualitative data in low-income Latino populations, where literacy level presents a challenge for educators.

A review of the literature found no rigorously validated Spanish-language food behavior evaluation tools to assess diet quality. Tools for low-literacy populations are especially lacking (2). Tools with a low respondent burden that can be administered in a group setting are needed (3–5). Tools that have exhibited adequate validity and reliability in a particular population need to be re-evaluated for use in another population that differs significantly in terms of cultural background, country of origin, or language (6).

Evaluation of US Department of Agriculture (USDA) nutrition education programs such as the Expanded Food and Nutrition Education Program and the Supplemental Nutrition Assistance Program (formerly known as Food Stamp Nutrition Education) is essential to ensure effectiveness of education interventions and continued funding (5). An ideal evaluation instrument for these USDA programs should exhibit adequate validity and reliability in the target population (5). In addition, it should be sufficiently brief to avoid detracting from the education portion of the intervention and should include key behaviors discussed in the education sessions (4,5,7). An appropriately low reading level is also an important characteristic for tools designed for low-income audiences (3). Predominantly Spanish-speaking Latinos, in particular, are often less educated than their counterparts in the United States and require instruments adapted to meet their literacy level (2).

There are currently no existing instruments used to assess food behaviors in low-literate Spanish-speaking populations; however, several short dietary assessment tools (ie, food behavior checklists or food checklists) have previously been developed to evaluate various other aspects of dietary intake. These include Townsend and colleagues’ Food Behavior Checklist (3,4,7), the National Cancer Institute’s 5-A-Day for better health fruit and vegetable screener (8), Kristal and colleagues’ Food Behavior Checklist (9), Connor and colleagues’ Diet Habits Survey (10), and Wakimoto and colleagues’ brief dietary screeners (short food frequency questionnaires) (2). Of these tools, two included cognitive testing procedures as an assessment of face validity (2,4), one used focus groups with the aim of ascertaining questionnaire acceptability in the target audience (9), and two made no mention of this step in the validation procedure (8,10).

The aforementioned food behavior checklist for low-income English speakers was previously developed as an evaluation tool for nutrition education interventions implemented through USDA programs (4,7). Using visual information processing theories, researchers improved the readability of the checklist, increasing its ability to accurately capture existing changes in dietary behavior (3,11). This tool underwent rigorous validation procedures in an English-speaking, low-income population, including assessment of criterion validity using a serum measure, convergent validity using multiple 24-hour dietary recalls, content validity and face validity, as well as internal consistency of subscales, test–retest reliability, and sensitivity to change (3,4,7). We sought to adapt this tool (12) and its accompanying instruction guide (13) for a Spanish-speaking, low-income population in California, an audience with high chronic disease risk (14). This article describes assessment of face validity of a Spanish-language diet quality food behavior checklist with a low-literate audience through a multistep method involving the use of a novel process of translation as well as techniques based on cognitive science.

These procedures may serve as a framework for other researchers collecting qualitative data for questionnaire validation for USDA’s education and food assistance programs where readability and response burden are primary concerns.

PROCEDURES

A bilingual native Spanish speaker performed five translations and suggested appropriate accompanying photographs. Extensive cognitive testing of text and visuals in the target population resulted in modifications. A bilingual panel of experts (n=7) in the field of nutrition of Latino populations, including Public Health Institute personnel, a registered dietitian, a bilingual University of California Cooperative Extension Advisor, and the university’s official translator, conducted a final review. The Institutional Review Board of the University of California at Davis provided approval for this study.

Translation

In lieu of the potential problems associated with the back translation method (6), we chose a three-step alternate method. First, a Mexican scholar with an advanced degree in nutrition visiting our university campus performed the preliminary translations. To create options for this audience, she created five translated versions of the original English checklist, deemed necessary due to the fact that five different translations were possible for many questions. These versions differed in terms of word choice (ie, expressing vegetables as verduras vs vegetales), phrases used to express desired concepts (ie, expressing regular soda as refrescos regulares vs refrescos no de dieta), as well as other aspects affecting question meaning. Second, these five versions were taken to members of our target population and tested for clarity, appropriateness and preferences of translation using cognitive interviewing procedures described subsequently (15). Third, after members of the target audience selected preferred translations, a group of experts provided final approval. They reviewed item wording and response options, taking into consideration optimal diction with regards to word length and frequency of use in the target population. Modifications were made, and a team of Public Health Institute Spanish-speaking experts then reviewed the current text.

Modification of Photographs

Color photographs included in the original food behavior checklist were modified or replaced to reflect food choices of Spanish speakers in California. The visiting Mexican scholar guided the selection of foods. Many items were simply replaced; others were presented in a different form (16).

Cognitive Testing of Text

Cognitive testing is a form of structured interviewing designed to improve face validity of a survey or evaluation tool (15). Willis (15) developed three strategies that may be used to uncover the cognitive processes that occur as respondents think about and develop answers to survey questions. The first is the concurrent think-aloud technique, in which respondents are asked to verbalize their thought processes as they respond to the question. This requires some practice, as the procedure does not come naturally. The second is the use of paraphrasing, which requires respondents to restate the item using their own words in response to this question: “What does this question mean to you in your own words?” For example, if the question is “Do you drink regular soda?” the paraphrase might be “This question is asking me if I drink soft drinks that are not diet.” The third strategy is the use of probes, a set of questions the interviewer uses to prompt respondents to further elucidate their responses. Examples of probes include, “Is there a better word we could use?” or, “Can you think of a better way to ask this question so that it would be clearer to other clients?” and, “Are there any words in the question that other clients may find confusing?”

Cognitive testing of other aspects of the questionnaire was also conducted. The aforementioned procedures were applied to the title, instructions, and response options.

Cognitive Testing of Photographs

Cognitive testing procedures were also applied to the photographs. Specific probes for the photographs included, “What do you see in this photo?” and, “Looking at this photo, is there a better way we might illustrate this question for other clients at this food bank?” Given that our goal was an improved readability score compared to a text only version, we also asked, “Are there any words in the question we might remove and illustrate in a photo?”

Cognitive testing sessions took place at the agencies where subjects were recruited. Interviews were mainly conducted in Spanish, and the interviewer (first author) took brief notes during the session. Interviews were generally one-on-one, with each one lasting 40 minutes to 1 hour. The interviewer referred to a pre-prepared interview guide to provide consistency among interviews (available from the first author). The interview process was iterative until all disparities and disagreements were resolved.

Subjects

Subjects for cognitive testing procedures were recruited from agencies serving Expanded Food and Nutrition Education Program and the Supplemental Nutrition Assistance Program. All were women who met the requirements for classification as low-income (ie, household income 185% or less of the US poverty income guidelines).

Instruction Guide for Field Staff

To address any questions that might arise during the administration of the checklist in the field, a comprehensive instruction guide was developed including checklist items, food products in each photograph, potential client inquiries regarding items, and answers to inquiries (17). After initial translation of the English guide with the help of the Mexican scholar, an expert panel of California Department of Health Services personnel reviewed the guide. Discrepancies were resolved by conference call.

Readability

Readability of questionnaire text was estimated using the Fernández-Huerta formula (18), the equivalent of the Flesch Reading Ease for English text (19). Both formulas consider word length and count the number of syllables per word. In addition, the Flesch formula considers number of words per sentence (19), whereas the Fernández-Huerta uses number of sentences per 100 words as an indicator of difficulty.

FINDINGS

Subjects (n=20) were recruited from Head Start sites, a food bank, a YMCA day care center, and university Cooperative Extension offices. All subjects were fluent in Spanish and reported Spanish as their first language. Most also spoke some English. The majority of subjects were of Mexican origin (95%) and all were women.

Cognitive testing was performed as an iterative process, with each interview building upon previous client suggestions. Cognitive interviewing procedures revealed several broad issues with the text and photographs (Figure 1). These sessions uncovered differences in language use among the Spanish speakers interviewed, possibly attributable to varying levels of acculturation and years spent in the United States. Client preferences for photographic content revealed client dietary habits, which are also sensitive to acculturation. An example of changes made to the photograph for one item using cognitive testing procedures is shown in Figure 2. This item, which assesses intake of fruit drinks, sport drinks and punch, originally contained a photograph of American drinks such as Gatorade (Quaker Oats Company, Chicago, IL) and Kool Aid (Kraft Foods Company, Chicago, IL). This photograph was replaced with predominantly Mexican drinks such as Jumex (Jumex Group, Mexico City, Mexico) and Del Valle (Coca-Cola Company, Atlanta, GA) in an effort to more accurately reflect intake of new immigrants. American soft drinks were replaced with Jarritos (Novamex, Guadalajara, Mexico) Mexican soda and Manzanita Sol (PepsiCo, Purchase, NY). During cognitive testing, participants reported consumption of not only Mexican beverages, but also American beverages. Clients preferred to have the final photograph contain a split cell with both Mexican and American beverages. Other changes to photographs included simple replacement of items based on client preferences. Fish sticks were replaced with a whole fish. Baby carrots were exchanged for carrot sticks. In addition, the Asian female model was replaced with a Mexican woman, and a Spanish-language version of the Nutrition Facts label was added. The result of these procedures was a tool that had been evaluated by the target audience and modified in an iterative process until clients expressed satisfaction with the outcome.

Figure 1.

Figure 1

Figure 1

Open in a new tab

Four issues identified with the Spanish-language food behavior checklist by respondents during cognitive testing sessions at community sites during a study to test validity of a low-literacy Spanish-language dietary assessment tool.

Figure 2.

Figure 2

Open in a new tab

Three versions of one evaluation item demonstrate changes made to photographs through cognitive testing interviews with low-income Spanish-speaking clients: original photograph of American fruit drinks (panel a), new photograph of mainly Mexican fruit drinks (panel b), and split photograph with both American and Mexican fruit drinks as final version of text and photograph per clients’ recommendations (panel c). (Figure was created by Jinan Banna, Marilyn Townsend, and Kathryn Sylva, and is used with their permission.)

A readability index calculation using the Fernández-Huerta formula for Spanish text (18) revealed a score of 93 (range 0 to 100), indicating low reading difficulty. The aforementioned accompanying instruction guide, used to resolve any questions that arise during the process of questionnaire administration, was developed to address the fact that the checklist was designed using very limited text (17).

DISCUSSION

The lack of standardized methods for questionnaire development and pretesting using qualitative methods is reflected in the absence of these procedures in many validation studies. A review of the literature did not reveal any studies that have used cognitive testing procedures for assessment of translations and visuals by Spanish speakers. Our study demonstrates an iterative process used to modify an evaluation tool based on client input. The example presented here may be used as a model for assessment of face validity of questionnaire text and visuals for low-income and often low-literate populations. The first phase of questionnaire development rendered a culturally adapted food behavior checklist that clients in our sample deemed to have adequate face validity (Figure 3) (20).

Figure 3.

Figure 3

Figure 3

Figure 3

Figure 3

Open in a new tab

Food behavior checklist (Lista de hábitos alimenticios) resulting from assessment of face validity in sample of low-income Spanish-speaking women in California. (Figure was created by Jinan Banna, Marilyn Townsend, and Kathryn Sylva, and is used with their permission.)

Translation

Several options were considered in determining the best method of translating the English-language food behavior checklist. Back translation is often the method of choice in adapting measures for culturally diverse audiences (6). This procedure involves allowing an original translator to render items from the current language to the second language and then asking a second translator to translate the items back into the first language. The original and back-translated versions are then compared, and any discrepancies in wording are resolved. While this method is frequently used, several potential problems may arise (6). First, translators who are aware that their work will be subject to back translation often use wording that would unquestionably reproduce the original translation instead of the most appropriate word in the target language (21). In addition, back translation does not allow for the substitution of item content for cultural reasons. In order to avoid these problems, a potentially more effective technique involving expert review has been suggested (6). This procedure involves translation by one party and subsequent review by a panel of experts in the field in question who are fluent in both languages and familiar with both cultures. This review may take the form of a group meeting or individual reviews by each of the group members (6). In lieu of the potential problems associated with back translation, we used an alternate three-step method consisting of translation by one party, cognitive interviewing, and final approval by a group of experts.

Readability

In considering the validity of a particular tool in a specific segment of the population, it is essential to consider the characteristics of the group in which the tool was validated. Many brief tools have been validated using a middle-class population with a relatively high level of education. Of the short dietary assessment tools mentioned previously (2,4,7–10), only two were validated in a low-income audience (2,4,7), and only one (4,7) mentioned the readability score of the instrument. For English text, the Flesch Reading Ease formula (19) is a well-known method for measuring readability. The Flesch Reading Ease formula indicates level of reading ease using the same scale of 0 to 100, with higher scores indicating greater reading ease. The Huerta Reading Ease (18) used in this study is the equivalent formula for Spanish text. Neither of these formulas assesses readability of photographs. Post hoc, the Flesch Reading Ease for English text (19) was applied, producing a score of 39.5 (also range 0 to 100). The contrasting results indicated that one of these two formulas was in error. We suggest that the error lies in applying the English text formula to the Spanish text. The difference in these scores explains the existence of different formulas for the two languages. Generally, translation into Spanish yields both longer words and sentences, necessitating use of a formula specifically designed for the Spanish language.

The inclusion of images in the questionnaire also reflected the effort to enhance understanding in this low-literacy population. Past studies have indicated that color photographs are favored by Expanded Food and Nutrition Education Program and the Supplemental Nutrition Assistance Program clients compared to black-and-white photographs, black-and-white drawings or text-only questions (3). Client reasons included improved understanding of the behaviors in the question (ie, cognitive function) (22,23), enhanced readability by reducing the word count (19), as well as more cues to understanding the text (ie, compensatory function) (24). The cognitive and compensatory functions of the color photographs were particularly important for clients whose primary language was not English (3).

Response Burden

One final consideration for tools intended for use with a low-income audience in a community setting is the ease with which they may be administered. Many existing evaluation methods such as the 24-hour recall and food frequency questionnaire require considerable amounts of time and effort on the part of the participant and the instructor, resulting in a loss of instruction time available in community classes (5). The current 16-item questionnaire may be self-administered in a group setting in a short time period (approximately 15 minutes) with the supervision of one community health worker per group, leaving ample time for the education session.

Limitations

The resulting version of the questionnaire is primarily appropriate for Spanish speakers of Mexican descent. The food behavior checklist evaluated here may not exhibit acceptable face validity beyond the sample of Spanish-speaking limited-resource women participating in this study.

Next Steps

To assess convergent validity of this food behavior checklist with other measures as well as reliability, further testing of the instrument is planned (Phase 2). Assessment of convergent validity will involve comparison of questionnaire responses to mean nutrient intakes from multiple 24-hour recalls, and reliability will be assessed through administration of the questionnaire on two occasions without an intervention.

CONCLUSIONS

Nutrition educators working in USDA education programs such as Expanded Food and Nutrition Education Program; the Supplemental Nutrition Assistance Program; the Special Supplemental Nutrition Program for Women, Infants, and Children; and Head Start may find the description of our process and our observations useful for developing instruments that are culturally appropriate.

Acknowledgments

FUNDING/SUPPORT: Funding for this study was provided by the USDA Food Stamp Program via the Network for a Healthy California, University of California Cooperative Extension, and the Gustavus and Louise Pfeiffer Research Foundation through the Program in International and Community Nutrition, UC Davis.

The authors thank Sharon Sugerman for her work on the study protocol and budget; Myriam Grajales-Hall and Gloria Espinosa-Hall for their work on questionnaire translations; and Kathryn Sylva and Lynn-Kai Chao for the questionnaire photographs. They also thank Andy Fourney, Monica Perez, Carmen C. Heredia, Ignacio Romero, Virginia Chaidez, and Joan Gonen for their assistance with the Spanish translations for the instruction guide, and Christine Davidson and Larissa Leavens for their review of the guide.

Footnotes

STATEMENT OF POTENTIAL CONFLICT OF INTEREST: No potential conflict of interest was reported by the authors.

References

  • 1.Lessler J, Tourangeau R, Salter W. Questionnaire design in the cognitive research laboratory: Results of an experimental prototype. Washington, DC: National Center for Health Statistics; 1989. Series 6, No. 1. [Google Scholar]
  • 2.Wakimoto P, Block G, Mandel S, Medina N. Development and reliability of brief dietary assessment tools for Hispanics. Prev Chronic Dis. 2006;3:1–12. [PMC free article] [PubMed] [Google Scholar]
  • 3.Townsend MS, Sylva K, Martin A, Metz D, Wooten-Swanson P. Improving readability for an evaluation tool for low-income clients using visual information processing theories. J Nutr Educ Behav. 2008;40:181–186. doi: 10.1016/j.jneb.2007.06.011. [DOI] [PubMed] [Google Scholar]
  • 4.Townsend MS, Kaiser LL, Allen LH, Joy AB, Murphy SP. Selecting items for a food behavior checklist for a limited resource audience. J Nutr Educ Behav. 2003;35:69–82. doi: 10.1016/s1499-4046(06)60043-2. [DOI] [PubMed] [Google Scholar]
  • 5.Townsend MS. Evaluating food stamp nutrition education: Process for development and validation of evaluation measures. J Nutr Educ Behav. 2006;38:18–24. doi: 10.1016/j.jneb.2005.11.008. [DOI] [PubMed] [Google Scholar]
  • 6.Geisinger K. Cross-cultural normative assessment: Translation and adaptation issues influencing the normative interpretation of assessment instruments. Psychol Assess. 1994;6:304–312. [Google Scholar]
  • 7.Murphy SP, Kaiser LL, Townsend MS, Allen LH. Evaluation of validity of items for a food behavior checklist. J Am Diet Assoc. 2001;101:751–756. 761. doi: 10.1016/S0002-8223(01)00189-4. [DOI] [PubMed] [Google Scholar]
  • 8.Thompson F, Kipnis V, Subar A, Krebs-Smith S, Kahle L, Midthune D, Potischman N, Schatzkin A. Evaluation of 2 brief instruments and a food-frequency questionnaire to estimate daily number of servings of fruit and vegetables. Am J Clin Nutr. 2000;71:1503–1510. doi: 10.1093/ajcn/71.6.1503. [DOI] [PubMed] [Google Scholar]
  • 9.Kristal A, Abrams B, Thornquist M, Disogra L, Croyle R, Shattuck A, Henry H. Development and validation of a food use checklist for evaluation of community nutrition interventions. Am J Public Health. 1990;80:1318–1322. doi: 10.2105/ajph.80.11.1318. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Connor S, Gustafon J, Sexton G, Becker N, Artaud-Wild S, Connor W. The Diet Habits Survey: A new method of dietary assessment that relates to plasma cholesterol changes. J Am Diet Assoc. 1992;92:41–47. [PubMed] [Google Scholar]
  • 11.Townsend MS, Sylva K, Martin A, Metz D, Wooten-Swanson P, Follett J, Keim N, Sugerman S. Visually enhanced evaluation for low-income clients. J Nutr Educ Behav. 2005;37(suppl):S49. doi: 10.1016/j.jneb.2007.06.011. [DOI] [PubMed] [Google Scholar]
  • 12.Sylva K, Townsend MS, Martin A, Metz D. Food Stamp Program Food Behavior Checklist (English, 16 items reflecting the Food Guide Pyramid guidelines; Visually enhanced food behavior checklist in 4-page booklet designed for clients with limited literacy skills) [Accessed September 1, 2008];Townsendlab Web site. http://Townsendlab.ucdavis.edu. Posted 2006. Updated July 9, 2008.
  • 13.Townsend MS, Davidson C, Leavens LL, Metz D, Martin A. Administering the Food Stamp Program Food Behavior Checklist: Instruction Guide. [Accessed September 1, 2008];Townsendlab Web site. http://Townsendlab.ucdavis.edu. Published 2006. Updated July 9, 2008.
  • 14.Heart Disease and Stroke Statistics—2005 [database online] [Accessed September 1, 2008];American Heart Association Web site. http://www.americanheart.org/. Updated October 15, 2008.
  • 15.Willis G. Cognitive Interviewing and Questionnaire Design: A Training Manual. Hyattsville, MD: Centers for Disease Control and Prevention, National Center for Health Statistics; 1994. Working Paper Series No. 7. [Google Scholar]
  • 16.Levie W, Lentz R. Effects of text illustrations: A review of research. Educ Comm Technol J. 1982;30:195–232. [Google Scholar]
  • 17.Banna JC, Townsend MS, Vera Becerra LE, Fourney A, Perez M, Heredia C, Romero I, Grajales-Hall M, Davidson C, Leavens LL. Administering the Food Stamp Program Lista de hábitos alimenticios (Food Behavior Checklist) for Spanish speakers: Instruction Guide. English and Spanish. [Accessed September 1, 2008];Townsendlab Web site. http://Townsendlab.ucdavis.edu. Published 2008. Updated July 9, 2008.
  • 18.Fernández-Huerta J. Medidas sencillas de lecturabilidad. Consigna. 1959;214:29–32. [Google Scholar]
  • 19.Kincaid J, Fishburne RP, Rogers RL, Chissom BS. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Millington, TN: Naval Technical Training, US Naval Air Station; 1975. [Google Scholar]
  • 20.Banna JC, Townsend MS, Sylva K. Food Stamp Program Lista de hábitos alimenticios (Food Behavior Checklist) for Spanish speakers (Spanish, 16 items reflecting Food Guide Pyramid guidelines; visually enhanced food behavior checklist in 4-page booklet designed for clients with limited literacy skills) [Accessed September 1, 2008];Townsendlab Web site. http://Townsendlab.ucdavis.edu. Published 2007. Updated July 9, 2008.
  • 21.Hambleton R. Technical standards for translating tests and establishing test score equivalence. Presented at 101st Annual Convention of the American Psychological Association; Toronto, Ontario, Canada. 1993. [Google Scholar]
  • 22.Sweller J, Van Merrienboer J, Paas F. Cognitive architecture and instructional design. Educ Psychol Rev. 1998;10:251–296. [Google Scholar]
  • 23.Mayer R. Multimedia Learning. New York, NY: Cambridge University Press; 2001. [Google Scholar]
  • 24.Berry L. An Exploratory Study of the Relative Effectiveness of Realistic and Non-realistic Color in Visual Instructional Materials [dissertation] State College, PA: The Pennsylvania State University; 1974. p. 7717a. [Google Scholar]

RESOURCES