Assessing Face Validity of a Food Behavior Checklist for Limited-resource Filipinos

Abstract

Diet-related chronic health conditions are prevalent in the Filipino American community; however, there is a lack of rigorously validated nutrition education evaluation tools in Tagalog for use in this population. This study aimed to develop and evaluate the face validity of a Tagalog-language food behavior checklist (FBC). A multi-step method was used, involving translation of questionnaire text from English to Tagalog by a team of professionals, creation of accompanying color photographs, cognitive testing with the target population, final review by the team of professionals, and assessment of readability. Subjects for cognitive testing were men (n=6) and women (n=14) 18 years or older in Hawai‘i who received or were eligible to receive Supplemental Nutrition Assistance Program (SNAP) benefits, self-identified as Filipino, and preferred Tagalog rather than English. Participants were recruited from churches, the Filipino Center, and other community sites. Cognitive interviews revealed several issues with text and photographs, such as preferences for specific terms, and images that did not adequately illustrate the text. Image changes were made to reflect items most commonly consumed. The team of professionals agreed with participant suggestions. Assessment of readability revealed a reading level appropriate for a low-literacy population of grade 5.9. The multi-step process, which allowed members of the target audience to reveal the appropriateness of the questionnaire, yielded a Tagalog-language FBC found to have adequate face validity. After further evaluation of validity and reliability, this tool may be used to evaluate behavior change resulting from the United States Department of Agriculture's (USDA) nutrition education programs.

Introduction

Asian-Americans are the most rapidly growing population of immigrants in the United States, and are a highly diverse group. Within this broad category, Filipino Americans are both the second-largest and the second-fastest-growing Asian subgroup in the United States.¹ Diet-related chronic health conditions are common in this group, with a recent review of rates of overweight and obesity among Asian-American subgroups revealing that Filipinos had the highest reported mean body mass index (BMI) of all Asian populations, at 26.8 kg/m(2).² Previous studies have shown that the Filipino-American population exhibits health disparities regarding conditions closely tied to dietary behaviors, including diabetes³ and hypertension.⁴ In Hawai‘i, rates of diabetes in the Filipino population (12.0%) are similar to those seen in the Native Hawaiians, and more than double that of the White population.⁵ Both adult and adolescent Filipinos in Hawai‘i are more likely than the general population to be overweight or obese.^6,7 Given the high rates of diet-related chronic conditions in this population, it is imperative to promote dietary habits that will lead to maintenance of a healthy weight and ultimately prevention of disease.

To meet the needs of low-income populations regarding nutrition education, the United States Department of Agriculture (USDA) administers several programs, such as the Supplemental Nutrition Assistance Program Education (SNAP-Ed) and the Expanded Food and Nutrition Education Program. These programs have demonstrated positive effects on nutrition behaviors of limited-resource populations across the lifespan.⁸ Evaluation of these programs is essential in determining effectiveness and providing justification for continued funding.^9,10 In conducting the evaluation, educators must have culturally-appropriate tools tailored to the target population in terms of language, reading level, and vocabulary.^9–11

A 2011 report from the US Census Bureau revealed that since the year 2000, the number of Tagalog speakers in the United States has increased significantly; moreover, of the total population of Tagalog speakers in the United States, 32.8% speak English less than “very well.”12 In Hawai‘i, where Filipinos represent 14.4% of the population, 21% speak English less than “very well.”13 Per capita income among Hawai‘'s Filipino population is 69% of the state average, and Filipinos represent almost a quarter (22%) of the state's Food Stamp Participants.¹³ To assess the impact of their programs on low-literacy Filipinos, nutrition educators must have evaluation tools in the language of the target population, using vocabulary common to this audience.

A review of the literature, however, reveals the absence of rigorously validated nutrition education evaluation tools in Tagalog for use in the Filipino population in the United States In examining studies reporting on the development of instruments to assess dietary intake in Filipino Americans, only one qualitative study was identified that focused on providing suggestions for modification of a tool designed for the general population.¹⁴ Focus groups were conducted with Filipino Americans to pinpoint problems with the content of the English-language food frequency questionnaire (FFQ) under study, and suggestions were gathered with regards to tailoring the tool. Focus group results indicated that three quarters of the Filipino-American study participants reported that the FFQ did not include food items that were commonly eaten in the Filipino-American culture.¹⁴ The results of this study provide preliminary evidence that dietary assessment tools designed to capture intake of the general population in the United States may not effectively characterize the diets of Filipino Americans.

One method of assessing whether a tool measures what it purports to measure is to assess its face validity. Face validity, as the name suggests, is a measure of determining to what degree a target population finds the tool to be reasonable and understandable at face value.^9,15–17 In outlining a systematic process for development and validation of evaluation measures for nutrition education programs, Townsend, et al, identifies assessment of face validity as an essential step in the first stage of developing evaluation tools and determining whether a tool is valid.⁹ To evaluate face validity, members of the target audience or those familiar with the target audience determine how practical and relevant the questionnaire is for the group.¹⁷ In previous studies seeking to develop tools focused on various aspects of diet and nutrition, face validity has often been assessed through the use of interviews with the target population to determine how appropriate the questionnaire content is for the group, as well as through expert assessment of reasonableness, appropriateness, and attractiveness.^18–22 While assessment of face validity allows for determination of acceptability for a target audience, it is also necessary to assess the correlation with other measures, as well as repeatability, making evaluation of face validity a preliminary step in questionnaire testing.¹⁵

The objective of the current study was to develop and assess the face validity of a food behavior checklist (FBC) in Tagalog using a multi-step method involving translation, creation of images to accompany the text, and techniques based on cognitive science.

Methods

Development of the English-language FBC that formed a basis for development of the Tagalog-language tool has been previously described.^10,11 This tool assesses diet-related practices such as consumption of sugar-sweetened beverages, fruits and vegetables, and low-fat dairy products, as well as overall diet quality and food security.

A multi-step process was used to develop and assess the face validity of the Tagalog-language FBC, as follows: (1) translation of text from English to Tagalog by a team of professionals; (2) creation of culturally-appropriate color photographs to accompany text; (3) cognitive testing of the draft instrument with the target population; (4) final review of questionnaire text by the team of professionals; (5) assessment of readability of text; and (6) development of an instruction guide for professionals. The criteria used to determine whether the tool demonstrated adequate face validity were: (1) approval of text and photographs by members of the target population; (2) approval of text and photographs by a team of Tagalog-speaking professionals; and (3) achievement of a readability score appropriate for a low-literacy audience. While there was no quantitative metric used with regards to the first two criteria, face validity was deemed acceptable when all suggestions from the target population and professionals had been addressed. The Institutional Review Board of the University of Hawai‘i at Manoa provided approval for this study.

Translation

A team-based approach was used to translate the English-language FBC^10,11 into Tagalog.²³ Three individuals who formed part of the team had the following qualifications: (1) graduate degree in nutrition or public health; (2) fluent in English and Tagalog; and (3) experience working in the limited-resource Filipino population. First, one professional generated a preliminary translation of the survey. The two other professionals then reviewed the translation and made comments in writing. A conference call was held with all three professionals, during which any discrepancies regarding the translation wording were resolved.

Creation of Photographs

The team of three bilingual health professionals also provided guidance regarding food preferences in the Filipino population and generated suggestions for color photographs of Filipino foods to replace the images on the English-language FBC and accompany the text. Suggested items were selected from local markets in Hawai‘i containing Filipino foods typically consumed in the population of interest. A graphic design consultant took color photographs using the selected foods reflecting choices of the Filipino population for each behavioral item. While many photographs on the English-language FBC were replaced, others were deemed acceptable for the new population and retained based on the decisions made by the team of health professionals.

Cognitive Testing of Instrument

Subjects

Subjects for cognitive testing were men (n=6) and women (n=14), 18 years or older who were low-income (demonstrated by receipt of or eligibility for SNAP benefits), self-identified as Filipino, and preferred Tagalog rather than English. A convenience sample was used as the most feasible manner of accessing this hard-to-reach population. Recruitment was conducted by a bilingual health professional at churches frequented by limited-resource Filipinos, as well as the Filipino Center on Oahu and other community sites. Thirty-two participants were screened to determine eligibility, and 20 (63%) were deemed eligible based on inclusion criteria.

Interview procedures

Informed consent was obtained from eligible participants. Prior to beginning the interview, participants were asked if they spoke other languages in addition to Tagalog, and this information was recorded. Cognitive testing procedures were then completed with members of the target population. Cognitive testing is a structured interviewing method used to determine whether the target population understands questionnaire items as scientifically intended.²⁴ The bilingual health professional who conducted recruitment received training in testing procedures from author no 1, who has had extensive experience conducting cognitive interviews in limited-resource populations. The trained professional then performed one-on-one interviews using an interview guide, which outlined a step-by-step procedure for testing of each item and photograph. Reflecting strategies developed by Willis, et al,²⁴ for each item, the interviewer used the following script: “Look at the question. Respond as you generally would. (Wait) Now go back to the question and tell me in your own words what the question means to you. Next, can you think of a better way to ask this question to make it clearer for another Tagalog speaker? Are there any words in the question that others may find confusing?” A similar script was used for the response options. For testing of the images, subjects were asked to describe what they saw in the photograph, and state whether there may be a better way to illustrate the question for other Tagalog speakers. Subjects were compensated with a $20 gift card for a local supermarket in exchange for their time.

Between interviews, the questionnaire was revised in an iterative process. This involved the interviewer (author no 2) and author no 1 reviewing participant suggestions periodically, making modifications to text and photographs based on feedback, and using the modified version of the questionnaire to perform further testing. Interviews were discontinued when no new suggestions for modifications to the instrument were suggested. Procedures used reflect standard methods previously employed in assessing the face validity of dietary assessment tools.^9,15–17

Final Review by Team of Professionals

At the conclusion of cognitive testing, the revised instrument was provided to a team of three professionals drawn from the community and fluent in English and Tagalog for finalization of the wording and photographs. The graphic design expert created a final set of photographs to be included on the tool based on results of cognitive testing and the professionals' comments.

Assessment of Readability

Tools for low-literacy audiences should be designed with text at about a 5th grade reading level.²⁵ Readability was assessed using the Spache Readability algorithm for Tagalog.²⁶ This formula takes the following into account to determine readability: (a) sentence length and (b) frequency of common words. Assessment of readability using this formula reveals the grade level of questionnaire text based on the US education system.

Development of Instruction Guide for Professionals

An instruction guide was developed for use by professionals administering the questionnaire as part of the USDA's nutrition education programs, using the same format as that of the existing instruction guides for the English- and Spanish-language food behavior checklists.^27,28 The guide is intended to assist professionals in answering any questions that may arise while the participant is responding to the items on the questionnaire, and contains the following information: (1) the text and images that correspond to each item; (2) a description of the items pictured; (3) an explanation of the information the item is intended to solicit; (4) potential questions from clients responding to questionnaire items; and (5) responses to potential questions that professionals may provide to participants.

Results

Participant Characteristics

Twenty individuals were eligible for study participation. All participants preferred Tagalog rather than English as a requirement for participation in the study, and 14 of the 20 individuals interviewed reported speaking other languages as well. Of these, eight spoke Ilocano, four spoke Kapampangan, one spoke Bikol and one spoke Visayan. No other demographic information was collected as part of the cognitive testing procedures.

Interview Findings

Cognitive interviews revealed several issues with text and photographs, such as preferences for specific terms, and images that did not adequately illustrate the text. Participant comments provided insight into habitually consumed foods as well as the most commonly used vocabulary to describe food items.

A number of changes were made to the text and images based on cognitive testing procedures. Interviews revealed preferences for the use of specific terms to refer to food items, and ways in which text could be reduced while still retaining meaning. For example, in referring to “habits”, the word gawi was replaced with ugali, the preferred term in the target population. Image changes were made to reflect items most commonly consumed in the target population and the form in which they are typically consumed. For example, for an item focused on fruit and vegetable consumption, carrots were replaced with Okinawan sweet potatoes, and cucumber slices were pictured with the skin rather than without. Examples of the most common issues identified during the cognitive testing sessions with low-income Filipino participants with regards to the text and images included in the Tagalog-language FBC are shown in Tables 1 and 2.

Table 1.

Issues identified related to text on the Tagalog-language questionnaire during cognitive testing sessions with low-income Filipino participants (n=20)

Original item/text	Translated item used for cognitive testing	Issues identified in cognitive testing interviews	Solution proposed by participants	Revised item
Food Behavior Checklist (Title)	Listahan ng mga pag-uugali sa pagkain	The word pag-uugali, meaning “behavior,” is not commonly used in everyday speech and is lengthy.	Participants suggested shortening the word pag-uugali to more common and concise version, ugali.	Listahan ng mga ugali sa pagkain
Do you eat 2 or more vegetables at your main meal?	Kumain ka ba ng mas mahigit sa dalawang klaseng gulay sa tanghalian o hapunan?	The word mas is redundant, as mahigit refers to “more than” when used alone.	Participants suggested removing the word mas.	Kumain ka ba ng mahigit sa dalawang klaseng gulay sa tanghalian o hapunan?
Do you use this label when food shopping?	Binabasa mo ba ang mga labels pag namamalengke ka?	The word “label” is ambiguous, and does not necessarily refer to the nutrition label.	Participants suggested adding the word “nutrition” to make it clearer.	Binabasa mo ba ang mga nutrition labels pag namamalengke ka?
Fruit: How much do you eat each day?	Prutas: Gaano karami ang kinakain mo araw araw?	Question is not clear when word prutas, meaning “fruit,” is mentioned first.	Participants suggested changing the order of the text to make it clearer.	Gaano karaming prutas ang kinakain mo araw araw?

Table 2.

Issues identified related to photographs included on the Tagalog-language questionnaire during cognitive testing sessions with low-income Filipino participants (n=20)

Original photo in English-language questionnaire	Photo used for cognitive testing interviews	Issues identified in cognitive testing interviews	Solution proposed by participants	Revised item
Original photograph for the item “Do you take the skin off chicken?” contained an image of someone taking the skin off of uncooked chicken.	The image of someone taking the skin off of uncooked chicken was retained.	Participants indicated that while skin is sometimes removed before cooking, it is also taken off of cooked chicken before consumption.	Participants suggested picturing removal of skin from chicken both before and after cooking to clearly convey the message that both should be counted in responding.	Revised photograph contains two images, one of someone removing the skin of the chicken before cooking and one of someone removing the skin from cooked chicken.
Original photograph for the item “Do you drink fruit drinks, sport drinks or punch?” contained an image of several beverages: Sunny Delight, Hawaiian Punch, Propel Fitness Water, Gatorade, Country Time Lemonade, Kool-Aid packets.	Photo contained some of the previous beverages and some new: Kool-Aid, Tang, Capri Sun, Powerade, Sunny Delight, Hawaiian Punch.	Clients indicated that while they were familiar with and consumed beverages pictured, there were also several others commonly consumed that should also be shown.	Participants suggested picturing several other beverages commonly consumed in the Filipino population in addition to the American beverages pictured.	Revised photograph contains two additional beverages commonly consumed in the Filipino population, so that items pictured are: Kool-Aid, Tang, Capri Sun, Powerade, Sunny Delight, Hawaiian Punch, Phillippine Mango Juice and Coconut Water.
Original photo for the item “Did you eat red meat or pork yesterday?” contained the following: 1) Plate of cooked beef strips with lettuce and tomato; 2) Plate of chunks of cooked beef with lettuce and tomato; 3) Tostada with ground beef, lettuce and tomato; 4) Tacos with shredded beef and salsa; 5) Plate of raw meat, including ground beef, steak, a pork chop and pig's feet.	Photo was simplified to contain a single plate of raw meat items: goat, beef, pork and liver.	Participants indicated that the meat items would be easier to identify if they were pictured as separate items.	Participants proposed presenting images of each meat item separately rather than all on one plate.	Revised photo contains fourse parate images of the followingitems: goat, beef, pork and liver.

Upon reviewing the questionnaire following cognitive testing procedures, the team of professionals agreed with participant suggestions. Assessment of readability of questionnaire text revealed a reading level of grade 5.9.

Results from cognitive interviews, the professional review, and readability assessment indicated that the questionnaire developed met the three aforementioned criteria for adequate face validity. The Tagalog-language tool resulting from all procedures reported is shown in Figure 1.

Figure 1.

Discussion

The current study rendered a Tagalog-language FBC found to have acceptable face validity in the limited-resource Filipino population and approved by an expert panel. In this study, face validity was deemed adequate based on the following: (1) approval of text and photographs by members of the target population via cognitive interviews; (2) approval of text and photographs by a panel of professionals using a team-based approach; (3) achievement of a readability score appropriate for a low-literacy audience. Through the use of systematic procedures, suggestions from both participants and professionals were incorporated into development of the tool, ensuring relevance to the target population. These procedures represent the first step in assessment of validity of the questionnaire, and must be followed by additional testing to further examine the psychometric properties of the tool.

As limited-resource Filipinos are at high risk of development of chronic conditions,^2–4 it is important to develop tools that assess behaviors that are associated with obesity and chronic disease. Filipino Americans have previously been shown to fail to meet the recommendations for intake of fruits and vegetables and other dietary components that are important to address for disease prevention.²⁹ Nutrition education is one way in which a change in habits may be promoted, and programs that have a focus on behavior change have been shown to be effective in altering dietary habits in diverse low-income populations.³⁰ The behaviors addressed in the FBC, such as increasing intake of fruits, vegetables, and dairy, may be targeted in nutrition education interventions geared toward improving intake and reducing risk of chronic disease. Unlike existing dietary assessment methods such as 24-hour recalls and food frequency questionnaires, the tool requires little time to complete and is easily administered in a group setting.⁹

Among the considerations for the design of such tools for a low-literacy audience is determination of the degree to which the target population comprehends the content. Of note, there have been no previous studies reporting the use of cognitive testing procedures to evaluate Tagalog-language tools, with the current study demonstrating the way in which such interviews may be utilized in the low-income Filipino population. These methods have previously been used to evaluate the face validity of instruments in other underserved populations, such as low-income Spanish-speaking audiences,¹⁶ and have revealed the need to modify instruments per the suggestions of the target population to reflect the preferred vocabulary and food items pertinent to the group in question. Cognitive testing interviews performed in the Filipino population in the current study similarly demonstrated participant preferences with regards to questionnaire content, yielding a tool developed in collaboration with the group in which the tool is to be used. These methods represent an important component of procedures used to develop dietary assessment tools, as they allow the researcher to uncover the thought process of the participant in responding to the items and ensure that the participant's understanding is in line with the scientific intent of the question. Further assessment of validity and reliability of the tool may reveal the need for further cognitive testing procedures to ensure items are best suited to the target population.

As a result of testing with the target population, both text and photographs were altered to reflect participant preferences. General issues uncovered were similar to those revealed in testing the FBC in the Spanish-speaking population.¹⁶ The Spanish-language tool was developed to address the needs of the large Hispanic population in the United States, and is currently in use in the USDA's nutrition education programs.^16,31 Cognitive testing in the Spanish-speaking population indicated the need for changes to photographs such as replacement of some food items originally included on the English-language checklist with others (ie, a can of milk powder instead of chocolate milk), inclusion of members of the target population in images (ie, a Mexican woman instead of an Asian woman consuming an apple), and changing the presentation of some items (ie, whole carrots instead of baby carrots).¹⁶ Similarly, cognitive interviews with Tagalog speakers led to the replacement of some items in the photographs (ie, sweet potato instead of grapes), and changing the presentation of some items (ie, half a papaya instead of chunks of papaya). The resulting tools contain minimal text and color photographs that illustrate item content to address the needs of low-literacy audiences.³²

The questionnaire resulting from the procedures used in the current study will be appropriate for use in limited-resource Filipinos who prefer Tagalog rather than English. Of note, while Tagalog is the official language of the Philippines, a variety of other languages are also spoken, including both Ilokano and Visayan. Tagalog is, however, the dominant Filipino language spoken in the United States,¹² ensuring that the questionnaire will reach the broadest segment of the population when used as part of the USDA's nutrition education programs.

In addition to selecting the appropriate language for the target population of interest, it is also important to assess the readability of tools developed to ensure they are at the reading level adequate for the group targeted. It has been suggested that the 5th grade level is appropriate for low-literacy audiences, lower than the eighth-grade level often recommended for a general audience.²⁵ In the current study, the tool developed was found to be at the grade 5.9 reading level, reflecting the recommendations. Of note, while readability of English-language instruments may be calculated using the Flesch Reading Ease formula,³³ the Tagalog language requires use of a different formula, the Spache Readability algorithm.²⁶ While the Flesch Reading Ease formula takes into account the number of syllables per word and number of words per sentence, the Spache Readability algorithm disregards syllable length. In assessing readability, it is important to select the correct tool for use, as components examined vary depending on the language in question.

Limitations

The Tagalog version of the FBC evaluated in this study was tested in a sample of Filipinos residing in Hawai‘i. Results may not be generalizable to Tagalog speakers residing in other parts of the United States. However, the team of translators included one professional from the US mainland, who had similar suggestions regarding questionnaire wording and photos to the professionals from Hawai‘i. In addition, the current study only examined face validity of the instrument, which is the first step in the validation process. To determine the acceptability of the tool for use in evaluating the USDA's nutrition education programs, further testing must be performed to determine the degree to which multiple administrations of the tool yield consistent results, and whether responses on the questionnaire adequately correlate with measures of dietary intake such as food records or serum indicators. These measures will allow for assessment of convergent validity and reliability. The current study represents a preliminary step in the validation of the checklist and must be followed by additional research.

Conclusions

The procedures described in the current study, including translation of text by a team of professionals, creation of color photographs, and cognitive testing with members of the target population, yielded an instrument measuring dietary behaviors found to have adequate face validity. This study represents the first step in development of a tool that will inform programs aimed at preventing diet-related chronic disease in the low-income Filipino population both in Hawai‘i and the continental US. To assess the degree to which this tool is correlated with other measures of dietary intake and demonstrates adequate consistency when administered at several time points, a subsequent study should be performed to assess criterion and convergent validity, as well as reliability. The rigorously validated FBC may be used in the USDA's nutrition education programs to set goals for sessions planned, as well as to evaluate behavior change resulting from participation in these programs. Programs may be tailored based on results from checklist administration to most appropriately address the needs of the low-income Filipino population and ultimately reduce health disparities.

Conflict of Interest

None of the authors identify any conflict of interest.