Formative research in a school-based obesity prevention program for Native American school children (Pathways)

Abstract

This paper describes how formative research was developed and implemented to produce obesity prevention interventions among school children in six different Native American nations that are part of the Pathways study. The formative assessment work presented here was unique in several ways: (1) it represents the first time formative research methods have been applied across multiple Native American tribes; (2) it is holistic, including data collection from parents, children, teachers, administrators and community leaders; and (3) it was developed by a multi-disciplinary group, including substantial input from Native American collaborators. The paper describes the process of developing the different units of the protocol, how data collection was implemented and how analyses were structured around the identification of risk behaviors. An emphasis is placed on describing which units of the formative assessment protocol were most effective and which were less effective.

Introduction

Earlier in this century, heart disease and diabetes were rare in Native Americans (Gilbert, 1955; Smith, 1957; Clifford et al., 1963; Fulmer and Roberts, 1963; Sievers, 1967; Knowler et al., 1983; Welty, 1991). In recent years, the prevalence of both diseases has increased dramatically (Rhoades et al., 1987; Becker et al., 1988; Alpert et al., 1991) and heart disease is now the leading cause of death in this population (Sievers and Fisher, 1979; Klain et al., 1988; Young et al., 1993). Being overweight in childhood may impose a risk for obesity in adulthood (Frerichs et al., 1979; Casey et al., 1992; Serdula et al., 1993; Guo et al., 1994), which in turn may increase the risk for heart disease and other chronic diseases (Johnston, 1985; Clarke et al., 1986). Several studies have shown a secular increase over the past three to four decades in overweight and obesity in Native American school children with prevalence estimates ranging from 19.5 to 33% of children being obese (Sugarman et al., 1990; Broussard et al., 1991; Knowler et al., 1991; Gilbert et al., 1992). In this population, then, efforts to establish healthy lifestyle behaviors should begin in childhood.

The Pathways study is a two-phase, multi-site clinical trial to develop and test a school-based intervention to prevent obesity in Native American school children, grades 3–5. The planning and feasibility phase was initiated in October 1993. Pathways involves six Native American nations: White Mountain Apache, Tohono O’odham, Gila River (Arizona); Oglala Lakota, Sicangu Lakota (South Dakota); and Navajo (New Mexico/Arizona); and five universities: Johns Hopkins University, The University of Arizona, The University of Minnesota, The University of New Mexico and The University of North Carolina.

Developing an effective, multi-culturally appropriate school intervention program to prevent obesity in six different Native American communities is perhaps the greatest challenge Pathways has faced. The key is developing an intervention and measurement instruments that are acceptable and may be implemented in a standard manner across all sites. To this end, a formative assessment was conducted in the early stages of the study.

The goals of the Pathways formative assessment were to provide information to:

1. Select, design, implement and evaluate culturally appropriate interventions to reduce the prevalence of obesity among school children in grades 3–5 in selected schools on reservations.

2. Develop a baseline qualitative description of the study communities and schools.

3. Respond directly to parent, teacher and community needs regarding the health of their children.

4. Provide information that will assist in developing instruments for structured data collection and for the process evaluation.

This paper describes the development and contents of the formative assessment, the types of information produced, and lessons learned. The approach used may serve as a model for others who are developing intervention programs or services for different cultural or ethnic groups.

The formative assessment approach

Formative assessment studies use social science techniques to assess people’s beliefs, perceptions and behaviors for the purpose of developing culturally appropriate interventions. In addition, they seek to describe the context in which these behaviors take place and to understand why people do what they do. Several studies have documented how this information can be used by health program planners to enable them to anticipate reactions to interventions and better adapt programs to local conditions (Atkin et al., 1991; Helitzer-Allen et al., 1993a; Davis, 1994; Gittelsohn et al., 1994; Lloyd et al., 1994; Young, 1994; Gittelsohn et al., 1996). Such adapted programs may be more effective in changing human behavior and improving health status.

Formative assessment studies employ a combination of qualitative and quantitative data gathering methods (Helitzer-Allen and Kendall, 1992; Gittelsohn et al., 1995). The unique advantage of unstructured qualitative data collection methods (i.e. focus groups, in-depth interviews and direct observations) over surveys and questionnaires is that the non-directive process allows subjects considerable opportunity to comment, explain and share their experiences, attitudes and perceptions. The opportunity for subject interaction generates discussions that reveal a greater depth of understanding about why certain opinions are held (or behaviors performed) than is possible with more quantitative methods. The analysis of qualitative data can suggest strategies that might be particularly effective with the target audience. Formative assessment approaches allow solutions to arise out of the data, rather than applying predetermined concepts or solutions (Glaser and Strauss, 1967). Pathway’s formative research involved initial, exploratory qualitative data gathering, followed by more structured, quantitative data gathering. Quantitative information helped us pinpoint and prioritize key features relating to local perceptions and behaviors.

It was also found that conducting a formative assessment in a community intervention serves an additional purpose: to build positive anticipation and support for the resulting program. Participants began to view the Pathways program with a sense of ownership because their opinions guided its development and implementation. Formative research can also foster positive attitudes toward researchers who take the time to ask for and respect the opinions of the target population rather than impose their opinions. It can serve as a reality check to force program developers out of self-reinforcing cycles. Many program developers are eager to test interventions which have been shown to be effective in one population. They then ask questions relating exclusively to these experiences, rather than maintain an open mind about potential new factors in a new setting.

As with all data collection methods, formative research has limitations (Helitzer-Allen, 1993a,b; Atkin and Freimuth, 1991). Compared to epidemiological studies in general, formative research studies tend to have fairly small samples. Frequently respondents are selected on the basis of expertise and special knowledge, rather than using random representative sampling techniques. These aspects of data collection reduce the generalizeability of findings to the general population. Yet, as we hope to demonstrate, these limitations are outweighed by the advantages of the approach and the usefulness of the information generated.

Description of the formative assessment protocol

This section briefly describes the Pathways formative assessment protocol, including the development process, format, components, training and data management and analysis.

Development process

The protocol was developed over several months in multiple stages. Initial drafts and data collection units were developed by a working group composed of representatives from each of the field sites, the coordinating center, and the National Institutes of Health, National Heart, Lung, and Blood Institute. This working group included representatives from two of the six Native American nations that are part of the Pathways study. The primary organizing structure for the design of data collection instruments was a series of programmatic questions on planning effective and culturally appropriate interventions raised by Pathways investigators (Table I). Ongoing requests for information from other Pathways intervention working groups, including those working on the development of specific interventions (food service, curriculum, family and physical activity) and data collection instruments were also incorporated into the design. Drafts of the protocol were circulated to reviewers outside the formative assessment working group, including representatives from all participating Native American nations. Every effort was made to develop methods and instruments that would yield useful information in a culturally sensitive manner. All components of the protocol were pre-tested at one or more of the study sites and further modified.

Table I.

Some programmatic question! addressed in the formative assessment

Intervention area	Programmatic question
Classroom curriculum	What is the existing curriculum in the school for health and nutrition education? What do school officials and teachers feel are the current needs in terms of the school curriculum? What are the existing classroom resources which would inhibit/assist with implementation of a new curriculum? How should teachers and teacher’s aides be trained in the use of the curriculum?
Physical education and activity	What physical education/activity resources (i.e. facilities, teachers, equipment, dedicated time, etc.) are currently available at the schools? What are the constraints (including time availability)/acceptability for integrating daily physical activity in schools? How can a new program be made sustainable after the study ends?
School food service	What is the management and purchasing system in the school meal program? What foods are presently being served and how often? How well are these foods being accepted? How can foods presently being served be modified? What new healthy foods are economically, socially and culturally feasible for introduction to children’s diets? How do food service personnel need to be trained (in terms of food preparation, serving, etc.)? What foods are available to children outside of those provided by school food service (e.g. vending, nearby stores, etc.)?
Family/home	What foods do children eat at home, school and elsewhere? Who decides what foods are prepared for meals at home? Who decides how much or how often the child should eat at home? What is the child’s role in these decisions? Who buys food in the household? Where? How often? What communication concerning food, health and nutrition goes on between parents and children? What are existing family activity patterns? What are the recreational facilities in the community?
Grocery stores, food sellers	What are all the different food stores or markets that service the community? Which stores are utilized by school children? How are they utilized? What kinds of foods are being sold in these stores? What types of fruits and vegetables are available, both in terms of quality and quantity? What types of snack foods are available? What kinds of lower fat snack foods are available?

Format

The formative assessment protocol used a variety of different qualitative and quantitative methods to gather information: in-depth interviews, semi-structured interviews, focus groups and direct observation. Data collection for the formative assessment moved between several types of respondents: school staff (i.e. teachers, food service workers and administrators), grades 3–5 students and their caregivers, and other community members.

The protocol is divided into a series of nine units. Each unit contains three sub-headings: purpose, data collection and data analysis. The statement of purpose describes the types of information that will be collected in the unit, as well as the overall goals and use of the information. These purposes are directly linked to the key programmatic questions (shown in Table I).

The data collection subsection of each unit contains the following elements: (1) guidelines on how to prepare data collection materials; (2) how to locate and select respondents; (3) suggested sample size, usually as a range; (4) suggestions on how to word or present questions to respondents; (5) blank data collection forms and interview/focus group/observation guides; and (6) how to record responses on data collection forms.

The section on data analysis contains some or all of the following types of information: (1) how to expand raw field notes into text files (i.e. what to include, degree of detail, etc.); (2) guidelines for detecting trends, perceptions and patterns; and (3) sample methods of tabulating and presenting the data.

The purposes, data collection methods and respondents used for each method are summarized in Table II. Several of the units were split into two subunits as the protocol evolved. For instance, Unit 7a focus groups with teachers were followed up with individual semi-structured interviews with teachers (Unit 7b). The follow-up data collection units tend to be more structured and quantified, focusing on specific themes and behaviors that emerged from a less structured, qualitative mode of information gathering.

Table II.

Data collection units in the formative assessment: purpose, methods used and sampling

Unit/purposes	Data collection methods	Respondents
1: Direct observations in the community
Determine community resources (human and physical)	Direct observation	Community leaders
Gather up-to-date information on the study community	Semi-structured interviews
2: In-depth interviews with school officials and food service personnel
Provide preliminary information on school resources, including classrooms, food service, PE facilities	In-depth interviews	School officials
Assess curriculum structure of study schools, feasibility of changes/ modifications to curriculum, openness to change		School food service staff
Understand process for ordering, preparing and serving food
Learn how current food service system can be modified to encourage school children to eat healthfully
Learn about the ‘informal culture’ of food service in school, including how foods are served to children
Assess potential barriers to change
3a: Direct observation in schools
Obtain information on children’s physical activity patterns, food consumption and classroom behaviors Observe patterns of food serving, preparation in school Observe how children are currently being taught about health (including physical activity and physical education) and nutrition	Direct observation kitchen cafeteria recess PE regular classroom	Actors: children, food service staff, teachers
3b: Observation of school events
Obtain information on types, price, source of foods sold or distributed to children during school events	Direct observation school games carnivals	Actors: children, food sellers
4: Paired child interviews: talking to children about food activity and health
Explore aspects of children’s beliefs and behavior regarding common foods they eat and language they use to talk about these foods	In-depth interviews Free listing	Pairs of children grades 3–5
Explore attitudes and perceptions about weight, physical activity, hunger and satiety
Identify preferred learning activities in classroom settings
5: Child interviews: how children group foods
Determine how children categorize and group different foods	Semi-structured interviews	Children, grades 3–5
Estimate frequency of consumption of commonly consumed foods	Pile sorts
6: Focus groups with child caregivers: identifying cultural norms
Determine normative values and patterns of food preparation and consumption that affect children’s dietary intake in households	Focus groups	Children’s parents and other caregivers
Explore parent’s perceptions of how children should be/are brought up in the community
Explore parent’s perception of their role and the role of the family in changing diet and activity in children
7a: Focus groups with teachers and teachers aides
Identify health curriculum currently being used in the classroom	Focus groups	Teachers and teacher’s aides
Explore how curricula could be incorporated into the teaching
How they like to receive information in terms of training and manuals
Explore options for communicating information to the parents
Identify perceived barriers to successful implementation of Pathways
Describe personal beliefs/attitudes about eating habits/physical activity
7b: Follow-up interviews with teachers
Obtain more specific information on media used in classroom teaching, role-modeling during meals, etc.	Semi-structured interviews	3–5 grade teachers
8a: Follow-up interviews with parents and other community members: specific household beliefs and behaviors
Determine existing beliefs and behaviors of individual parents (and other caregivers) regarding weight and their children Assess adult household member’s interest, knowledge and attitudes concerning their own child’s dietary intake and physical activity levels Identify adults personal beliefs and reported behavior regarding healthy eating and physical activity Determine willingness of parents to attend/work on extra-school health activities (e.g. ‘family night’ programs, fitness fairs, health fairs, etc.).	In-depth interviews	Caregivers of children in 3–5 grades
8b: Follow-up interviews with caregivers
Obtain information on specific child care and feeding behaviors, such as meal patterns within the household, uses of food for comfort and reward, etc.	Semi-structured interview	Child caregivers
9: Observation of food purchasing and consumption by children
Determine availability, costs and varieties of different foods Develop a preliminary, quantified estimate of children’s food selection and purchasing patterns	Direct observation local stores	Actors: children, parents

Identifying and training of data collectors

At least two representatives from each field site were trained to conduct the formative assessment. Of these two, at least one was a Native American. The trainings were organized by Pathways staff experienced in formative research methods. Trainings were conducted on three separate occasions, immediately prior to each of the three data collection phases (see below). Each training session lasted approximately 2 days (approximately 6–8 h per day), and involved didactic presentation, demonstration and role play. For certain units of the formative assessment protocol, trainees were able to practice performing the interviews on test subjects and observations, and later discuss experiences. Videotapes of portions of the training were made for future reference and to improve standardization of data collection procedures. Feedback on quality of data collection was provided by the Johns Hopkins team.

Sequence of data collection

Data collection proceeded in three phases: (1) school-focused (Units 2, 3a, 4 and 7a), conducted March–June 1994; (2) community/home-focused (Units 1, 6, 8a and 9), conducted July–August 1994; and (3) assessment of obesity risk behaviors (Units 3b, 5, 7b and 8b), conducted September–November 1994. The timing of these three phases was primarily related to logistic issues. The first phase took place prior to the end of the 1994 school year, while the second phase was put off until school-based data collection had been completed. The third phase was an expansion of phases one and two, and focused on identifying specific behaviors that appeared to be contributors to childhood obesity across sites.

Data management and analysis

The formative assessment resulted in two types of data: (1) textual and (2) numeric. At each site, textual data (from field notes and taped information) were expanded or transcribed and entered into the computer on WordPerfect Selected numeric data were tabulated and recorded in the appropriate forms and matrices (provided in each of the units). All textual and numeric data were sent to Johns Hopkins University for coding and tabulation. Field sites provided summaries of key information in the form of completed priority question matrices. The data collected in the priority question matrices from each site were combined to give cross-site summaries. The textual and numeric data were also used as a reference database of information for intervention development. The textual data were coded, managed and analyzed using the software package GOfer 2.0 (Microlytic, 1989) according to the methods described in Gittelsohn (1992). Selected numeric data (including the child food pile sorts) were analyzed using ANTHROPAC 3.2 (Borgatti, 1990). Other quantitative data were hand-tabulated. A complete set of data and analyses were archived at the coordinating center.

One of the primary analyses was the identification and prioritization of a set of obesity risk behaviors to be used to structure and focus intervention strategies (Table III). A preliminary list of potential risk behaviors was identified through mid-term review of the existing formative assessment data, and from suggestions from field personnel and intervention working groups. The initial list of behaviors included positive as well as negative behaviors. Positive behaviors were rephrased negatively in order to make the statement of behaviors consistent. Additional data were gathered on potential risk behaviors at the different sites using semi-structured interview techniques (in Units 3b, 7b and 8b). This list was then refined and prioritized through a three-stage process: (1) a team of four researchers conducted analyses of the entire formative assessment database to assign rough prevalence scores for each risk behavior (i.e. four levels indicating the evidence that risk behavior does or does not exist in the community) for which there was sufficient data to make a determination; (2) individual site field teams (including the Native American staff) reviewed the scores and offered input regarding potential changes; and (3) third phase risk behavior data were collected and analyzed to confirm the prevalence of particular risk behaviors.

Table III.

Prioritization of risk behaviors based on formative assessment data collection

Priority/behavior	Category
	Home	School	Diet	Activity
High
little family-based activities (especially outside home)	✓			✓
children watch (a lot of) TV, movies, VCR	✓			✓
little home-based role modeling for physical activity	✓			✓
children drink (a lot of) sugar drinks at home/community	✓		✓
eating high fat foods at home meals	✓		✓
parents encourage children to finish all their food	✓		✓
not enough physical activities for individuals in school		✓		✓
a lot of TV watching in class		✓		✓
eating high fat foods at school meals		✓	✓
consumption of seconds of high fat foods at school		✓	✓
Moderate
children drink (a lot of) whole milk at home/community	✓		✓
eating (a lot of) fast foods outside the home	✓		✓
eating a lot of high fat snacks at home	✓		✓
foods high in fat and sugar used as rewards at home	✓		✓
children drink (a lot of) whole milk at school		✓	✓
foods high in fat and sugar used as rewards at school		✓	✓
high fat/sugar foods offered at special events at school		✓	✓
does not follow existing school recipes		✓	✓
teachers encourage children to finish all their food		✓	✓
eating a lot of fried foods at school		✓	✓
not consuming enough fruits or vegetables at school		✓	✓
Low
family does not eat main meal together	✓		✓
food is used for comfort at home	✓		✓
not consuming enough fruits or vegetables at home	✓		✓
not enough general encouragement to do physical activities in school		✓		✓
children drink (a lot of) sugar drinks at school		✓	✓
positioning of low fat milk at school makes it less accessible		✓	✓

Selected results

The formative assessment data collection yielded an extremely rich database, which is currently being used to develop Pathways interventions in the classroom, physical education/recess, food service and homes, and to improve measurement methods. Tables IV–VI present brief examples of the kinds of information collected. Data are presented in summary tables by site and school within site. Tribal and school names have been omitted to protect the confidentiality of schools, students and communities.

Table IV.

Selected information from formative assessment concerning classroom curriculum

Attribute	Site 1		Site 2		Site 3		Site 4
	School A	School B	School C	School D	School E	School F	School G	School H
Type of health education in existing curriculum	No formal health curriculum. PE teacher uses a variety of sources. Some teachers teach health, based on their own experience.	Up to the individual teacher. Some teachers use ‘out-of-date’ health book…‘some are not touching on health’.	Taught by school nurse. They follow state guidelines for AIDS curriculum. Use Growing Healthy issues and the Food Pyramid.	PE teacher teaches about health, nutrition, aerobics, AIDS for grades 5–8. Classroom teachers teach same for grades 1–4.	Health curricula in library; health books, work books, challenge sheets (one set shared among 12 classrooms). Also Jump Rope for Heart (AHA).	Substance abuse curriculum: ‘BABE, DARE, BEAUTY, WAY’. Incomplete curriculum kits.	Growing Healthy: a developmental program.	Health curriculum taught last 3 months of school in place of science; twice a week for 30–45 min
Process for modifying curriculum	Teachers make recommendations; School Improvement Team contributes. Large-scale modifications must be approved by school board.	No specific process. Principal plans to make changes herself over summer. Teachers will review it with her.	Must go through board action.	Curriculum committee, then reviewed by school board.	BIA Chinle Agency curricula committee.	Grade level teacher to school committee/principal to district-wide committee to school board.	Teacher training, followed by implementation.	Teachers should be included in curriculum development. They do not like ones just handed to them.
Types of classroom activities that interest students	Puzzles, word games, contests, things that challenge, group work, continuous rewards, activities that incorporate their interests/way things are at home.	Hands-on activities, things on computer, teamwork (work better because responsible for others, getting physically involved, children learn when they teach other children).	Hands-on experiences, film strips, video.	Food, a point system, reward system, instant gratification.	Hands-on, limited lecture, highly visual.	Hands-on, show and do, limited lecture, activities that apply to lessons, problem-solving strategies.	Films, games, sharing sessions; hands-on activities, group activities and games.	Those that involve student participation, e.g. making applesauce.

Table VI.

Selected information from formative assessment concerning physical activity

Attribute	Site 1		Site 2		Site 3		Site 4
	School A	School B	School C	School D	School E	School F	School G	School H
Amount of PE time currently	3–4 45-min sessions/week (PE teacher).	1–2 45-min sessions/week (regular class teacher).	1–3 30-min sessions/week.	2–3 45-min sessions/week.	No PE for grade 3. 1 50-min session/week grades 4 and 5.	1 45-min session/week (focussed on skill development)	3 40-min sessions/week	15 min a.m. 20 min p.m.
Teacher interest in being involved in PE of students	Varies. Some teachers (own initiative) conduct organized physical activities	Varies. Some teachers already do PE (informally for their students).	Yes, very interested.	Not very interested.	No information available.	PE assistant very involved PE teacher somewhat involved.	Teachers must themselves complete 1/4 mile laps periodically; very interested.	Many exercise with children.
Constraints for integrating additional physical activity time into the school day	None voiced directly. Have PE teacher, two assistants.	No PE teacher. Older kids may not want to exercise. Concern that kids need to spend more time on academics.	Must follow state guidelines. Depends on weather, day of week, what’s scheduled at the gym.	Must follow state guidelines.	No structured class. Responsibility lies on individual teacher.	School pilot-testing walking program. Plan to expand it to other grades this fall.	Students need to first complete assignments in other areas in a timely manner; teacher availability; adverse weather; limited space	No PE teacher currently

Table IV presents a sample of the kinds of information being used to develop a school curriculum to foster healthful eating choices and encourage increased physical activity. By comparing information across sites and across schools within sites, we are able to develop interventions that work within the resources available and address factors common to all sites. For instance, teachers across all sites emphasized that students like hands-on type activities and/or activities where children worked together in groups.

Table V describes some key aspects of school food service across sites. Food service staff were able to describe, usually from decades of experience, the foods that children prefer and dislike. This information is useful in developing the food service protocol for reducing fat in school meals and training materials for food service staff.

Table V.

Selected information from formative assessment concerning school food service

Attribute	Site 1		Site 2		Site 3		Site 4
	School A	School B	School C	School D	School E	School F	School G	School H
Foods students prefer to eat	Apples, oranges, bananas, fruit bars, carrots, granola bars, green beans, corn, white hominy Fish: no longer available Preprepared: pizza, cheeseburgers and tocos. Prepared. burritos. Milk: 2%.	Pizza, corn, peaches, white bread, whole milk.	Hamburger and fries, homemade bread and buns.	Rice, hamburger gravy.	Hot dogs, hamburgers and fries.	Hamburgers and fries.	Pizza, watermelon.	Meats, tacos.
Foods students dislike to cat	Spinach, wheat bread, vegetables (especially canned).	Wheat bread, 2% milk, chilli powder, onions.	Tuna casserole, tuna.	Roast beef, steak.	Unfamiliar foods: enchiladas, meatloaf, beef stew, vegetables, salad.	Unfamiliar foods: enchiladas and meatloaf	Spinach, celery, cabbage.	Vegetables.
Methods of preparation of key foods (especially to reduce the amount of fat)	Ground beef: rinsed twice in hot water; spiced with garlic powder; chilli beans, spaghetti, lasagne and chilli macaroni. Chicken: washed in cold water, fat cut out, sometimes seasoned with butter (margarine) and baked. Gravy: water, beef base, flour.	Ground beef: rinsed in warm water. Chicken. hen served sometimes; head cook prefers pork chops. Oils and lard are used in accordance to Novell-Sysco recipes	Ground beef: brown and drain. Chicken: bake Gravy: with drippings.	Ground beef: brown and drain. Chicken. bake. Gravy: without drippings.	Ground beef: rinse in hot water. Most food precooked, no preparation is needed, just heat and serve	Most foods are pre-packaged, just put in oven. Temporary cook does drain the commodity beef. Regular cook does not drain off the fat.	Ground beef: drain or spoon off grease. Chicken: cut off fat. Use oil instead of lard to fry foods.	No information provided

Amounts of physical activity varied greatly between study schools (Table VI), from no physical education classes to three to four sessions per week. Teacher interest in conducting Physical Education classes for their students ranged from little to considerable. The Pathways physical activity intervention strategy is working systematically within these kinds of constraints.

Several papers currently in preparation will address more specifically findings from the formative assessment relating to nutrition and physical activity, and how they have been applied to the Pathways intervention development. However, it is important to mention that the prioritization of risk behaviors is a central guiding structure for the intervention process. Every component of the intervention is considered in terms of how well it addresses this list of behaviors. High priority behaviors are the focus of multiple intervention components. Low priority behaviors (meaning that these behaviors are not commonly found at a majority of sites) represent positive behaviors, e.g. consumption of sugar drinks at school is not common. These positive behaviors are encouraged and reinforced by the Pathways intervention.

Lessons learned

The active involvement of Native Americans in all phases of the formative assessment, particularly in the selection of appropriate methods and development of questions (e.g. deciding on wording, what is appropriate to ask, etc.), proved crucial to the success of this component of the Pathways study. In the process of conducting the formative assessment, we learned a good deal about what worked well and what did not work as well. In general, most components of the formative assessment were successfully carried out We found that parents, teachers and school administrators appreciated our seeking their opinions on what kinds of interventions might be most appropriate for and acceptable to the school, students and families. Having Native Americans conduct the interviews and focus groups was very important, and helped to increase participation rates and improve rapport, particularly among children, parents and other caregivers.

Training

Despite three separate trainings, each aimed at a different phase of the data gathering, several issues arose. One of the largest issues faced was the diversity of backgrounds of persons who were trained to conduct the formative assessment. Data collectors varied from little or no experience in qualitative and quantitative methods to fairly extensive experience in many of the methods. During the trainings, individuals experienced in using specific methods varied considerably in how they felt certain units of the protocol should be conducted. For instance, one data collector’s previous use of the focus group technique involved the use of flip-charts prepared by participants as the sole recording technique, while the protocol called for the use of tape recorders. Ideally, most of these issues would have been settled in advance of the training so that the training could proceed smoothly.

Perhaps the greatest problem that occurred when data collection began was a lack of good open-ended probing for the in-depth interviews. There was a tendency to use the interview guides as a strictly followed, structured survey instrument. This was true for all but the most experienced interviewers. We feel that most of these problems could have been alleviated through more extensive training. In particular, the first training should have been longer as this is when open-ended interviewing skills were introduced. It should have involved more frequent practice interviewing, writing up of interviews and feedback on the quality of interviews from the trainers.

Child interviews

Paired child interviews (Unit 4) involve in-depth interviews with two children simultaneously. These proved very effective with grades 3–5 children. Children appeared to feel comfortable when interviewed with a friend. We found that only rarely would one child dominate the other. While one child was responding to a question, the other would commonly be thinking of additional responses. One child’s responses served as a probe for the other child. Some students did begin to become bored after about 30 min of interviewing. On the other hand, the free listing (Unit 4) and hands-on activities, like the pile sorting exercises (Unit 5) were perceived as playing a game which children seemed to enjoy. Several of the interviewers remarked that the pile sorting, in particular, was valuable in determining what the child was actually eating versus what he/she may have thought the interviewer wanted to hear. Taking children out of the classroom for interviews proved somewhat disruptive of the class, although efforts were made to minimize this effect.

Individual interviews versus focus groups with parents/caregivers

With child caregivers, one-on-one interviews were easier to arrange and more likely to provide adequate participation than group interviews. Individual interviews could be held conveniently in the participant’s home or place of work, whereas focus groups required participants to assemble at a central location which, in many cases, was a long drive from home. Despite several contacts and reminders, focus group attendance remained low. In particular, it was difficult to recruit men for focus groups. Offering incentives and (lower fat) refreshments helped to bolster attendance, as did providing transportation.

Focus groups with teachers and teacher’s aides

Although relatively easier than child caregivers, it still proved difficult to arrange focus groups with teachers and teacher’s aides (Unit 7a), due to scheduling limitations. The only feasible time for scheduling focus groups in most schools was immediately at the end of the school day, after the students had left. This is the teacher’s main daily meeting/work time and the available time never exceeded 45 min. We found that teachers required at least a week of advance notice and focus groups could only generally run 30–45 min. This time limitation meant that we frequently had to run two separate shorter focus group sessions with teachers.

We also found that it was difficult to hold focus groups with only grades 3–5 teachers and teacher’s aides because other teachers and school staff were interested in the Pathways study, and often participated in the focus groups. Others were allowed to participate in order to build rapport and communication between Pathways study staff and teachers. As a result, these discussions tended to be more general than originally intended—and less specific to grades 3–5 children. Therefore, we elected to conduct a series of additional in-depth interviews with individual grades 3–5 teachers (Unit 7b) partly to obtain more focused information.

Direct observations

Direct observation proved invaluable as a means of assessing the environment and behavior (Units 3 and 9). Direct observations in the school environment proved challenging for observers, mostly due to the great amount of activity occurring at any one time. To reduce this problem, observers were instructed to focus their observations on either descriptions of locations (e.g. the classroom), activities of a particular actor or set of actors (e.g. children sitting at one end of the table in the cafeteria only), or event (e.g. foods being sold at a school game).

Many insights were derived from the observations that probably would not have been obtained through interviews alone. For instance, in several school cafeterias, whole milk was put in a more accessible location than low fat milk. It was observed that many students, in particular young students, take the most accessible container of milk, without checking the label to determine fat content. It has been suggested to the food service working group that simply switching places between the two kinds of milk might lead to an increase in the quantity of low fat milk being consumed.

When directly observing behavior, reactivity is always a concern. Observations conducted of classroom activities appeared to have little effect on children’s behaviors, but seemed to make some teachers slightly uncomfortable (although each was asked for their permission for the observation to take place). No reactivity was observed by children engaged in Physical Education or recess activities. During school meals, children would frequently interact with the observer (usually by asking questions); however, this did not appear to affect food consumption behavior. Observations of food service staff during food preparation and serving did, in some cases, appear to make them uncomfortable initially, but this appeared to go away after 10–15 min of observation.

Analysis issues

One of the largest problems in the formative assessment was the large amount of unstructured, textual data that was produced. Interviews with children, parents, teachers and school officials produced nearly 100 h of tapes at each of the four sites. The task of transcription of these tapes was burdensome. One solution used at some of the sites was to hire a dedicated transcriptionist which freed up time for the data collectors.

Besides the sheer bulk of formative data, coding proved to be a substantial undertaking. All the transcribed focus groups, interviews and observations were sent to Johns Hopkins uncoded. An improved system would be to have coding conducted at the field sites by the data collectors themselves. This would encourage them to reread their transcripts and involve them more fully in the data analysis process.

In a large multi-site trial data analysis is usually centralized. This can be disadvantageous for formative assessment research where at least part of the resulting analyses constitute information about the cultural setting and human behaviors which may vary significantly from site to site. Quality analysis of formative data requires input from field workers familiar with each project site. In the Pathways study, this issue was addressed in two ways. First, field sites were asked to prepare summary tables of their main findings (selected results of which have been incorporated into Tables IV–VI). Second, all analyses were reviewed at several different stages by Native American staff members from each site. Differences of opinion about different components of the analysis, such as identified themes, prioritization of various risk behaviors and conclusions, were negotiated and settled to all parties mutual satisfaction through electronic mail and conference calls.

Conclusions

Formative information gathering is essential whenever developing health interventions in new cultural settings where information is limited. People’s understanding of their health and what makes their bodies healthy is embedded in personal experiences, and culture-based meanings and belief systems. Formative research methods, combining qualitative and quantitative approaches, help unlock these meanings and gain insight into developing relevant intervention strategies.

This paper has described the development and use of formative assessment methods in an intervention study aimed at reducing obesity in Native American school children. From a methodological standpoint, certain methods for eliciting information were more successful than others. Individual interviewing (either in-depth or semi-structured) proved more successful and easier to arrange than focus groups in this setting. Paired child interviews were a highly successful means of communicating with young children, eliminating problems of shyness and reluctance to speak associated with one-to-one interviews and disciplinary problems associated with focus groups. Direct observation of behavior was an important means of confirming obesity-related behaviors which became the focus of Pathways interventions.

In October 1996, Pathways moved into its second phase, a full-scale randomized trial involving 40 schools. Lessons learned and experiences gained during the formative assessment of the feasibility phase are being applied. While resources and time do not permit an extensive period of formative research at all schools, effective planning and implementation of the intervention is being preceded by a much reduced structured data collection instrument, the school resource/environmental checklist. Based on our experiences during the first phase of Pathways, this instrument will provide critical, focused information that will enable us to more successfully implement the Pathways interventions.