Integrating formative assessment and participatory research: Building healthier communities in the CHILE Project

Abstract

Background

The need to conduct formative assessment to inform the development of interventional studies has been increasingly recognized in community-based health research. While this purpose alone may provide sufficient justification to conduct formative assessment, researchers are also recognizing the importance of such efforts with regard to partnership building.

Purpose

This article reports a formative assessment process in a large scale randomized controlled trial in New Mexico aimed at preventing obesity in rural American Indian and Hispanic children in Head Start programs.

Methods

We interviewed Head Start staff and conducted observations to understand the context of food service and physical activity in these sites. We also collected data from other community partners, including grocery store managers and primary care providers, to assess appropriate strategies regarding their engagement in the study.

Results

Formative assessment findings helped modify the planned intervention while allowing for variation relevant to cultural and Head Start organizational conditions in each community. Rather than view formative assessment only as a planning phase of the research, our experience illustrates the need to conceptualize these activities more broadly.

Discussion

Integrating formative assessment and participatory research raises the need to address the challenge of ensuring standardization and consistency across varied community settings, the evolving nature of initial formative relationships and the need to build trust in academic/community partnerships.

Translation to Health Education Practice

In our work with American Indian and Hispanic communities in New Mexico, formative assessment represents a partnership building opportunity.

BACKGROUND

The need to conduct formative assessment to inform the development of interventional studies has been increasingly recognized in community-based health research.^1–3 Formative assessment is typically employed to collect relevant data pertaining to cultural attributes and contextual factors believed to affect the implementation of intervention activities.^2,4 Such attention to local context is seen as useful for researchers conducting smaller scale pilot studies as well as multi-center randomized controlled trials. In each case, researchers may uncover implicit institutional norms and/or behavioral attitudes that, left unidentified, could potentially delay or derail a research protocol created outside the study setting.⁵ Whereas these reasons alone may provide sufficient justification to conduct formative assessment research, there is another important aspect of this process that may have as much, if not more, impact on the outcome of the intervention: partnership building.

Consistent with the broader call from the National Institutes of Health and other funding agencies for researchers to engage communities—especially indigenous and historically marginalized groups—as partners rather than subjects,^6,7 formative assessment may hold the most important value in forging a partnership between researchers and communities.⁸ Whereas much of the formative assessment literature has focused on data collection activities to facilitate intervention goals, there has been less attention on the broader contextual factors that influence the process by which researchers and communities form such collaborations.⁹ In minority communities where the collective memory is often keenly aware of the exploitive impacts of research,¹⁰ formative assessment activities represent the first point of contact in introducing and negotiating boundaries for a proposed study.

PURPOSE

We conducted formative assessment research in American Indian (AI) and Hispanic communities throughout New Mexico for the CHILE (Child Health Initiative for Lifelong Eating and Exercise) study. CHILE is a randomized controlled trial aimed at reducing obesity and risk for diabetes among American Indian and Hispanic children ages 3 to 5 enrolled in Head Start programs in rural New Mexico. The CHILE study uses a socio-ecological framework and is designed to prevent obesity by intervening on several levels including the child, the family, the Head Start Center staff and environment, the community grocery store, and local health care providers. In this paper, we present formative assessment findings in the context of the prior relationships and research history that influenced various aspects of the planned protocol for the CHILE study including site selection, recruitment, and study implementation.

METHODS

Design and Sampling

In this study, formative assessment activities were conducted over a 12-month period and were designed to gather information from the full range of community and cultural contexts that we believed would impact the acceptability and implementation of the intervention. To enhance these goals, our formative assessment was aimed at: 1) providing a full description of the Head Start facilities, including scheduling and the overall curriculum, 2) gathering information about dietary and physical activity patterns to inform interventional components, and 3) engaging with the full range of interventional stakeholders—grocery stores, primary care providers, and families—to review recruitment processes and strategies for disseminating study materials.

We used a mixed method design consisting of semi-structured interviews and observations, as well as structured surveys to gather formative data. Given that the focus of this intervention is the Head Start Program, we concentrated data collection efforts in these Centers to collect information from differently positioned staff including administrators, teachers and food service employees.

The sampling frame for the CHILE project consists of a total of 16 rural American Indian and Hispanic communities. Reflecting the smaller number of American Indian tribal communities in the State compared to the Hispanic population, we proposed that 10 of these sites would be in Hispanic communities. Apart from obvious differences in cultural and ethnic composition, these sites experience high rates of poverty and health disparities that are common in New Mexico, especially in minority communities.¹¹ A total of four Head Start venues, divided between American Indian and Hispanic communities, were purposefully selected for formative assessment data collection. Criteria used to choose these sites were based on factors pertaining to organizational complexity to ensure exposure to the full range of issues believed to influence implementation of the CHILE study.

Data Collection

Head Start Staff Interviews

We conducted in-depth interviews with Directors of the four Head Start Centers as well as with the teachers and food service staff. These interviews were conducted using a semi-structured interview guide developed to triangulate perspectives among Head Start staff. Each interview took place in person at the Head Start site. The interview guides focused on the following areas: 1) food service—including food procurement, preparation and meal service; 2) physical activity—related to activities, available resources, and the daily schedule; 3) family involvement and perceived concerns; and 4) overall curriculum and use of media. The interviews lasted between 30 to 45 minutes and were transcribed for subsequent review and analysis.

Grocery Store Interviews and Food Source Checklist

The socio-ecological approach used in CHILE called for linking intervention themes in the Head Start to a local grocery store. Based upon feedback from Head Start staff, we identified a local grocery store used by families. Given our interest in modifying display and purchasing patterns in the stores, we purposefully selected smaller stores, in contrast to larger retail chains, based on our belief that such sites would be more receptive to making changes. We conducted interviews with grocery store managers focused on storage and stocking procedures at the store, the feasibility of ordering new items and feedback pertaining to the process of featuring targeted foods throughout the store. We also used the formative assessment period to implement a food source checklist developed by the CHILE team in the grocery stores. This instrument provided a baseline overview of foods anticipated to be featured during the CHILE intervention as well as pricing information.

Primary Care Providers

Finally, recruitment and formative assessment efforts were focused on identifying an interested primary care provider in each of the eight intervention communities. The CHILE research team partnered with RIOS Net (Research Involving Outpatient Settings Network), a statewide group of primary care providers.¹² The goal was to identify at least one RIOS Net clinician in intervention communities to attend a “parent night” at the Head Start. The intent of these sessions was for the clinician to engage parents and Head Start as a health role model and review CHILE themes relating to healthy eating and physical activity. The CHILE team worked with RIOS Net staff to recruit at least one clinician from local primary care settings and met with them to review the study, discuss scheduling conflicts, and establish a process to disseminate regular updates to the clinician.

Data Analysis

All interviews were transcribed and imported into NVivo (version 8), a qualitative data analysis software program used to facilitate coding and text retrieval. The analyst grouped data into thematic categories (physical activity, food service process, parental concerns, etc.) and created summary reports for the research team to review. These reports included the full range of responses and were used by the group to both inform subsequent intervention planning as well as to identify potential implementation barriers. The analyst also reviewed the site observations and integrated summary findings into the analytic reports.

RESULTS

Demographics

A total of 11 interviews with Head Start administrators, teachers and food service staff were conducted in the two American Indian communities. In the two Hispanic sites, a total of 12 interviews were conducted. Additionally, interviews with grocery store managers were conducted in all four communities along with recruitment meetings with four primary care providers.

Head Start Facilities

Interviews with Head Start teachers, food service staff and Directors provided insight into the process by which each Head Start organizes meals, incorporates physical activity and implements its educational curriculum. These discussions also focused on current levels of parental involvement at the Head Start as well as views about cultural acceptability of the proposed intervention. We have grouped key findings from these interviews used to inform the final intervention design into the following categories: 1) food service; 2) parental involvement; and 3) Head Start organizational issues. We have also included a sub-section entitled “CHILE Challenges” to further examine the implications of these findings in our intervention planning.

Food Service

A key component of the CHILE intervention involves modifying current Head Start food planning efforts by incorporating strategies to enhance the nutritional value of meals and snacks. Most of the Head Start facilities offered two full meals (breakfast and lunch) during full- and half-day sessions. There were differences across Head Start venues with regard to the development of menus and food purchasing arrangements. In some cases, Head Start facilities contracted with an outside food vendor through which yearly menus were created in advance. In these situations, Head Start food service staff indicated greater difficulty in modifying menus and purchasing alternative food items.

Meals were either served in a common space such as a cafeteria or multipurpose gym or delivered to the individual classrooms. In the latter cases, meals were typically served “family style” with teachers overseeing food distribution. Teachers noted the importance of encouraging children to try new foods and using meal times to foster independence and communication. One theme that consistently emerged in these discussions was the general concern about food insecurity and its impact on teachers’ approach to regulating portion sizes. As one teacher noted:

“We don’t want to push away our children because we don’t know how they’re served at home, if they don’t get enough at home, or they have too many brothers or sisters. But I wouldn’t say I try to limit them, but sometimes I will tell them to save enough for the others.”

CHILE Challenges—Food Service

In reviewing these findings with regard to intervention planning, there were two areas that presented challenges. In Head Start facilities that had contracts with outside food vendors, we were concerned about the limited flexibility in purchasing new or different foods as well as the ability to modify menu plans. Therefore, the CHILE intervention would need to work with Head Start programs to devise nutritional enhancement strategies that fit within ordering and menu planning constraints. The other issue that emerged involved the need to recognize that some families may face food insecurity and intervention messages pertaining to portion size need to be tempered with this understanding.

Parental Involvement

We also engaged Head Start staff about the type and level of parental involvement. In the formative assessment sites, parental involvement ranged from low to high and also was described as variable across classrooms within a given Head Start. There was consensus across Head Start venues that parents regularly attend special events planned in advance. Additionally, teachers noted that parents were clearly engaged with their own child’s experience but were unable, often due to work schedules, to participate in the daily activities of the Head Start. In some cases, however, parental volunteerism is encouraged as a way to address high rates of staffing turnover.

CHILE Challenges—Parental Involvement

Given the essential role that parents and caregivers play in making decisions about food and physical activity choices that impact children, the CHILE intervention needed to effectively reach families. To accomplish this goal, the CHILE team developed mechanisms to disseminate CHILE intervention materials to each participating family.

Head Start Organizational Issues

Some of the Head Start venues used curriculum packages as a basis for lesson planning and educational themes while others described a more flexible approach based upon parental input and children’s interests. This flexibility appeared to be particularly important to the AI sites given the concern about children learning the native language. In these Head Start programs, teachers described adaptations to translate educational materials and draw upon traditional themes in their daily activities. As one teacher explained at a tribal site:

“I think that the traditional (tribal) language is really, really important here. Each year, we have our own administration. When they come in they also have goals that they have developed for all the programs. And one of the biggest things is to keep the language alive.”

Given their need to meet and further their own educational goals, teachers expressed enthusiasm in attending in-service trainings made available at the Head Start facilities. Many of the teachers were interested in advancing their teaching credentials and often asked members of the CHILE research team whether the project would include such educational opportunities.

We identified two other issues that had direct implications on intervention planning. Throughout New Mexico, pre-K programs were expanding and drawing four-year-olds away from Head Start programs. Therefore, we observed diminished enrollment of these older children in communities with developed pre-K programs. We also found that most of the Head Start facilities had classrooms that integrated children of different ages. Our initial intervention design did not anticipate such integration; instead, we had assumed that entering three year olds continued through the two-year program as a cohort. We had initially planned to implement an age appropriate evolving curriculum for the entering three year olds that would continue during their second year. The mostly mixed classrooms, however, necessitated development of a CHILE curriculum that rotated themes to ensure full exposure for all children.

CHILE Challenges—Head Start Organizational Issues

Several challenges emerged from these findings. The first issue relates to creating intervention materials in different languages that have cultural transferability to a wide range of community settings. Secondly, we identified an important opportunity to assist teachers in using their involvement in the CHILE project as a mechanism to advance their career goals. The third challenge involves the uncertainty of diminishing enrollment during the second Head Start year to pre-K programs. Our goal was to follow the children for the full two years in the Head Start. Losing a substantial portion of the cohort in the last year would reduce our ability to measure primary outcomes. Lastly, the age integrated classrooms led us to rethink the dynamics of incorporating the CHILE curriculum into classrooms with children at varying cognitive levels.

Community Grocery Stores

In the formative assessment communities, we completed a “Food Source Checklist” at each of the proposed grocery stores to make initial observations regarding the availability and pricing of CHILE featured foods. The use of this checklist was intended as a pilot effort to assess the instruments’ feasibility during the intervention as both a process evaluation tool and outcome measure (e.g., inclusion of new foods). Whereas the intent is not to report these findings here, we found that many of the targeted CHILE foods in these stores were offered at higher prices compared to their less nutritional counterparts (e.g., whole wheat bread was more expensive than white bread). We further interviewed grocery store managers in each location to discuss logistics of incorporating CHILE display items in the store as well as barriers to purchasing and stocking different foods. In most cases, the stores were highly cooperative and willing to participate in the CHILE project. Managers indicated that they had limited staff to maintain CHILE displays and that decisions about stocking new foods were often out of their direct control but they were willing to explore this possibility.

CHILE Challenges—Community Grocery Stores

Although these grocery stores were consistently receptive to participation in the CHILE project, the challenge for the CHILE team would be to regularly monitor CHILE displays and restock as necessary. Additionally, we found that it would not always be possible to ensure that a CHILE featured food would be available for purchase at the grocery store.

Primary Care Providers

The last component of the CHILE intervention plan involved the participation of primary care providers from RIOS Net in each community. The RIOS Net group was responsible for recruiting interested member clinicians and working with the CHILE team to coordinate dates/times of parent night presentations as well as the thematic materials to highlight during these sessions. Given the relatively small role of the clinicians, most of the formative assessment work focused on clearly defining the clinicians involvement and logistical planning to accommodate their busy schedules.

CHILE Challenges—Primary Care Providers

Identified challenges mostly related to developing a communication plan with RIOS Net and the advanced planning necessary to involve primary care clinicians in community events. We also recruited additional clinicians in most communities in anticipation of a clinician either being away during a scheduled parent night or on-call and unable to attend.

DISCUSSION

During the formative assessment phase of the CHILE intervention study, important information pertaining to each intervention component—Head Start, family involvement, grocery store, and primary care provider—was obtained. As noted earlier in this paper, we viewed formative assessment data collection as both a way to identify issues to be addressed in the intervention design as well as an opportunity to continually build our partnership with each Head Start and the broader community in which it is embedded. With regard to the former, we focused our efforts on ensuring that the intervention was: 1) feasible for each Head Start given the current configuration of services; 2) appropriate for diverse community and cultural contexts; and 3) able to sustain meaningful linkages to the other intervention collaborators—the community grocery store and primary care providers. In terms of the latter, we actively engaged a range of key stakeholders in each community to demonstrate our commitment to this important health problem and our sensitivity to the exploitive history of research among American Indian and Hispanic communities. Whereas there is general consensus that formative assessment is useful in guiding the development of effective interventions,^2,3 we believe it is also important to link such efforts to the broader goals of participatory research.⁹

Benefits and Tensions in Formative Assessment Research

Although formative assessment research—identifying local contextual factors to inform intervention design—and participatory research—focusing on processes through which communities and academic researchers jointly address problems and create solutions—are interrelated and beneficial, there are some areas of tension for researchers to consider.

The first issue relates to the challenge of ensuring standardization and consistency across varied community settings involved in the intervention. Researchers struggle to balance the often conflicting pressures of internal versus external validity; that is, to create an intervention that is contextually sensitive (internal validity) but not to a degree that compromises its potential application in other settings (external validity).^13,14 This tension emerged in the CHILE study with regard to family participation. Parental involvement is central to Head Start and national performance standards set requirements for the amount and type of these activities.¹⁵ However, barriers to achieving consistent engagement may vary depending on a range of local factors including socio- economic conditions, language, and perceived receptivity to such involvement. Our formative assessment findings helped us to create parental involvement strategies that shared similar elements across the sites (e.g., number of contacts per year) but were tailored to address varied community and cultural conditions.

Another area of tension involves maintaining an actively engaged partnership with study sites throughout the project. In CHILE, we challenged the perception that formative assessment activities are completed once the intervention has been developed and initiated. Our initial discussions with Head Start programs led us to anticipate a high degree of staff turnover during our five year study. Changes in organizational leadership may pose challenges to previously established partnership agreements. We designed the formative assessment to develop a broad coalition of support at each Head Start (among administrators, teachers, parent groups, etc…) to ease the impact of expected staffing transitions on the intervention activities. Further, as we observed throughout the course of our formative assessment work, the interest at the State level to increase funding for pre-K programs could potentially affect enrollment at several of the Head Start facilities.

By building on established relationships and cultivating new ones, we sought to create a deeper connection with each community Head Start to both enhance communication throughout the intervention and overcome the inevitable changes that occur during the course of a long-term study. Therefore, it is important to view the study partnership as an evolving process and the initial formative assessment stage as inextricably connected to implementation activities.

Finally, it is necessary to acknowledge the importance of building and maintaining trust in academic-community partnerships.^16,17 Consistent with the participatory nature of our formative assessment process, we approached each community at the outset to gain their approval and investment in the study. During these initial presentations, most Head Start staff expressed an interest in “getting the project” (i.e., being part of the intervention). Anticipating this concern, CHILE was designed to have a one-year delayed intervention during which all control sites would receive the same intervention training and materials. As described above, we solicited input on ways the CHILE study could provide either material or training resources to benefit the Head Start facilities. In addition to providing all control sites with a monetary incentive, we also worked with each site to provide educational training to Head Start teachers to further their career goals. Collectively, these perceived benefits of participation, along with a strong investment in addressing childhood obesity, were compelling and virtually all sites approached agreed to be included in CHILE.

It is important to identify a limitation of this study. This formative research was conducted in Southwest American Indian and Hispanic communities that have unique cultural, historical and socio-economic features. However, the purpose of this article is not to suggest that the tailored intervention strategies used in the CHILE study are generalizable to other settings. Rigorous process evaluation findings, along with primary study outcomes, will be used to assess the appropriateness of these strategies. Rather, we believe that the broader conceptual approach—viewing formative assessment as part of a partnership that evolves throughout the course of the study—is an important component in interventional research.

TRANSLATION TO HEALTH EDUCATION PRACTICE

Based on these formative assessment process findings from the CHILE study, we offer the following observations to researchers engaging in similar efforts to build a partnership with minority and/or underrepresented communities:

• Involve potential community partners early in the process

• Identify key elements in recruitment/retention from prior experiences and/or other researchers familiar with the target communities

• Envision formative assessment as an iterative process, allowing communities time to reflect, provide feedback and negotiate conditions of participation

• Anticipate the need for flexibility in the intervention design and prepare alternatives that reflect “real world” conditions

• Identify opportunities to “add value” to partnership participation in the form of resources or training to enhance involvement

• Develop and maintain relationships with several community key stakeholders to ensure smooth transitions and open communication

The formative research approach utilized in the CHILE study followed participatory principles and demonstrates the benefits and challenges of this integration. This process of engagement helped reveal important information that influenced the design of the study and built a solid partnership foundation for the early stages of the intervention. Rather than view formative assessment as a phase of research that has fixed boundaries (prior to the intervention), our experience illustrates the need to conceptualize these activities more broadly. In our work with American Indian and Hispanic communities in New Mexico, formative assessment represents a continual cycle, building on previous relationships and evolving throughout the course of each study. We believe that this perspective is necessary as a way to conduct both culturally appropriate and contextually sensitive research in historically marginalized communities.