Process evaluation of Healthy Bodies, Healthy Souls: a church-based health intervention program in Baltimore City

Abstract

Soaring obesity rates in the United States demand comprehensive health intervention strategies that simultaneously address dietary patterns, physical activity, psychosocial factors and the food environment. Healthy Bodies, Healthy Souls (HBHS) is a church-based, community-participatory, cluster-randomized health intervention trial conducted in Baltimore City to reduce diabetes risk among urban African Americans by promoting healthy dietary intake, increased physical activity and improvement to the church food environment. HBHS was organized into five 3–8-week phases: Healthy Beverages, Healthy Desserts, Healthy Cooking, Healthy Snacking and Eating Out and Physical Activity. A three-part process evaluation was adopted to evaluate implementation success: an in-church instrument to assess the reach, dose delivered and fidelity of interactive sessions; a post-intervention exposure survey to assess individual-level dose received in a sample of congregants and an evaluation form to assess the church food environment. Print materials were implemented with moderate to high fidelity and high dose. Program reach was low, which may reflect inaccuracies in church attendance rather than study implementation issues. Intervention components with the greatest dose received were giveaways (42.0–61.7%), followed by taste tests (48.7–53.7%) and posters (34.3–65.0%). The dose received of general program information was moderate to high. The results indicate successful implementation of the HBHS program.

Introduction

Obesity in the United States has reached alarmingly high rates [1]. Data from the 2009–10 National Health and Nutrition Examination Survey showed that some of the highest rates of obesity (49.5%) are among African American (AA) adults [1]. AAs have disproportionately higher levels of obesity compared with their Caucasian counterparts, as well as substantially higher rates of diabetes [1–4]. Additionally, from 1980 through 2010, the age-adjusted percentage of persons with diagnosed diabetes more than doubled (increasing from 4.5 to 9.5%) for AAs, which was consistently higher than the rates for Caucasians [5].

Dietary intake, physical activity level and food environment are risk factors that play major roles in the high prevalence and incidence of obesity and diet-related chronic diseases among AA populations. Low-income AAs have been found to have high consumption of sugary drinks and high-fat food and extremely low intake of fruits and vegetables, with two-thirds reporting no fruit consumed in the previous day and three-quarters reporting no vegetables consumed at all [6, 7]. Physical activity rates among AA adults are lower than those among Caucasian adults, particularly with regard to AA women [8–10]. Environmental factors have also been shown to play a major role in the development of chronic diseases among inner-city AA communities [11, 12]. Individual eating habits are directly influenced by access to healthier food options; however, the higher cost of healthier foods can also serve as a barrier to access, particularly for low-income households [13–15].

Churches are potential settings for delivering health-behavior interventions owing to the central role they play as spiritual, cultural and social support networks in AA communities [16–18]. Churches have many advantages as intervention venues, including designated spaces for gatherings/meetings, regularity in scheduling owing to weekly worship services and other church activities, regular contact with church members and strong social influence on community members with regard to shared core values and behaviors [16, 18, 19].

Despite these strengths, faith-based intervention programs that aimed at the prevention of chronic diseases among AA communities have had mixed success on increasing physical activity and limited success in changing dietary behaviors [10, 20–22]. Lack of a substantial process evaluation, and therefore decreased ability to interpret impact of these programs in the context of the program delivery and receipt, may have contributed to these mixed results [23, 24]. Faith-based interventions, like most community-based interventions, are often complex, multifaceted and often require adaptations to the standard design. Process evaluation provides vital information needed to fully explain why certain results were achieved [24, 25]. Of 67 church-based intervention trials published between 1990 and 2008, approximately 80.4% measured reach, 28.4% assessed dose delivered, 27.3% measured dose received and only 9% assessed fidelity [26]. Only 21% of them included a combination of any of the measures: reach, dose delivered, dose received or fidelity [26].

One exception is the ‘Partnership to Reach African Americans to Increase Smart Eating (PRAISE!)’ program that demonstrated an extensive process evaluation. Project PRAISE was a faith-based, multisite randomized controlled trial to assess the effectiveness of a year-long intervention to increase fruit, vegetable and fiber intake and to decrease fat consumption among AAs using culturally relevant approaches [27]. PRAISE! took novel approaches to process evaluation, such as determining the success of interventions at specific study locations, thoroughly documenting the program implementation, assessing initial and potential sustainability and dissemination of intervention components and evaluating the research partnership [27]. Building on the lessons and experiences learned from PRAISE!, Healthy Bodies, Healthy Souls (HBHS) was designed with a systematic process evaluation model for a church-based intervention program.

Few church-based health intervention trials have attempted to simultaneously influence diet, physical activity and the food environment [11]. HBHS was a multisite, multiphase randomized controlled trial that investigated the efficacy of faith-based behavioral and environmental interventions in preventing diabetes risk factors within low-income communities in Baltimore City. The program was conducted in nine predominately AA Baltimore churches from August 2009 to December 2011. The primary aim of HBHS was to reduce diabetes risk among AAs through an integrated nutrition and physical activity program using a comprehensive approach of extensively modifying the food environment and positively influencing individual health behaviors.

The HBHS program focused on improving eating behaviors, food preparation methods, psychosocial determinants (self-efficacy, knowledge and behavioral intentions), physical activity and the food environment through an integrated and participatory approach among AA churchgoers. Program strategy development was done in a participatory manner with regular feedback and input from church members to ensure cultural relevance and appropriateness and to strengthen the relationship between the research team and the community [28]. Funded by the American Diabetes Association (ADA), HBHS was developed to complement Project POWER (PP), an AA church-based diabetes education program currently being implemented by the ADA, which uses workshops to train congregants on a healthy diet and lifestyle.

Two levels and three main instruments of process evaluation were used for the HBHS intervention: an in-church interactive session form to assess church-level delivery of the intervention components, a post-intervention exposure survey to assess individual-level dose received and an in-church environmental evaluation form to assess the church food environment. This comprehensive process evaluation approach sought to identify factors that would account for successes and failures of the program, and potential for sustainability at the church level. This article addresses the following questions: (i) how well was HBHS implemented in terms of reach, dose delivered and fidelity, and (ii) what was the level of dose received by church members?

Methods

Evaluation design

HBHS was conducted in nine predominately AA churches. Churches were selected based on the following criteria: (i) not having previously participated in the PP program, (ii) having self-reported regular attending congregations of at least 100 adults, (iii) having a mostly AA congregation, (iv) having more than 2 miles distance from other trial churches, (v) were not currently participating in any health program sponsored by outside organizations and (vi) pastor was willing to provide a letter of collaboration. The nine churches were stratified into three cohorts based on size. Each church in one cohort was randomly assigned to one of three groups. Group 1 churches received the HBHS intervention and PP interventions simultaneously (treatment), Group 2 churches received PP only and Group 3 churches received a delayed PP intervention (comparison). In Group 1 churches, PP was implemented during the HBHS intervention period but occurred in different phases of the intervention according to the preferences of each church. The three Group 1 churches (church A, B and C) had weekly worship service attendance of approximately 125, 250 and 500, respectively, on average, as reported by church leaders.

Before the intervention, there was a 2–3-month phase during which HBHS program staff conducted a series of planning workshops with each of the three HBHS intervention churches to determine the main foods and health behaviors for promotion, develop appropriate intervention strategies and identify the most appropriate nutrition and physical activity information to be disseminated. A series of 4–5 planning workshops were completed at each intervention church, resulting in a total of 13 workshops. On average, 12.3 (range: 6–20) adults attended the 13 workshops, and included various church stakeholders such as pastors, health ministry leaders and church congregants. Common findings across churches helped form the basis of the HBHS intervention [28].

Each HBHS intervention church received five phases of the HBHS intervention, and each phase was composed of 3–8 weeks of intervention activities focusing on a single theme (Table I). An intervention team of trained staff visited the intervention churches every week to deliver intervention activities and conduct process evaluation. Selection of intervention activities such as taste tests, interactive sessions and distribution of educational materials to address specific nutrition/lifestyle barriers during each intervention phase differed somewhat across intervention churches based on the needs and strategies identified in each church’s planning workshops.

Table I.

Phases of the Healthy Bodies, Healthy Souls program

Phase	Weeks	Topics	Target behavior	Behavioral objective
1: Healthy beverages/introduction	5	Introduction Pedometer Flavored water Sugary beverages Cost-effectiveness	Diet, physical activity	Increase physical activity by increasing walking.
				Consume low-sugar beverages and 100% fruit juices over sugary beverages.
2: Healthy desserts	4	Dessert control Alternative sweeteners Nutrition label Discretionary calories	Diet	Consume low-fat, low-sugar dessert alternatives; understand and use labels to choose healthier dessert alternatives.
3: Healthy cooking	5	Vegetables Whole grains Fats Herbs and spices Cooking methods	Diet	Choose low-fat, high-fiber, low-sodium ingredients when cooking; use healthier cooking methods to reduce consumption of fat, sodium, and calories.
4: Healthy snacks/eating out	5	Snack alternatives Snack habits Healthy restaurant/carryout Healthy fast food Pack your own lunch	Diet	Choose low-fat, low-sugar, low-calorie snacks; understand and use portion control.
				Choose healthier menu alternatives when eating out; increase packing of low-sugar, low-fat, low-calorie lunches.
5: Physical activity/general	3	Warm-up Aerobics Strength training	Physical activity	Increase physical activity through cardiovascular and strength training exercises.

Intervention components were delivered in multiple forms (Table II) during the weekly intervention church visits and included: (i) weekly church bulletin inserts; (ii) educational handouts distributed to the church congregation by interventionists; (iii) posters and educational displays corresponding to each HBHS phase displayed in the church throughout the intervention; (iv) taste tests with accompanying recipe cards of modified popular dishes, such as a roasted vegetable casserole as an alternative dish for pan-fried vegetables with fatback to promote the consumption of vegetables; (v) promotional giveaways given out intermittently depending on the intervention topic, such as shopping bags, specialized cookbooks for people with diabetes, water bottles and measuring spoons and (vi) a pedometer challenge competition. The pedometer challenge was a walking promotion program developed to encourage daily physical activity. The rules of the competition differed by each church’s preferences, and adult church members were all eligible to participate at any stage of the intervention period. The interventionists tracked the daily and weekly steps of participating individuals at each HBHS interactive session and recognized participants who stepped the most or demonstrated the greatest consistency during a celebration event at the end of the program. Educational presentations, although not a part of the official intervention protocol, were given on request of the church.

Table II.

Intervention components and standards for delivery

Component	Description	Objective	Frequency conducted
Bulletins	Pamphlet or sheet covering various nutrition and physical activity topics according to the phase; incorporates church message and testimonials as well	Provide health information	Weekly
Handouts and fliers	Sheet that accompanies the weekly bulletin; expands on the topics covered in the bulletin including practical application of nutrition/physical activity information presented	Provide health information	Weekly
Posters and educational displays	Summarizes phase information and display health catchphrases; serve as reminders for participants to practice the information presented throughout the program	Increase program exposure, provide health-promotion information	Weekly
Taste tests	Healthier food dishes based on program recipes; provides participants the opportunity to sample recipes	Promote healthier recipes/cooking methods of existing popular dishes	Only when recipe cards were present as part of the weekly topic
Giveaways	Free materials handed out to draw in additional participants or reinforce continuing interest in the program	Promote program, increase program exposure	Random
Pedometer challenge	Churchgoers received a pedometer and were asked to report their steps every week. Prizes were awarded to individuals who demonstrated the best performance or consistency of walking steps at the end	Promote physical activity	Weekly

The environmental component of HBHS was tailored to the specific needs of each intervention church. Universal food environmental interventions across all intervention churches included providing each church with a water pitcher and filter to promote drinking water as a healthy alternative to sugar-sweetened beverages, displaying posters and educational displays in high-traffic areas of the church and introducing Community-Supported Agriculture programs. One or two shares (equal to eight fruit/vegetable items per share) of local organic fresh produce were donated to each intervention church through Community-Supported Agriculture, which provided these shares on a weekly basis for 6 months.

Process evaluation instruments

The three process evaluation instruments were an in-church interactive session form, a post-intervention exposure assessment survey and an in-church food environment evaluation form.

In-church interactive session form

The form was completed during each church intervention visit by a trained process evaluator. Each church visit comprised an interactive session where various interactive intervention components were delivered. The form was used to evaluate reach, dose delivered and fidelity by recording the number of participants, types of activities, giveaways and print materials, visibility of educational displays or posters and feedback from participants. Reach was defined as the percentage of church attendees contacted in the interactive sessions out of the total congregation. Dose delivered was defined as the average number of intervention components distributed to an individual in one visit. Each intervention session had multiple components that a churchgoer could participate in, but churchgoers were not required to participate in all components of the interactive session (i.e. they could get the giveaway item but choose not to participate in the taste test). Therefore, the minimal dose delivered was defined as receiving one item per participant per visit. Fidelity of an intervention component was defined as the percentage of effective visits out of the total visits per program phase. A church visit was considered effective if an intervention component (i.e. taste tests and handouts) was delivered in that visit. Low fidelity was defined post hoc as 0–49% of the standard met, moderate as 50–74% and high as 75% and greater.

Post-intervention exposure assessment survey

The form was used to assess individual exposure (dose received) to certain intervention components and to provide an estimation of reach at the individual level. The form was a questionnaire that was to be completed by trained data collectors as part of the post-intervention interview to track changes of the outcome over time in a subsample of church members. Therefore, only respondents from both intervention and comparison churches that had completed baseline questionnaires were surveyed. Because these measures were taken from a subset of church members, we were unable to conduct analyses of reach at the church-wide level. Dose received was defined as the proportion of interviewed participants who successfully recalled exposure to any intervention components. Dose received was determined by asking the respondents to recall interactive activities, food environment changes and a selection of the most commonly used intervention components. Sample pictures of materials used in the intervention were shown during the interview to facilitate recall. Four red herring questions (questions intentionally asking about components that were not actually used in the program) were included to identify respondents exhibiting reporting bias (none were identified). A problematic respondent was defined as giving affirmative answers to three or more of these questions and would be eliminated from the analysis. Low dose received for each intervention component was defined post hoc as being successfully recalled by 0–49% of all sampled participants, moderate as 50–74% and high as 75% and greater.

In-church food environment evaluation form

The form was used to (i) measure the availability (number of available items out of the total items assessed in the form) of certain healthier food items in meals or snacks served at the church, in-church stores or vending machines; (ii) identify physical activity equipment/programs in the church and (iii) to assess the availability of health-promotion media such as flyer and posters. The evaluation form assessed 27 types of foods that may be served in meals, 18 types of snacks, 10 vending machine items, seven types of physical activity equipment, lessons and group activities and seven types of health-promotion media. Low availability was defined post hoc as having 0–29% of all listed healthy items/components in the church, moderate as 30–59% and high as 60% and greater. The form was completed three to seven times in each church by process evaluators during a randomly selected subsample of church intervention visits.

Data collection

To examine differences among treatment groups, post-intervention exposure survey data were collected from church members in all nine churches, whereas in-church process evaluation data were collected only in the three HBHS intervention churches. Of 75 total visits to the three intervention churches, 64 in-church interactive session forms and 16 food environment evaluation forms were filled out. The exposure data were collected within 1–3 months after the intervention. In total, 196 post-intervention interviews were completed, including 77 from three HBHS intervention churches. The follow-up rate was 68% for all respondents and 73% for HBHS intervention churches. The post-intervention survey non-respondents (n = 92) had comparable demographic characteristics with respondents, except being younger (P = 0.04) (J. Gittelsohn, submitted for publication). Interviews took place in respondents’ homes, churches or other locations convenient for the participant. Data collectors were all trained public health graduate students and project staff members, who were blinded to the red herring questions. The study and all data collection instruments were approved by the Johns Hopkins Bloomberg School of Public Health Institutional Review Board.

Data analysis

Exposure data were analyzed using STATA version 11 [29]. The reach, dose delivered, fidelity, dose received and the availability of healthy items in church environment were calculated as described previously. The pedometer participation indicators and general program reach were tabulated by each intervention church as well as overall. These indicators were tabulated by each intervention church and the differences among churches were compared using Pearson chi-squared test (those presented as percentages) or t-test (those presented as mean values).

Results

Reach, dose delivered and fidelity

Table III presents the reach and dose delivered per visit of all intervention components. Overall, the program had sufficient intervention visits to churches, with the total visits of each phase being 94–157% of the standard number of planned visits. The standard number of visits was the minimal number of visits required to sufficiently cover one full phase (Table I). Although the assumption was that each of the topics in a phase could be covered in one visit, more complex topics were reinforced with one to three extra visits. Observed numbers of visitors at intervention sessions had wide ranges and were 6–63, 10–35 and 19–160 persons for each intervention church, respectively (data not shown). On average, each visit reached approximately 23 (8% of the total congregation) to 31 (11% of the total congregation) church members (Table III). Handouts and flyers were distributed the most frequently (mean = 18.5/visit), whereas taste tests and giveaways were less frequently administered. Each person received 1.0–1.9 intervention components per visit, with an average dose of 1.3 items per person per visit, exceeding the standard dose received of one item per participant (Table III).

Table III.

Reach and dose of intervention activities delivered by HBHS interventionists

		Mean number per visita
Phase	Actual visits/standard visits (%)b	Number of visitors	Reach, % of total congregationc	Taste test samples distributed	Handouts/flyers	Giveaways	Intervention components delivered per visitor (dose)d
1	100	28.9	10%	8.9	14.5	6.6	1.0
2	108	23.2	8%	3.6	16.5	4.8	1.1
3	126	30.8	11%	3.8	13.5	21.2	1.2
4	94	31.0	11%	5.4	24.0	12.9	1.4
5	157	25.6	9%	5.4	26.0	16.8	1.9
Overall	114	28.2	10%	5.4	18.5	13.0	1.3

^aThree intervention churches combined. ^bMinimum standard for intervention visits per church: Phase 1 = 5; Phase 2 = 4; Phase 3 = 5; Phase 4 = 5; Phase 5 = 3. ^cDenominator was the average regular attendance of three intervention churches based on church’s self-reported data: 192 per visit. ^dStandard was the minimum that one component received by each visitor per visit.

The program achieved a moderate to high level of fidelity in implementing bulletins, handout/flyers and poster/displays per intervention visit (Table IV). Educational print materials were administered in 67–100% of all intervention visits, with an overall mean availability of 95, 81 and 76% for bulletins, handouts/flyers and posters/displays, respectively, across all program phases. The program achieved a low to moderate level of fidelity in giveaways, with the availability ranging from 36 to 53%. Giveaways had an overall mean availability of 44% across all intervention visits. The lower fidelity in giveaways is due to the fact that they were intended to serve as a marketing strategy to increase the participation rate and did not have a fixed implementation schedule as other components. The level of fidelity for taste tests varied from 14 to 69%, with an overall mean availability of 45% (data not shown) across all intervention visits. As Phase 5 focused on physical activity and was not planned to have taste tests, when it is excluded from the analysis, the overall mean availability of taste tests in diet-related phases (Phases 1–4) is more accurately calculated as 56% (Table IV).

Table IV.

Fidelity of intervention activities by phase (% times a component delivered of total visits in each phase^a,b)

	Phase 1	Phase 2	Phase 3	Phase 4	Phase 5	Overall
Intervention component	Healthy beverages	Healthy desserts	Healthy cooking	Healthy snacks/ eating out	Physical activity	–
Church bulletins	100	92	100	93	86	95
Handouts/flyers	67	92	79	79	93	81
Posters/educational displays	67	85	79	86	64	76
Giveaways	40	38	53	36	50	44
Taste tests	40	69	58	43	14c	56d

^aTotal number of church visits: Phase 1 = 15; Phase 2 = 13; Phase 3 = 19; Phase 4 = 14; Phase 5 = 14. ^bPercentages based on at least one intervention item administered per visit, i.e. multiple giveaway items at one visit counts as one giveaway intervention. ^cTaste tests were not planned for this phase. ^dOverall average for taste tests not including physical activity phase.

Dose received

Table V shows that 40–60% of the respondents remembered seeing or participating in at least one intervention component in any given phases. Giveaways were the most highly recalled component during three of the four applicable phases. Participants who successfully recalled taste tests were high in diet-related phases, whereas posters and educational displays were best remembered in physical activity/general program information phase. Giveaways and taste tests had moderate dose received in most phases, and visual materials had low to moderate dose received across all phases. The comparison groups (Group 2 and 3 churches) showed a significantly lower percentage of exposure to all intervention components, as expected (P < 0.0001).

Table V.

Description of the intervention components and percentages of church members who received any intervention in each phase (dose received, individual exposure)

			Percentage of participants received any intervention component
Phases	Intervention component	Number of the intervention componenta	Interventionb N = 94 %	Comparisonc N = 84 %
1. Healthy beverages	Taste tests	2	48.7	1.1
	Giveaways	1	54.3	4.3
	Poster/educational displays	3	46.9	15.2
2. Healthy desserts	Taste tests	4	52.5	3.2
	Giveaways	0	N/A	N/A
	Poster/educational displays	2	52.5	4.3
3. Healthy cooking	Taste tests	11	53.7	3.2
	Giveaways	3	59.2	2.1
	Poster/educational displays	1	34.4	10.2
4. Healthy snacking/eating out	Taste tests	3	52.5	2.1
	Giveaways	2	61.7	4.3
	Poster/educational displays	1	41.8	1.1
5. Physical activity and general	Taste tests	0	N/A	N/A
	Giveaways	3	42.0	7.5
	Poster/educational displays	4	65.0	11.8

^aThe intervention components listed here were a reasonable subset of all components, selected based on how common they were used across intervention churches. ^bIncludes a random sample of participants from three Group 1 churches. ^cIncludes a random sample of participants from 6 Group 2 and 3 churches.

The majority of participants (>89.5%) in HBHS intervention churches had seen pedometers from the program (Table VI). Participation rate in the pedometer challenge was moderate to high, ranging from 61.9 to 90%. Churches B and C had lower participation rates due to the limited number of pedometers available to church members despite the overwhelmingly positive responses to the program. Respondents who participated in the pedometer Challenge wore the pedometer for 2.5–3.9 months, approximately half the length of the program implementation timeline (6 months). The average time of wearing a pedometer did not differ significantly by intervention church (Table VI). The majority of the respondents in intervention churches had heard about HBHS (>95%) and many of them had recognized the project logo (71.4–90%), as shown in Table VI. HBHS’ presence was also seen in control churches, but only 25.8% have seen project logo and 36.7% have seen print materials. Church bulletin inserts had a high reach, with an average 75.3% of the respondents in intervention churches remembering seeing them. Most participants reported receiving more than five different types of the church bulletin over the course of the entire program (data not shown).

Table VI.

Receipt of the pedometer challenge and general program information

	Church A	Church B	Church C	Intervention total	Control total
Indicators	N = 20	N = 19	N = 42	N = 81	N = 93	P-valuea
Have seen any pedometer component, %	100.0	89.5	90.5	92.6	–	0.342
Pedometer challenge participation rate, %	90.0	68.4	61.9	70.4	–	0.075
Average length of wearing pedometer, months	3.9	2.7	2.5	2.8	–	0.216
Have heard about the project, %	100.0	100.0	95.2	97.5	82.8	0.386
Have seen the project logo, %	90.0	79.0	71.4	77.8	25.8	0.275
Have seen project-related print materials, %	75.0	80.0	73.8	75.3	36.7	0.922

^aThe P-values indicated the difference among Church A, B and C. The average length was obtained using t-test, others using Pearson χ² test.

The church food environment

Table VII shows the availability of in-church healthy foods, healthy snacks, physical activity components and health-related media for each church. In two intervention churches that served meals and snacks, the average number of different types of healthy food was greater in meals (8.0 and 5.7 for churches A and C, respectively) and lower in snacks (0.3 and 2.3 in churches A and C, respectively). The average types of healthy foods were 1.9 in church B’s store and 1.7 in church C’s vending machine. Church A had moderate availability of healthy foods served in church meals (30% of all assessed items), and churches B and C had low availability of healthy foods served or sold in the church (20 and 17%, respectively). All intervention churches had physical activity programs and health-promotion media from HBHS program, with an average of 1.4 physical activity programs/components (19.7% of all assessed items) and 3.0 types of health-promotion media (43.0% of all assessed items) across churches. The availability of physical activity components was considered low in all intervention churches (10–29% of all assessed items), and the availability of health-promotion media was considered moderate to high in churches A and B (60 and 50%, respectively).

Table VII.

Availability of healthy foods, physical activity and health-promotion media components in church environment (average number and percentage of healthy items for each component)

	Meals		Snacks		Store/vending machine		Physical activity		Media
Church	n	%a	n	%	n	%	n	%	n	%
Church A	8.0	30	0.3	0	N/A	N/A	0.8	10	4.2	60
Church B	N/Ab	N/A	N/A	N/A	1.9	20	1.3	20	3.4	50
Church C	5.7	21	2.3	13	1.7	17	2.0	29	1.3	19
Average	6.9	25.5	1.3	6.5	1.8	18.5	1.4	19.7	3.0	43.0

^aTotal evaluated items/components for each component in the evaluation form (the denominator): meals = 27, snacks = 18, store/vending machine = 10, physical activity = 7 and media = 7. ^bN/A indicates this component does not apply to this church, same for other N/As.

Discussion

HBHS successfully demonstrated the feasibility of implementing a comprehensive community-participatory health program in multiple church sites. Overall, the program achieved moderate to high dose delivered, and fidelity. Print education materials were the most successfully implemented components with the highest dose and fidelity, whereas taste tests and giveaways were implemented at lower doses due to project design. The intervention components that were recalled with the highest success were giveaways and taste tests. Based on a subsample of church members, dose received of intervention components ranged from low to moderate. Similar levels of dose received were also seen in another church-based intervention program—Body & Soul, which also collected post-intervention information on individual exposure [30]. It would appear that a subsample of church members were most engaged in the program, and these were not necessarily the church members represented in the evaluation sample. Recognizing this provides insight for interpreting the impact of the intervention.

In the analysis of the impact of this trial (Gittelsohn J, submitted for publication), HBHS intervention church members (as compared with members in PP alone or comparison group) significantly decreased systolic and diastolic blood pressure (P = 0.02 and 0.01, respectively), showed greater improvement in food-related self-efficacy (P = 0.04) and had an increased purchasing of healthy foods (P = 0.06) and decreased purchasing of unhealthy foods (P = 0.02). However, no impacts were seen on physical activity level and weight indicators such as waist circumference and body mass index. Process evaluation allows further understanding of these results. We can see that the high dose delivered and fidelity of educational materials in this program aided the intervention in impacting multiple important health indicators and behaviors (blood pressure, self-efficacy and food purchasing). Finally, despite delivering the HBHS intervention with moderate dose delivered and fidelity, we were unable to detect significant differences in physical activity level, body mass index and waist circumference, indicating that we have a true need for more comprehensive and multilevel interventions to create impact on these factors rather than improvement in study implementation.

Evaluation of the HBHS trial had several challenges and limitations. Despite efforts to engage as many church members as possible in intervention activities, we were unable to measure participating members’ receipt of full intervention or the engagement level of the entire church population. Some church-wide activities occurred, such as distributing church bulletins that incorporated spiritual messages and HBHS intervention, but assessing dose received of these activities was difficult because our exposure data were only collected on a subsample of church members. Another difficulty of interpreting the program reach was the varying church attendance each week and the discrepancies between the observed and self-reported regular attendance. The calculated reach in this study may be underestimated due to an inaccurate report of church attendance (observed attendance was much lower). Future faith-based studies should use more valid methods of recording church attendance rather than relying on self-reported or roster data. Additionally, intervention contamination may have occurred in comparison churches because urban congregants may visit more than one church for weekly service, although they may have a primary affiliation. Recall bias may have occurred if respondents were not clear about whether the taste test/giveaway they saw was related to this program. Recall may also be influenced when intervention materials were similar in style with other church materials, especially the posters or educational displays. Other similar health materials were visible in churches as well, which were either developed by the church itself or by other health programs. The post-intervention exposure survey was a major strength of this study because it is not commonly used in process evaluation and demonstrated its great importance to reveal the less ‘memorable’ intervention components. However, we recognized that the survey participants may not necessarily represent all church members, as we had no knowledge about whether they differed from the non-participants.

The food and physical activity environment intervention strategies were developed over time based on the accumulated knowledge of the church environment. The in-church environment process evaluation started much later than the interactive session evaluation and was only conducted in a subsample of intervention visits. The relative small sample size of the church environment assessments can hardly reflect the church environment impact as a result of HBHS intervention, which was measured in other instruments and reported elsewhere (Farner et al., in preparation). The design of the in-church environment evaluation form failed to account for each HBHS intervention church’s unique food and physical activity environment and corresponding intervention approach. Future development of food environment evaluation instrument should allow for the flexibility to incorporate church-specific intervention strategies and to distinguish program-specific environmental changes.

To our knowledge, HBHS is the first church-based health intervention program to provide substantive process evaluation data combining reach, dose delivered, dose received, fidelity and the availability of health components in church environment. The two-level three-part approach to process evaluation in this study proved highly complementary. The discrepancy between the in-church and post-intervention assessments provided insight on the effectiveness of the intervention components. We speculate that interventions that more heavily involved participant interaction may be more readily recalled and recognized, whereas passive approaches are less likely to be recalled. The findings from the two-level evaluation of the HBHS project help us better understand the gaps between intervention delivery and an individual exposure, as well as strengthen our understanding of the impact of this specific trial (J. Gittelsohn, submitted for publication). Compared with previous church-based trials that reported primarily qualitative process evaluation results [20, 27, 30], these techniques used in HBHS can serve as a model for the future evaluation of community-participatory program efforts quantitatively. In addition, future process evaluation of community-participatory research should make full use of the process evaluation’s continual monitoring function to provide real-time feedback on interventions in conjunction with participant feedback to improve the intervention on a regular basis. A more systematic and simultaneous development of process evaluation instruments is essential to obtain comparable, interpretable and complementary results in multiple-level assessment.

Funding

This work was supported by ADA (grant number: 7-08-CR-23) and the Center for a Livable Future at the Johns Hopkins Bloomberg School of Public Health. A.C.S. was supported by the Center for a Livable Future pre-doctoral fellowship and a CURE supplement to the Johns Hopkins Training Program in Cancer Epidemiology, Prevention and Control from the National Cancer Institute, National Institutes of Health [T32 CA009314] during this project.

Conflict of interest statement

None declared.