Development and Testing of the Church Environment Audit Tool

Abstract

Objectives:

To describe development and reliability testing of a novel tool to evaluate the physical environment of faith-based settings pertaining to opportunities for physical activity (PA) and healthy eating (HE).

Methods:

Tool development was a multistage process including a review of similar tools, stakeholder review, expert feedback, and pilot testing. Final tool sections included: indoor opportunities for PA, outdoor opportunities for PA, food preparation equipment, kitchen type, food for purchase, beverages for purchase, and media. Two independent audits were completed at 54 churches. Interrater reliability (IRR) was determined with Kappa and percent agreement.

Results:

Of 218 items, 102 were assessed for IRR and 116 could not be assessed because they were not present at enough churches. Percent agreement for all 102 items was over 80%. For 42 items, the sample was too homogeneous to assess Kappa. Forty-six of the remaining items had Kappas greater than 0.60 (25 items 0.80–1.00; 21 items 0.60–0.79), indicating substantial to almost perfect agreement.

Conclusions:

The tool proved reliable and efficient for assessing church environments and identifying potential intervention points. Future work can focus on applications within faith-based partnerships to understand how church environments influence diverse health outcomes.

INTRODUCTION

Over the past decade, there has been a growing call to consider multilevel or ecological approaches to health promotion to boost the impact of health behavior change interventions.^1,2 These ecological approaches focus on making changes that target individual, social, physical and policy changes in environments where people spend time to work, play, and live, including parks, workplaces, schools, and faith-based settings.^1–3 Churches represent an important opportunity for health interventions, as they are key community settings that attract members of all ages, socioeconomic standings, and ethnic backgrounds. Comparable to workplaces and schools, churches are often composed of organizational hierarchies, unique social networks, strategies and practices, and at times, health-related goals.⁴

Although weekend worship attendance is slowly decreasing across the U.S.,⁵ roughly 36% of Americans report attending a religious service at least once per week and 33% report attending once or twice per month, totaling over 223 million regular attenders.⁵ Therefore, churches offer pronounced reach, including many members of communities that are frequently underrepresented in preventive health research,⁶ including people of low socioeconomic status,⁷ individuals with limited education,⁸ racial minorities,⁹ and rural populations.¹⁰ Consequently, communities of faith provide abundant opportunities for health promotion programs, reaching congregants in a way that is viewed as both socially and culturally acceptable.¹¹

One recommended strategy for improving community-level cardiovascular health in faith-based settings is creating environmental and policy changes.¹² Heart disease and stroke are currently the highest ranking causes of death worldwide, and physical inactivity and poor dietary habits are considered the main underlying behaviors contributing to these diseases.¹³ Churches often house a variety of resources that can be used to promote physical activity and healthy eating, including bulletin boards, kitchens, and fellowship halls, and thus are viable settings to intervene and address preventable health problems faced by congregants.

Indeed, several studies have identified church environments, both social and physical, as being related to health behaviors among congregants.^14–18 For example, perceived support in the church social environment has been associated with increased intake of fruits and vegetables and fiber and decreased intake of fats¹⁴ along with increases in physical activity.^16,18 Similarly, perceived physical environments, such as the presence of sports leagues, exercise facilities, exercise programs, and church-level social support, were positively related to physical activity engagement among congregants.¹⁶ However, evaluation of church environments and their relationship to physical activity and healthy eating in current research are often limited to congregant perceptions, and objective measures of church environments are lacking.^14–18

Assessments of environmental factors and their relationship to physical activity and eating behaviors have been conducted in a variety of settings, including neighborhoods, schools, and worksites. ^19–24 Generally-accepted methodologies for these assessments include collecting perceived measures via survey or interview data, the use of archival data, and objective observations conducted either by researchers or community members using validated audit tools.^25,26 Objective assessments using audit tools allow for the systematic collection of environmental data, may reduce bias in self-report, and can provide researchers and communities with real-time information about the physical environment as it relates to health outcomes.²⁵ Audit tools vary in length and the type of information collected depending on the setting in which they are conducted (eg, neighborhoods, schools), their intended purpose (eg, research, community improvement), and the health behavior of interest (eg, physical activity, healthy eating).²⁵ Validated tools to collect data about physical activity environments are commonly employed to assess neighborhoods and community features such as streets,^27,28 parks,^29,30 trails,³¹ and worksites.^23,24 Observational audits have also been performed to assess healthy eating environments in contexts such as neighborhoods,³² schools,³³ restaurants,²⁶ stores,^30,34 and worksites.²⁴

Purpose

Assessing the church environment using an objective tool is important for a comprehensive understanding of how structural features may impact health behaviors and outcomes. However, published tools to measure elements of the physical environment in faith-based contexts are limited³⁵ and have not been extensively tested. Therefore, the purpose of this paper is to describe the conceptual development and pilot testing of a church environment audit tool that can be used to evaluate the physical environment of faith-based settings as it relates to opportunities for physical activity and healthy eating.

METHODS

Study Design and Sample

Data were collected from Phase 1 of a 2-phase dissemination and implementation study of a faith-based physical activity and nutrition intervention in South Carolina. Phase 1 of the study is a group-randomized trial in a rural and medically-underserved community, with an estimated 23,363 residents and a larger population of Black/African American residents relative to the state as a whole (58.6% vs. 28.0%). All churches in the target county (N = 132) were invited to participate in the study and were recruited through phone calls, emails, and in-person visits as needed. Interested churches completed a telephone screening and were enrolled. Churches were eligible if they were located in the target county, had a membership of at least 20 individuals, and were willing to be randomized to either early or delayed training. Early churches attended a training session for 3 to 5 church committee members, including a program coordinator who was appointed by the Pastor; delayed churches received this training one year later.

Development

This paper describes the development, refinement, and testing of the audit tool developed for the parent project. Similar to the process used for other environmental audit tools, the Church Environment Audit Tool was developed through a set of iterative stages. The preliminary tool development was conducted by a team of 3 behavioral experts with experience in environmental audits and measurement in behavioral interventions. An initial step searching for existing tools to survey the physical activity and healthy eating environment of churches did not return any published instruments. As a result, the Church Environment Audit Tool was constructed using elements of other types of existing tools. For example, items about the indoor and outdoor physical activity opportunities at the church (eg, exercise equipment, walking paths) were derived from the Community Park Audit Tool (CPAT)²⁹ and the unpublished Congregational Health Index,³⁵ and supplemented with items created by the research team. Items about the availability and sale of food and beverages (eg, from vending machines or concession stands) were adapted from the Healthy Vending Toolkit,³⁶ and items about media on display with information about physical activity and healthy eating (eg, bulletin boards or brochures) were adapted from a tool developed by the research team for a previous faith-based intervention.³⁷ All other sections of the tool were drafted in a similar style to the sections adapted from other instruments, and were designed to assess the opportunities for healthy or unhealthy eating in the church, including kitchen items (eg, refrigerator, indoor grill, deep fat fryer).

A draft of the tool was presented to a larger team tasked with overseeing the measurement and evaluation processes for the whole intervention. The larger group met frequently and provided input and suggested revisions to multiple rounds of drafts of the tool until a full version was complete. The edited tool was then distributed to the study community advisory board as well as key contacts in the faith community for their input. The feedback of these community groups was presented to the measurement group and revisions were incorporated (eg, adding more categories of casual food vending that occur in local churches, such as donation/collection snack boxes).

The Church Environment Audit Tool has 7 sections, with each covering a different area of the physical activity and healthy eating environment, plus a brief space for recording information about the audit location, time, and data collectors. There are extensive instructions on the first page about data collection procedures, including definitions of key terms used in the tool (eg, present, usable/functional, good condition). The audit sections include: A – indoor opportunities for physical activity (42 items; eg, yoga mats, exercise videos); B – food preparation equipment (45 items; eg, stovetop, healthy cookbooks); C – type of kitchen (4 items; eg, commercial, standard); D – food for purchase (41 items; eg, vending machine, concession stand); E – beverages for purchase (53 items; eg, vending machine, concession stand); F – media assessment (6 items; eg, contents of bulletin boards, health brochures), and G – outdoor opportunities for physical activity (27 items; eg, walking track, playground). The full tool is available for download at: http://prevention.sph.sc.edu/Resources/churchaudit.html.

When a full draft of the tool was compiled and edited by the measurement work group, it was distributed to 5 content area experts for their constructive comments and suggestions. These experts had extensive experience in research and practice related to faith-based settings, environmental audits, and promoting organizational change around healthy eating and physical activity; all 5 worked externally from the main site of the study group. The experts were given background about how the tool would be administered (ie, teams of 2 would fill it out in rural churches), but instructions and context were kept to a minimum as a goal of the tool is to have broader applicability than the immediate setting of the present study. The experts provided independent feedback and review of the audit tool on a specific form, including issues seen with the tool as a whole, and specifically with content validity, potential for inter-rater reliability, and clarity in meaning and wording. Overall, the 5 experts reported that the tool appeared well-constructed, thorough, and appropriate for the setting in which it was designed to be used. Feedback from the experts led to a series of minor edits to the tool, including small additions (eg, adding the start and end times of data collection), wording changes (eg, deep fryer changed to deep fat fryer for clarity), and a few mentions of where training on how to code ambivalent situations would be key (eg, how deeply should data collectors engage with materials on church bulletin boards to make their ratings).

After incorporating the feedback from the expert reviewers, 3 measurement team members pilot tested the tool in 3 churches in the intervention county. Churches were representative of the variation in environment characteristics of the overall county, with a range of rural and urban, size, and early versus delayed training groups. The team members met with a church representative at each church to tour the facilities, taking notes on the tool and asking for input as needed to clarify tool items. The team debriefed after the pilot testing to compare notes and offer further revisions to the tool, including the inclusion of a script to help guide and standardize the interactions of the data collector and church representative, as well moving the section about outdoor physical activity opportunities to the end of the tool for ease of data collection.

Reliability Testing

To examine the reliability and other properties of the newly developed tool, each of the 54 churches participating in the FAN evaluation were audited by trained research staff between June through October 2016 (8 to 12 months after the training of early training intervention churches but before the training of delayed training intervention churches). As shown in Table 1, the churches varied with respect to a number of key attributes, including church size (37.0% had 25–49 members), predominant race of members (92.6% were predominantly Black/African American), and denomination (46.3% Baptist). Approximately 94.4% of the churches contained a fellowship hall, 96.2% had a standard or commercial-sized kitchen, and 55.6% and 57.4% had bulletin boards displaying physical activity and healthy eating information, respectively.

Table 1.

Characteristics of Participating Churches

Church Characteristic	Full Sample %(N)
Church size
< 25 members	13.0 (7)
25–49 members	37.0 (20)
50–74 members	25.9 (14)
75+ members	24.1 (13)
Predominant race of members
Black/African American	92.6 (50)
Caucasian	5.6 (3)
Multi-racial	1.9 (1)
Religious denomination
Baptist	46.3 (25)
Non-denominational or independent	20.4 (11)
Presbyterian	5.6 (3)
African Methodist Episcopal (AME)/AME Zion	13.0 (7)
Pentecostal	7.4 (4)
Methodist	5.6 (3)
Episcopal	1.9 (1)

Prior to data collection, research staff (primarily graduate students) participated in a 6-hour training session that focused on fundamental elements of the FAN project, strategies for collecting data in faith-based settings, and extensive practice and discussion of the primary FAN data collection instruments (church audit tool and a survey of church members). This training session included a visit to a local church to conduct a mock audit, which involved each staff member conducting independent ratings along with group discussion to clarify certain components of the tool.

Taking into account each staff member’s schedule and availability, data collection personnel were assigned to visit specific churches across the county (one per team per day). In order to facilitate inter-rater reliability testing, 2 staff members visited each church. Audits were conducted on each church’s day of worship (primarily Sundays) immediately before or after the service and were facilitated by a church staff member, often the person coordinating the FAN project activities for that site. In contrast to a park or street where most auditable amenities are accessible and viewable, it was deemed necessary to engage church personnel in the audit process for several reasons. These included the need to respect privacy (eg, to avoid searching through cupboards for items related to physical activity or healthy eating) and to ensure that key areas of the church facility were not missed (many churches have complicated floor plans often including annexes and separate buildings). Using a standardized script, data collection staff asked the FAN coordinator if the facilities or resources on the tool were available (eg, Do you have a bulletin board or table displaying physical activity or healthy eating information?). Subsequently, the FAN coordinator took the staff members to see each item in order to verify its presence and rate its usability, condition, and other details. Several resources, such as easily accessible kitchen areas or outdoor facilities, were audited with minimal or no involvement of the FAN coordinator. Although much of this process occurred jointly among the FAN coordinator and both auditors (in order to save time and reduce burden on the FAN coordinator), all ratings were assigned independently by each research staff member without discussion.

Analyses

Percent agreement and Kappa analyses were used to examine the inter-rater reliability of all items in the audit tool. Percent agreement is more appropriate when little to no variability exists among ratings (eg, all refrigerators are functional; few churches with vending machines present) and was considered acceptable if greater than 70%.^29,38 Kappa accounts for chance agreement between raters and was interpreted using established guidelines: 0–0.20 = slight, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = substantial, and 0.81–1.00 = almost perfect.³⁹

RESULTS

Of the 59 churches originally enrolled in Phase 1 of the overall research study, 54 took part in the evaluation and each was visited by a pair of trained data collectors. 35 churches were categorized as early intervention churches (65%) and 19 were delayed churches (35%). On average, the environmental assessments were completed in 19 minutes (range = 8–59 minutes). 108 individual audits were returned and assessed for completeness.

Of the 218 audit items appraised during on-site data collection, 102 were assessed for inter-rater reliability (Table 2). For the other 116 items, predominantly from the sections measuring food and beverages available for purchase, reliability could not be assessed because less than 3 pairs of ratings were available (eg, food and/or beverages were only available for sale at 2 churches).³⁰ Percent agreement for 99 of the remaining 102 items was over 80%, with 83 questions having agreement exceeding 90% (Table 2). For 42 of the 102 items (41%), Kappa could not be assessed or was not applicable because the sample of ratings was too homogeneous (eg, all churches had a refrigerator and all auditor pairs agreed a refrigerator was present). Twenty-five of the remaining 60 items (42%) assessed using Kappa had a value greater than 0.80, indicating almost perfect agreement; 21 items (35%) had a Kappa between 0.60 and 0.79, indicating substantial agreement; 6 items (10%) had a Kappa between 0.40 and 0.59, indicating moderate agreement; 2 items (3%) had a Kappa between 0.20 and 0.39, indicating fair agreement; and 6 items (10%) had a Kappa between 0.00 and 0.19, indicating poor agreement (Table 2).

Table 2.

Inter-rater Reliability of Church Environment Audit Tool Items

Tool section	Total items (# items)	Reliability not assessed (# items)a	Kappa						Percent Agreement
			Almost Perfect (0.80– 1.00) (# items)	Substantial (0.60–0.79) (# items)	Moderate (0.40– 0.59) (# items)	Fair (0.20– 0.39) (# items)	Poor (0.00– 0.19) (# items)	Not applicable (# items) b	>80% (# items)	<80% (# items)
Indoor opportunities for PA	42	10	9	5	2	0	1	15	32	0
Food preparation equipment	45	8	7	7	1	1	1	20	35	2
Type of kitchen	4	0	2	0	2	0	0	0	4	0
Food for purchase	41	40	0	1	0	0	0	0	1	0
Beverages for purchase	53	52	1	0	0	0	0	0	1	0
Media assessment	6	0	3	3	0	0	0	0	6	0
Outdoor opportunities for PA	27	6	3	5	1	1	4	7	20	1
Total	218	116	25	21	6	2	6	42	99	3

^aReliability not assessed because less than 3 pairs of ratings were available.

^bKappa could not be calculated based on homogeneity of the homogeneity of the sample.

With respect to specific tool sections, the majority of the items concerning indoor opportunities for physical activity, food preparation equipment, type of kitchen, media assessment, and outdoor opportunities for physical activity had either substantial or almost perfect Kappa scores, and all had high percent agreement (Table 2). Overall, reliability for all but 2 (deep fat fryer functional, deep fat fryer good condition) of the 102 audit items that were assessed was acceptable by at least one of the measures (ie, percent agreement or Kappa).

DISCUSSION

Churches and other faith-based organizations have a long history of involvement in both disease treatment and prevention.^4,11,40–42 Recently, the faith-based sector has been identified as a key strategic partner in health promotion.^37,43–49 To this extent, the 2016 National Physical Activity Plan specifically identifies faith-based settings as key partners for intervention to improve health.³ Disease prevention and health promotion efforts have been successful at delivering health information to congregants and community members through a variety of mechanisms, many focused on creating environments that support healthy eating and physical activity. However, few tools exist to effectively assess the church’s physical environment as it relates to physical activity and healthy eating. The Church Environment Audit Tool is a reliable and efficient resource that fills this gap.

Development of the Church Environment Audit Tool was a multi-stage process that included a search of existing literature, assessment of related environmental audit tools, expert review, and community advisory board review that resulted in a 6-page tool assessing 7 domains of the physical environment of faith-based organizations. The results of reliability testing indicated that the Church Environment Audit Tool is highly reliable for a broad range of questions across 5 out of the 7 church environment domains: indoor opportunities for physical activity, food preparation equipment, type of kitchen, media assessment, and outdoor opportunities for physical activity. For each of these domains, a majority of questions (over 75% for each section) could be assessed by either Kappa or percent agreement, which remained high overall for almost all questions. For those items with poor Kappa measures (a total of 6 across all domains), less than 6 pairs of ratings were available to be assessed indicating that the item was not common within churches in this study. These 6 items included the condition of activity/aerobic equipment, whether the deep fat fryer was functional, whether a walking/biking track was usable, the condition of lighting along sidewalks and paths, whether vacant land/lot was usable, and whether such lots were in good condition. A larger and more diverse sample of churches may have provided more robust data to assess the reliability of these uncommon items.

Two sections of the tool could not be fully assessed using either Kappa or percent agreement: food for purchase and beverages for purchase. These sections were not evaluated for inter-rater reliability in the present sample due to the homogeneity of churches visited (eg, few of the church facilities had either foods or beverages available for sale). While these sections may not have been assessed in the present sample, input from expert and stakeholder groups suggested that the information from these sections could be beneficial and will therefore remain as part of the tool in future iterations. Additional testing in a more heterogeneous sample of churches would be beneficial in assessing the reliability of these items, and plans include collecting these data as part of future community-based interventions. More specifically, future research will include training church congregation members in use of the Church Environment Audit Tool to assess and evaluate their church environment, similar to what has been done with citizen stakeholders for other environmental audit instruments addressing physical activity and healthy eating.⁵⁰ These data will be collected in a more diverse group of churches and will provide an opportunity to assess the reliability of items not evaluated in the present study.

The final version of the Church Environmental Audit Tool was 6 pages in length, including brief but detailed instructions, and took an average of less than 20 minutes to complete. Coupled with strong overall reliability, this suggests that the tool balances a desire to collect ample data with little burden placed on the potential user. Low user burden is particularly desirable in order to encourage wide-spread use among potential stakeholders or partners in health promotion programs. Likewise, minimal training was required to use the audit tool. Data collectors in this study completed a 6-hour training on all data collection procedures for Phase 1 of the FAN Dissemination and Implementation study. However, less than 2 hours were spent training to use the Church Environment Audit Tool, including a review of features to be assessed using the tool and a mock audit conducted at a church in close proximity to the training site.

The results of the reliability testing revealed several areas for possible improvements to the Church Environment Audit Tool moving forward. Three sections had at least one question with poor Kappa (indoor opportunities for physical activity, food preparation equipment, outdoor opportunities for physical activity), suggesting that specific items within these domains might need further editing or clarification in the future. Questions in these sections of the tool were designed to be answered in a 2-step process. In the first step, data collectors reported on the presence of an item on the audit. If an item was present, they were prompted to provide more details: “Is the item usable?” and “Is the item in good condition?” As described above, all items with low percent agreement or poor measures of Kappa were found in the second layer of questions. This distribution indicates that training to detect the presence of items was effective, but additional clarification to discern if an item is usable and in good condition would be beneficial for future data collection.

The development of the Church Environment Audit Tool demonstrates the potential for environmental audit instruments to be created for other health-promoting settings where such tools have not previously been used. While some venues, such as parks^29,30 and worksites,^23,24 have long been the focus of environmental interventions and measurement tools, opportunities exist to expand this work to other settings such as diverse health care facilities (eg, hospitals, older adult homes). The iterative process used in the present study could be extended to the development of future audit tools in these frequently-accessed settings, as well as inform the process of tool administration and reliability testing.

This study had several limitations. For example, all 54 churches were located in a single, rural county in the southeastern U.S., and using the Church Environment Audit Tool in other settings may prove more or less challenging than within this limited geographical context. Similarly, the variability of the church environments with regard to healthy eating and physical activity was limited for some items. Specifically, churches included in this sample seldom had food or beverages available for sale, and with less than 3 pairs of ratings, inter-rater reliability could not be assessed for items in those sections. As well, the observational audit tool described herein may not capture key elements of the church environment, such as food served at meals, policies affecting physical activity or healthy eating, or intensive details about the church media environment. Finally, the assessments conducted to test the Church Environment Audit Tool as part of this project were completed only by university-based data collectors. Other studies have created or adapted tools for use by community stakeholders in auditing parks²⁹ or neighborhoods²¹ and future research should examine the feasibility and reliability of similar evaluations undertaken by church members or leaders.

Several opportunities exist for future development and use of the Church Environment Audit Tool within research on faith-based settings and health. For instance, testing of the tool in diverse contexts and communities will likely elucidate additional items and elements that can be rated as part of such evaluations. Further, a simple yet comprehensive scoring protocol for the Church Environment Audit Tool has been developed to objectively assess and compare indoor PA, outdoor PA, HE, and healthy living media environments of churches. For indoor PA, outdoor PA, and HE opportunities, items present in a church receive 1 point. For items present, an additional 0.5 points were awarded if the item was “useable” and an additional 0.5 points if it was in “good condition.” If an item was present but was either not “usable,” 0.5 points were deducted from the score; likewise, if the item was not “in good condition,” 0.5 points were deducted from the score. Elements of healthy living media were awarded 1 point if they were present in the church, and no follow-up questions were asked about these items. Ultimately, in the present sample, total church environment scores ranged from 20.0 to 56.5 out of a maximum of 78 (M = 30.16, SD = 7.55).

The Church Environment Audit Tool can be incorporated into the growing number of interventions addressing faith-based settings and health promotion. For example, audits of the church environment may reveal strengths of the facility as well as opportunities for improvement, both of which can be targets of policy or environmental strategies to promote physical activity and healthy eating among church members. Finally, future data collection, including as part of community-engaged interventions, should involve congregation members, pastors and other leaders, and other key stakeholders (eg, local policymakers) in collecting and critically analyzing data about opportunities and challenges for obesity prevention within the church environment.

As research on the impact of the built environment on health expands into more settings where people live, work, and socialize, it is imperative that tools exist to objectively quantify these environments. This research provides a detailed description of the development and testing of one such tool, with methods that can be generalized to more settings as research about the built environment continues to grow and more tools are needed. Overall, results of testing of the Church Environmental Audit tool were promising. The findings reported here demonstrate that the tool allowed teams of trained data collectors to assess a large number of aspects of the church environment related to physical activity and health eating reliably and efficiently. As such, the tool provides researchers and stakeholders from churches with an objective method to measure the physical environment and assess its potential impact on congregational health. Future work using this tool should focus on its integration into existing health programs in faith-based organizations, and its further application to better understand how features of the church environment relate to health-promoting behaviors within faith-based partnerships.