Validity of a Measure to Assess the Child Care Nutrition and Physical Activity Environment

Abstract

Background

Licensed childcare centers, represent an opportunity to positively influence children's health behaviors. Valid and easy-to-use measures of the childcare environment are needed to assess the impact of environmental change on health.

Objective

To develop and validate a self-administered survey to assess the nutrition and physical activity environment of child care centers, and to identify domains which may be evaluated adequately through self-report.

Design

A survey was developed to assess four areas related to nutrition and physical activity: center policies, practices related to the social environment, physical environment, and nutrition quality. Development involved review of literature, existing measures, and regulations/standards; and collaboration with a working group. The survey was piloted and feedback sought from expert consultants. It was administered statewide and validated against a menu rating tool, a center director interview, and a direct observation tool developed for this study.

Participants/Setting

Participating sites were drawn from CACFP-participating licensed Connecting childcare centers serving 13 or greater 3 to 5 year olds. Survey responses from 146 center directors were included, as were 62 center menus, and director interviews and observational data from 33 sites.

Primary Outcomes/Statistical Analyses

Criterion validity of the survey was assessed through percent agreement with mirroring items in the additional measures. Healthy and unhealthy food scores were calculated for menu and survey tools, and Pearson correlations computed.

Results

Percent agreement with criterion outcomes ranged from 39 to 97%, with 61% of items achieving agreement at or above 80%. Agreement was highest for nutrition and policy domains, and lowest for physical activity and barriers to promoting health. Correlations between food scores across measures were moderate.

Conclusions

The self-report survey demonstrated adequate criterion validity; recommendations are made for improving validity of low-agreement items and for the use of more labor-intensive evaluation procedures for domains not adequately assessed through self-report.

Nearly 60% of three to five year olds attend a licensed child care center (1). Young children spend, on average, 28 hours per week in non-parental care (1), where it is suggested that they consume 1/3 to 2/3 of their daily recommended nutrition allowance (2) and engage in a majority of their daily physical activity (3). However, research has shown that preschoolers in childcare often do not meet nutrient recommendations (4) and are largely inactive (5). Thus, child care centers represent an opportunity to engage a large number of children in healthier behaviors (4).

A growing body of literature has documented the impact of environmental factors on children’s nutrition and physical activity. Environmental factors may be: 1) social (e.g., whether food is used to reward behavior); 2) physical (e.g., types of playground equipment); or 3) policy (e.g., nutrition standards for meals). Research has shown that length of outdoor play (6-9) and play equipment (3,7) influence children’s physical activity levels. Further, dietary habits are influenced by portion size (10-11), presence of high calorie, low nutrient-dense foods (12), the number of children at the dining table (13), teacher behavior and feeding style (14-16), and children’s involvement in mealtime set up and clean up (16).

The importance of the environment to health behaviors necessitates development of tools that accurately measure the nutrition and physical activity environment, particularly in childcare (17-18). A recent literature review of food environment assessments deemed this area a “nascent field” and noted that few researchers evaluate the psychometric properties of such instruments. (18) Similarly, a review of tools designed to assess the built environment's impact on physical activity revealed that the validity of self-report measures is rarely addressed (19).

Currently, two tools designed to assess the child care nutrition and physical activity environment have published psychometric properties. The Environment and Policy Assessment and Observation (EPAO) was developed to evaluate the Nutrition and Physical Activity Self Assessment for Child Care (NAP SACC) program, an environmental intervention (17). Through a one-day observation, EPAO measures several domains, including types of foods served to children, staff mealtime interactions, physical activity and sedentary opportunities, staff support, and the physical environment. It includes a document review of menus, handbooks, training documents, curricula, policies, and a playground safety check. While the EPAO has many merits, it is resource-intensive, requiring a full day of observation and thorough document review. Construct validity, predictive validity of the physical activity environment domains, and interobserver reliability have been published (7,17,20). The EPAO researchers are reviewing the tool for further validation (personal communication).

The NAP SACC intervention includes a self-assessment tool that allows child care providers to evaluate their facility's nutrition and physical activity environment (21). This instrument's validity and reliability are established, but it was designed to aide centers in identifying areas of improvement within their own sites (21) and may not be appropriate for researchers interested in studying the role of environmental factors across multiple child care centers.

We developed a self-administered child care director survey to assess the nutrition and physical activity environment of child care centers. The survey was designed to allow researchers to study environmental factors across a large number of child care centers. This survey development is part of a larger research project exploring the nutrition and physical activity environment in preschools serving low-income families. The present paper reports on the validity of the director survey, by comparing survey responses to observation, interview and document data.

Methods

Survey Development

We began with a review of existing measures (16,21-23,24 and U of Washington materials) and the public health and early education literature, as well guidelines from the National Association for the Education of Young Children (NAEYC), Head Start, Caring for Our Children (25), the National Association for Sport and Physical Education, and laws pertaining to the Child and Adult Care Feeding Program (CACFP) (26). We collaborated with the Connecticut (CT) Department of Education's Nutrition Education Coordinator, who works closely with the state's child care centers. The Robert Wood Johnson Foundation Working Group on Child Care also provided feedback. The survey was reviewed by three child care expert consultants, pilot-tested at three CACFP-participating sites, and modified according to feedback. These steps ensured that the survey had adequate content validity. The final survey contains 74 items and covers four broad areas related to nutrition and physical activity: center policies, practices related to the social environment, the physical environment, and nutrition quality. Response options for individual items varied according to the nature of the items.

Policies

The survey contains nine items addressing the strength of center nutrition and physical activity policies (e.g., staff use of food as reward for children's behavior). Four response options were provided: no policy, informal policy (spoken but not written), written policy (not included in parent handbook), and written policy (included in parent handbook). We consider a written policy included in a parent handbook to be the strongest form of policy. The purpose of creating a policy is to ensure that decisions are made in a consistent manner and reflect the values of the organization. Policies should be clearly written and publicly available in order to set the stage for effective and consistent implementation. If a question on implementation arises, the written policy can be consulted and changed if necessary to provide further clarity. If there is a change in personnel, written policies can remain to ensure that valued practices within the organization continue. Practices. Twenty-three items assessed center practices and aspects of the social environment (e.g., how health information is communicated to parents). Barriers to promoting healthy eating and physical activity practices (e.g., lack of support from teachers) were also assessed.

Physical environment

Three items assessed the availability of drinking water, the presence of 11 types of equipment, books, and posters, and the suitability of the indoor space for physical activity.

Nutrition quality

Thirty-six items addressed on how many days in the past week specific foods and beverages were served; response options included “none served”, “1-2 days”, “3-4 days”, and “5 days (every day).” Eleven additional items addressed the nutritional content of foods served in the past week (e.g., fat content of milk, for which response options were “none served”, “skim (nonfat)”, “1% lowfat”, “2% reduced fat”, “whole”, and “never served” ). Three items addressed the types of foods used in fundraising and center celebrations (response options varied across the three items).

Additional Measure Development

Three measures were developed to validate the survey instrument. These measures were designed to mirror items included in the survey to allow for direct comparison. An in-person Director Interview was created to help determine the validity of the practice and policy items on the director survey. A Direct Observation Tool was created to capture lunchtime practices, outdoor play practices, and the indoor and outdoor environment for comparison with survey practice and environment items. The Rudd Center Preschool Menu Rating Tool was created to assess the quality of foods offered, variety of foods served, and the degree of clarity in the menus. This tool renders a total menu quality score, as well as seven subscale scores for breakfast, lunch, snack, fruits and vegetables, meats and proteins, variety, and clarity. We defined “clarity” as the degree to which menus provided specific nutritional detail about the food. For example, a menu listing “milk” and “bread” would have a lower clarity score than onelisting “1% milk” and “100% whole wheat bread.” Clarity was deemed a critical component to the assessment of menu quality; if the menus do not contain this detail, evaluation of nutritional quality is hampered. Further, clarity may be viewed as a marker of transparent communication to parents: for menus with high clarity scores, parents will know exactly what children are served.

Final versions of all measures are available on the Rudd Center for Food Policy and Obesity website (www.yaleruddcenter.org).

Sample

Survey Sample

The survey sample included all Connecticut licensed child care centers that met the following criteria (N=221): participate in CACFP, serve 13 or more 3-5 year olds, and are not in-home facilities. CACFP, a federal program administered by the United States Department of Agriculture, subsidizes meals and snacks at child care, after school care, and adult day care centers. Reimbursement for meals and snacks is based on financial need (26). A total of 200 centers returned the surveys, for a response rate of 90%.

To ensure independence of observations, the sample for analysis was reduced. Of the 200 centers who returned surveys, 50 centers were independent centers not governed by a larger organizational body; these 50 were included in analysis. The remaining150 centers are governed by 31 different sponsor organizations. A sponsor often prescribes identical practices and policies among its participating centers. Among the 150 centers, 76 provided surveys with unique responses and were included in analysis. The remaining 74 participating centers are organized by 20 sponsors and returned surveys with identical information. Therefore, one center from each of these sponsors (for a total of 20) was selected to be included in analysis. In total, 146 centers were included in this analysis; this includes the 50 independent centers, the 76 sponsored centers with unique responses, and 20 sponsored centers representing the centers with non-unique responses.

Eighty-eight percent of the centers (n=195) returned a menu. One hundred and seventy-two (172) of these centers also returned the survey. Again, to ensure independence of observations, the menu sample was reduced to 62 for data analysis.

Site Visit Sample

Forty centers were randomly selected from the overall survey sample. Complete survey, menu, interview, and observation data were collected for all 40 centers. To ensure independence of observations, seven centers were removed, leaving 33 centers for analysis.

Sociodemographic and center characteristics of all three samples are provided in Table 1.

Table 1.

Characteristics of the Childcare Center and of the Census Block Group in which the Center is Located

	All Survey Respondents	Survey and Menu Respondents	Survey, Menu, Interview, and Observation Respondents
	n = 146	n = 62	n = 33
Center (mean (sd))
Number of children enrolled in school	73.7 (58.39)	78.6 (45.6)	70.1 (56.5)
Minimum enrollment age – years	2.1 (1.3)	2.0 (1.3)	2.4 (1.1)
Maximum enrollment age – years	6.1 (2.5)	6.4 (2.7)	5.8 (2.4)
% NAEYC1 Accredited	68.5	74.2	57.6
% Head Start	35.6	27.4	48.5
% CACFP2-reimbursed meals (%)
Breakfast	89.0	95.2	90.9
Morning Snack	23.3	16.1	24.2
Lunch	89.7	91.9	90.9
Afternoon Snack	89.7	90.3	81.8
Block Group (mean-sd)
Household Median Income	$47,653.75 (24,649.02)	$49,590.94 (26,229.84)	$43,539.39 (29,322.59)
% Residents with College Degree or Higher	22.0 (18.9)	25.5 (22.3)	17.6 (17.5)
% Hispanic	27.1 (26.5)	24.2 (26.7)	38.5 (32.5)
% White, Non-Hispanic	48.8 (33.2)	52.7 (32.7)	39.9 (34.8)

¹NAEYC: National Association for the Education of Young Children

²CACFP: Child and Adult Care Food Program

Procedure

Survey

An introductory letter and survey were mailed to center directors; completion of the survey and a weekly or monthly menu were requested. A $5 gift card to a regional supermarket was included as incentive. Follow-up phone calls and a second mailing were conducted to increase participation.

Site Visit

During the one-day site visits, two or three trained research assistants conducted the direct observations of the childcare environment and the 45-minute semi-structured director interview. Each research assistant independently collected information on the indoor, outdoor, and meal-time environment. In some cases, the interview was conducted with the education manager or the nutritionist in lieu of the director because either this individual had been the one to complete the survey or because the center director was unavailable. On average, visits were 3 to 4 hours in duration. Immediately following the visit, research assistants compared their observations and resolved any disagreements.

All methods were reviewed and approved by the Yale University Institutional Review Board.

Statistical Methods

Criterion Validity

For practice, physical environment, and policy domains, survey responses were compared to interview and observation data. Thirty-seven survey items were validated against the interview and nine survey items against the observation. To validate the nutritional quality segment of the survey, twelve survey items were compared to the menus. Some survey items were not validated because they could not be adequately or appropriately validated through these additional measures. For example, in a three-hour observation, it would be difficult to assess the frequency of computer use in the center or the frequency of nutrition instruction. Our goal in validating the survey was to arrive at a self-report measure that could be used in large-scale research projects to accurately assess many centers simultaneously. In using an interview with the survey respondent as validation, we were interested in whether there were items that required in-person probing and back-and-forth to arrive at an accurate response. For many of the items, the only accurate source may be the center director (for example, items asking about informal policies with no written documentation). While this validation strategy does not eliminate the possibility of a general bias pervading both the survey and interview due to social desirability or poor recall, our goal was to determine the most parsimonious way to pull the information from the center administration.

The validity of dichotomous data was measured by calculating the proportion of responses in exact agreement. For non-dichotomous data, the quadratic weighted percent agreement was calculated. Although percent agreement does not take into account agreement by chance, it is widely used and explicitly understood. If agreement is low, at least one of the instruments is incorrect. Agreements at or above 80% were considered to be in strong agreement (27). Cohen's kappa statistic is also commonly used in social science research to validate two instruments against each other in the absence of a gold standard (28). It is preferable to a percent agreement as it corrects for agreement due to chance. However, the kappa statistic is highly influenced by extremes in prevalence (high or low) of the measured characteristic in the study population (29). In particular, the probability for chance agreement is inflated when 2x2 tables have horizontal and vertical marginal totals that are symmetrically imbalanced (30), which occurred frequently in our sample. The Goodman and Kruskal's gamma statistic, another commonly used statistic, was also deemed inappropriate because it is unreliable when cells have zero values (31-32), which occurred frequently in our sample. Therefore, percent agreements were used as the sole measure of agreement.

To assess the validity of the survey's nutritional quality items, unhealthy and healthy food scores were calculated for both the menu and the survey and their degree of association was evaluated with the Pearson correlation statistic. While there are myriad potential ways of combining nutrition items to arrive at summary scores, we chose the parsimonious approach of using unweighted counts. The unhealthy food score was created for survey and menu food frequency data by summing the number of times breaded/fried meats, beef/pork, baked goods, and fried potatoes were served each week. Individual item scores ranged from 0 (“none served”) to 3 (“5 days/every day”). Possible unhealthy scale scores therefore could range from 0 to 12. The healthy food score was created by summing the number of times healthy food options were served each week. These healthier foods included fruits, vegetables, legumes, eggs, baked chips, and fish/poultry. Possible healthy scale scores could range from 0-18.

The statistical software packages SPSS Version 15 (33) and Stata Version 10 (34) were used to conduct all analyses.

Results

Validity

Results for the validity analysis are reported in Tables 2 and 3. A total of 48 comparisons were made for 46 survey items and two food score correlations (healthy and unhealthy). For the 37 survey items validated with the interview, the percent agreements ranged from 54.5% to 96.9% (Table 2). Of these, 27% had percent agreements at or above 90% and 57% had percent agreements at or above 80%. The survey items within this group that had highest percent agreements addressed the frequency of computer use and whether staff consumes the same foods as children at meals. The survey item that had lowest percent agreement addressed whether lack of time to teach nutrition was a barrier to creating a healthy child care environment.

Table 2.

Validity of Survey Items on Nutrition and Physical Activity

	Validation Method	N	No. Response Levels	Percent Agreement	Confidence Intervals
Nutrition (13 items)
Staff consume same foods as children	Interview	32	2	96.9	(90.5, 100.0)
Frequency of cooking activities	Interview	33	5	94.5	(92.0, 97.0)
Communication with parents via special events	Interview	33	2	93.9	(85.3, 100.0)
Food at celebrations	Interview	33	3	90.9	(82.1, 99.7)
Staff use food as a reward	Interview	33	2	90.9	(80.6, 100.0)
Frequency of parents receiving nutrition information	Interview	33	5	90.7	(86.2, 95.2)
Fundraising items	Interview	32	3	85.9	(75.2, 96.7)
Staff use food as a behavioral consequence	Interview	33	2	84.8	(71.9, 97.8)
Frequency of nutrition instruction	Interview	33	5	84.2	(77.5, 90.7)
Staff training on eating environment	Interview	29	2	69.0	(51.1, 86.9)
Staff sit with children during meals	Interview	33	3	63.6	(46.3, 81.0)
Staff consume unhealthy foods in front of children	Observation	31	2	90.3	(79.3, 100.0)
Staff sit with children during meals	Observation	32	3	88.3	(81.0, 95.5)
Staff consume same foods as children	Observation	31	2	87.1	(74.6, 99.6)
Water availability	Observation	32	3	82.0	(73.1, 91.0)
Physical Activity Environment (9 items)
Frequency of computer use	Interview	30	5	96.9	(95.7, 98.0)
Frequency of TV viewing	Interview	32	5	92.4	(85.5, 99.2)
Amount of active play	Interview	33	5	88.5	(83.2, 93.9)
Staff withhold physical activity as behavioral consequence	Interview	33	2	69.7	(53.1, 86.2)
Physical activity training	Interview	26	2	57.7	(37.3, 78.0)
Frequency of outdoor play	Observation	30	3	90.0	(82.4, 97.5)
Indoor space suitability for physical activity	Observation	31	3	80.7	(71.6, 89.7)
Staff withhold physical activity as behavioral consequence	Observation	32	2	81.3	(67.0, 95.5)
Physical activity books/posters in classroom	Observation	31	2	48.4	(29.8, 67.0)
Small equipment in classroom	Observation	28	2	39.3	(20.0, 58.6)
Policies (6 items)
Foods from home	Interview	33	4	87.8	(77.5, 98.0)
Celebrations at school	Interview	33	4	87.5	(76.6, 98.3)
Foods as reward	Interview	33	4	82.5	(73.3, 91.7)
Limits on computer use	Interview	31	4	81.7	(71.0, 92.4)
Physical education/physical activity	Interview	32	4	79.2	(67.8, 90.5)
Nutrition standards exceed CACFP requirements	Interview	33	4	70.0	(56.5, 83.5)
Barriers to Promoting Health (15 items)
Unhealthy food fundraisers	Interview	33	2	93.9	(85.3, 100.0)
Nutrition policies	Interview	33	2	90.9	(80.6, 100.0)
Inadequate food preparation/storage	Interview	33	2	84.9	(71.9, 97.8)
Physical activity policies	Interview	33	2	84.8	(71.9, 97.8)
Unhealthy food celebrations	Interview	33	2	81.8	(67.9, 95.7)
Quality of food service provider	Interview	33	2	78.8	(64.1, 93.5)
Teachers	Interview	33	2	78.8	(64.1, 93.5)
Parents	Interview	33	2	72.8	(56.7, 88.8)
Lack of staff training on nutrition education	Interview	33	2	69.7	(53.1, 86.2)
Limited opportunities for PE/quality of physical activity equipment	Interview	33	2	69.7	(53.1, 86.2)
Lack of nutrition education resources	Interview	33	2	66.7	(49.7, 83.6)
Lack of staff training on PE	Interview	33	2	66.7	(49.7, 83.6)
Lack of funding	Interview	33	2	63.6	(46.3, 81.0)
Lack of appropriate PE and PA resources	Interview	33	2	63.6	(46.3, 81.0)
Limited time to teach nutrition	Interview	33	2	54.5	(36.6, 72.4)

Table 3.

Validity of Survey Items on Nutrition Quality

	Validation Method	N	Pearson's Correlation
Nutrition Quality
Unhealthy Foods (4 items)	Document Review	61	0.260*
Healthy Foods (7 items)	Document Review	41	0.266

^*p<.05

Nine survey items were validated through observation, with percent agreements ranging from 39.3% to 90.0%. About 22% of items had percent agreements at or above 90%, and 78% of items had percent agreements at or above 80%. The survey item with highest agreement was whether staff consumes unhealthy foods in front of children. The survey item within this group that had lowest agreement addressed the presence of small play equipment.

The correlation between the unhealthy food score in the survey and the menu was moderate (r = 0.260; p < 0.05), as was the healthy food score correlation (r = 0.266) (Table 3).

Discussion

In this study we assessed the validity of a self-report survey measure of the nutrition and physical activity environment in childcare by comparison to data collected through in-person interview and direct observation. Items assessing child care policies and the nutrition environment had particularly strong agreement. Every policy item, with the exception of the policy on nutrition standards exceeding CACFP, had strong agreement, indicating that policies can be accurately assessed through self-report survey. The policy question on nutrition standards exceeding CACFP had lower agreement, likely due to a lack of item clarity. During interviews, respondents struggled to understand this item as it was intended. Given the important role policies play in creating healthy school environments (35), that our survey can accurately assess policies without a time-intensive document review is an important strength.

The nutrition environment is also accurately assessed with the survey, as 13 out of 15 items had strong percent agreements. Weaker agreement was found among two items. Survey-interview discrepancy on whether staff receives training on creating a positive nutrition environment was likely due to method variance: the survey required a categorical response, while the interview question was open-ended. Interview respondents were less likely to report that the staff sits with children during meals than their survey counterparts. This is an expected practice in child care. It is possible that the rapport built between interviewer and interviewee and the opportunity to elaborate on or explain responses encouraged interviewees to answer more truthfully; survey respondents may have been more influenced by social desirability.

For the physical activity environment, 6 out of 10 items had strong agreement. Weaker agreement occurred when comparing a categorical survey question to an open-ended interview question (physical activity training), from a lack of clarity on whether the item was referring to the center or to an individual classroom (presence of physical activity materials and small equipment), and from social desirability bias (staff withhold physical activity as a behavioral consequence) in survey reporting. Regarding this last item, withholding physical activity as a behavioral consequence is deemed an unacceptable practice in child care. Survey respondents were more likely than interviewees to indicate that this practice did not occur. It is likely that the interviewees responded more truthfully to this question for the same reasons outlined above in the case of sitting with children during meals.

Lastly, only one-third of the items on barriers to promoting a healthy environment had strong agreement. Interviewees tended to identify more barriers than survey respondents. These barrier questions were subjective, opinion-based items, on which achieving agreement is more difficult than on policy or environment items. The interview process may have allowed respondents to become more comfortable with the researchers and therefore more willing to disclose negative information. Additionally, engaging in a 45 minute interview, as opposed to a completing a shorter survey, may have afforded respondents greater opportunity to identify additional barriers. For these reasons, an interview may be superior to self-report survey for understanding barriers to health promotion in child care centers. It is important to note that when the barrier items are excluded, the percent of survey-interview items with strong agreement increases from 57% to 73%.

Correlations between survey and menu food scores were moderate. Survey responses and the menu review were completed during different time periods, which may have compromised agreement. Agreement on nutrition quality may be improved if the timing of the two measures is better aligned. Additionally, while the survey mentions “last week's menu” in the heading of the food frequency section, clearer directions to review the prior week's menu specifically, may also improve accuracy. Survey respondents may have provided a slightly more favorable view of the foods served than what was documented in the menus. Directors provided a more favorable picture for the frequency of serving three types of foods (legumes, beef/pork products, and baked goods), a less favorable picture for one type of food (fried potato products), and a similar view for the remaining seven foods. However, considering the discrepancy in timing during which the survey and the menu review were completed, it is ultimately difficult to assess if these differences were due to bias in reporting or to a genuine difference in what was actually served.

In general, survey respondents tended to provide more favorable responses than interview respondents. It is possible that survey respondents felt slight pressure to provide favorable responses because the survey was administered in conjunction with the CT Department of Education, the agency that oversees CACFP. During the interview, it was stressed that the purpose was to learn more about child care and not to audit centers or document poor practices. Additionally, interviewees could talk through answers and interviewers could ask follow-up questions, allowing for increased accuracy and explanation of responses. Lastly, due to the interview length, child care directors sometimes asked that other administrators, such as education managers or head teachers, complete the interview, and these individuals may have felt less pressure to represent the center in an overly positive light or may have had a slightly different perspective on certain items.

The observation did not appear to yield more accurate responses than the survey. In fact, when evaluating certain practices, such as withholding food as a punishment, the survey is a more accurate method, as it is difficult to capture sporadic or less frequent practices in a single day of observation when teachers may be particularly aware that their behaviors are being observed.

Similar to our measure, the self-report responses in the NAP SACC instrument validation study were more favorable than the data collected objectively (21). Benjamin and colleagues reported lower validity among items assessing center behavior (i.e. fundraising practices) and the overall environment, while items related to the provision of physical activity had higher validity scores. In contrast, we had higher agreement among items assessing staff behaviors, policies, and general practices (i.e. frequency of nutrition instruction and computer use) and lower agreement among items examining perceived barriers and the physical environment. At this time, it is difficult to compare our survey's validity to the EPAO measure, as its criterion validity has not yet been established. However, both instruments have undergone considerable review by child care experts and were developed on the basis of existing literature and guidelines, thereby establishing reasonable construct validity.

Our director survey is unique when compared to other measures with published psychometric data. First, it has been designed to allow researchers to examine predictors of the strength of the nutrition and physical activity environment and to study environments across child care centers. In contrast, the EPAO instrument was designed to evaluate an intervention, and the NAP SACC tool was created to assist child care directors in identifying areas of improvement within their own centers. Second, while other measures do address center policies to some degree, our survey examines policies in greater depth, as we inquire whether the polices are informal, written (but not in the parent handbook), or written in the parent handbook, as opposed to whether or not the center simply has a policy in place. Emphasis was placed on identifying where policies are located, as policies in parent handbooks are most publicly accessible, thus most likely to be upheld by parents and staff as perceived accountability is high. Third, we included questions on barriers to promoting a healthy environment in order to understand, from the director's perspective, what must change in order to create healthier centers. A fourth significant difference is that our survey was designed and validated among child care centers serving children in some level of economic need, as all of the centers participated in CACFP. While we do not know the racial and ethnic composition of the children in our participating child care centers, we do know that the neighborhoods in which the centers are located are ethnically diverse. In contrast, the NAP SACC measure was tested among centers in which 81% participate in CACFP and where the children attending the child care centers were ethnically more homogeneous (21). McKinnon and colleagues note that few instruments have been developed that assess the food environments of low-income and racial/ethnic minority populations (18). Therefore, our survey fills an important gap in research, especially in light of the fact that minority and low-income populations are at greatest risk for poor health outcomes related to sub-optimal nutrition and physical activity (36).

Validity of the director survey will likely increase with recent revisions to the survey. These include clarifying questions with weaker agreement, clarifying instructions, ensuring that the individual most familiar with daily practices completes the survey, and administering the survey without the collaboration of the agency providing oversight of the meal program in order to reduce socially desirable responses. Our group plans to field test the revised instrument in future research, and the tool is available to other interested researchers.

There are several limitations to this study. First, the sample size for the majority of items, excluding the nutrition quality questions, was relatively small. Second, results from this study are generalizable only to CT CACFP-participating centers; replication in other regions and in additional populations is recommended. Third, observation data were collected during a single day; observation over a longer period would have allowed for more accurate evaluation of some of the policy and practice items from the survey. Lastly, it is possible that staff behavior was affected by the observation; specifically, staff may have followed policies more closely or adhered more to best practices as a result of being observed. Again, observation over a longer period would help to address this issue by allowing for an acclimation period.

Conclusions

Overall, the self-report instrument demonstrates adequate criterion validity, providing a thorough assessment of the nutrition and physical activity environment of child care centers. Several measures of the nutrition and physical activity environment exist, but few researchers have assessed the psychometric properties of these measures. According to a review conducted by McKinnon and colleagues, when psychometric testing of food environment measures is conducted, it tends to focus on reliability, including test-retest, inter-rater and internal consistency; validity is rarely addressed (18). Similarly, among physical activity self-report and observational measures, validity is rarely reported (19). Valid measures are needed to assess the relationship between the environment, behaviors, and health. This survey provides researchers with a valid, cost-effective method of measuring the child care nutrition and physical activity environment. This survey is straightforward to administer, places significant focus on center policies. It was created specifically for researchers interested in understanding the environment across centers and predictors of the quality of the nutrition and physical activity environment. Given the survey's emphasis on policy, it has potential to inform state and federal policies that influence the nutrition and physical activity environment in child care centers.