Development and testing of mobile technology for community park improvements: validity and reliability of the eCPAT application with youth

Abstract

Creation of mobile technology environmental audit tools can provide a more interactive way for youth to engage with communities and facilitate participation in health promotion efforts. This study describes the development and validity and reliability testing of an electronic version of the Community Park Audit Tool (eCPAT). eCPAT consists of 149 items and incorporates a variety of technology benefits. Criterion-related validity and inter-rater reliability were evaluated using data from 52 youth across 47 parks in Greenville County, SC. A large portion of items (>70 %) demonstrated either fair or moderate to perfect validity and reliability. All but six items demonstrated excellent percent agreement. The eCPAT app is a user-friendly tool that provides a comprehensive assessment of park environments. Given the proliferation of smartphones, tablets, and other electronic devices among both adolescents and adults, the eCPAT app has potential to be distributed and used widely for a variety of health promotion purposes.

Childhood obesity is a significant public health issue with rates having doubled in children and quadrupled in adolescents over the past three decades [1]. This is particularly disconcerting because children who are overweight are 70 % more likely to be overweight or obese as adults putting them at increased risk for numerous health concerns [2]. Physical activity (PA) can significantly reduce the risk of childhood obesity and obesity-related chronic diseases; however, youth PA declines with age [3–5]. Modifying the built environment of neighborhoods and communities is recognized as a promising solution to the population-level obesity crisis [6, 7]. In particular, parks are key venues for youth PA, given their low cost and legislated ubiquity [8, 9]. A growing body of evidence suggests that a variety of park variables, especially the availability and condition of features within parks (e.g., playgrounds, trails, lighting, landscaping), are strongly related to their use for PA [10–13]. Further, research suggests that environmental improvements to parks, playgrounds, and other community resources can promote increased PA and other health outcomes among children and adults [14, 15].

Creating healthy communities, including better parks, will require the interest and participation of multiple constituencies [16]. For several reasons, youth can and should be an integral part of this change process. For example, youth voices can be especially powerful in influencing the priorities and decisions of policymakers [17, 18], and engaging youth in advocacy and community change efforts has critical implications for the development of the youth themselves and for the future of our public leadership [17–19]. Indeed, Millstein and Sallis referred to youth advocacy for obesity prevention as the next wave of social change for health [20].

While promising, advocacy for policy, systems, and environmental (PSE) change is an understudied and underevaluated approach [21]. The process of improving neighborhoods and parks will take time, but preparing today’s youth to be the future leaders of healthy communities is a crucial first step [18]. Accomplishing this will require finding ways to involve youth in PSE change efforts in ways that are appealing and engaging to them [18, 20]. One innovative technique involves using established audit tools to evaluate the health-promoting potential of community environments and then to work with this data to develop, implement, and evaluate a PSE action plan to create healthy community changes [22, 23]. Specific to parks and recreation resources, the Community Park Audit Tool (CPAT) was designed to aid citizens and community groups in planning and advocating for parks that promote PA, prevent childhood obesity, and contribute to overall healthy community design [24]. Originally, the CPAT was developed and widely used as a paper-pencil, user-friendly tool that enables diverse community stakeholders to quickly and reliably audit community parks for their potential to promote PA. The CPAT contains four sections: park information, access and surrounding neighborhood, park activity areas, and park quality and safety. It provides in-depth information regarding the presence/absence of 14 park facilities and 25 amenities as well as park quality and safety characteristics. When used in paper-pencil format among adults, it has demonstrated strong content validity and inter-rater reliability, with percent agreement for the vast majority of the items in the tool between 80 and 90 % [24]. However, initial experiences conducting park and neighborhood audit workshops with youth suggest that technology-based methods would be considerably more engaging than current paper-and-pencil tools [25]. Indeed, substantial research has shown that youth are frequently the earlier adopters of new technologies and that such technologies provide a more interactive and hands-on way for youth to engage with their local communities, thereby appealing to youth who might not normally take a leadership role in health promotion efforts [26–29]. Specifically, research indicates multiple benefits of using technology within youth participatory frameworks such as increased self-efficacy [30], improved motivation [31], enhanced voice in the community [32], heightened communication with adults [33], promotion of equitable power sharing [32, 34], and provision of political or social agency [27, 31]. Technology can also improve youth empowerment by combating common issues with youth participation such as fighting apathy [31], supporting reflective thought [35], improving self-sufficiency [36], reducing anxiety [32], and increasing civic engagement [32, 37]. A more complete review of literature on the use of technology for youth engagement in participatory processes can be found elsewhere [38].

This study addresses several gaps in the literature regarding youth, technology, and environmental audit tools to date. First, despite the existence of several types of observational environmental data collection tools [39–41], few have been developed and tested with diverse populations in mind, especially youth [42]. For example, DeBate and colleagues [43] evaluated the utility of the Physical Activity Resource Assessment tool [44] to assess child PA intervention environments and found that while useful, not all child-related environmental issues were captured with the tool. Additionally, they noted that the tool was biased toward larger resources and undervalued small, but safe locations for youth PA [43]. Similarly, Kaczynski and colleagues [24] summarized existing park audit tools and noted that few were youth-oriented and those that did exist were less user-friendly (i.e., longer completion time/length, more complicated). Further, limited research has explored the reliability and validity of environmental data collection tools with community stakeholders [45, 46]. For example, Moudon and Lee [47] noted that many tools designed for community stakeholder assessment of walking and bicycling environments are typically less detailed than those designed for research purposes and many have not been assessed for reliability. Moreover, while several researchers have developed tools intended to audit environmental characteristics that support youth PA [24, 48], the reliability and validity of these tools have not been assessed with youth populations. Finally, despite the benefits of technology for youth engagement in civic processes, to date, none of the existing park audit tools are available in an electronic format. The growing involvement of youth in civic processes coupled with a lack of tools validated among this population highlights the need for further research efforts. Consequently, additional development and testing of electronic data collection tools for use by youth is warranted. Therefore, to advance this research and practice agenda, the purpose of this paper is to describe the development and validity and reliability testing of an electronic version of the Community Park Audit Tool (eCPAT) for use by youth.

METHODS

eCPAT app development

Multiple iterative stages were used to comprehensively develop and test the eCPAT app. Initially, an extensive literature review of youth, technology, and health advocacy identified theoretical frameworks and key methodologies for developing mobile applications to engage youth in health promotion efforts [20, 26, 49, 50]. Specifically, the larger project combines theoretical underpinnings from Millstein and Sallis’ model of youth empowerment and advocacy for obesity prevention [20] with the Flicker and colleagues e-PAR framework [26] to better understand how technology contributes to how youth might be seen as valuable resources that can create healthy social and environmental change in their communities. The combination of these frameworks highlights the use of technology as a format that in and of itself can increase youth engagement in participatory action activities, which in turn improves youth levels of empowerment and advocacy for community change efforts. Additionally, to aid operationalization of technology characteristics that contribute to a youth’s experience and engagement with our application, we based technology development and testing on O’Brien and Toms model of user engagement [49]. This theoretical framework summarizes four distinct stages of technology engagement (point of engagement, engagement, disengagement, and reengagement) and mobile application attributes corresponding with each stage (e.g., interface aesthetics, sensory appeal, control, usability, positive/negative affect). These attributes formed the foundation of application development and capacity testing and aided transdisciplinary interactions across study personnel. To further inform application development, key informant interviews (n = 5) were conducted with academic and technology experts in youth advocacy for obesity prevention, health information technology, and technology within parks and recreation settings. Key informants commented on application format, design, functionality, and preferred operating systems and mobile devices. As well, key informants offered advice regarding what should be considered important when designing an electronic tool that is (1) focused on park-based PA, (2) user-friendly, and (3) engaging to youth. Linking this information to technical programming design, a team of health promotion and computer science academics used PhoneGap [51] (a cross-platform framework that allows application design for both Android and iOS platforms) to create the eCPAT application for use on Android tablets. Technical application development phases followed standard system design protocol and included a system requirement analysis, software design, program coding, and unit alpha (capacity) testing by computer programmers [52]. Concurrently, a preliminary Microsoft SQL database on a web server at the University of South Carolina was designed to house wireless data transfer from the eCPAT app upon data submission. Upon application and server design completion, a second round of extensive capacity field testing of both the eCPAT app and wireless data transfer and storage was conducted. Finally, the application was beta-tested with a small cohort of youth to obtain feedback which was used to improve and refine the application. The resulting eCPAT app consisted of two main interface screens, including a home page (Fig. 1a) with park auditing instructions, icon legend, and login button, as well as a single, scrolling data entry screen (Fig. 1b) of 149 items under four main headings (i.e., Park Information, Access and Neighborhood, Activity Areas, Quality and Safety) that contained all items from the original CPAT tool. The eCPAT app incorporated text instructions and definitions (e.g., a description of an activity area such as a splash pad) as well as example pictures (e.g., photos of a splash pad) directly into the data entry interface in an effort to support tool validity. Answer validation (as indicated by the green checkmarks; Fig. 1b) and wireless data transfer were also included as a way to ensure complete data collection and reduce data entry error. The eCPAT app also included enhanced data collection technology capabilities over the original CPAT. For example, the eCPAT app integrated a camera function that can take photos associated with items assessed to be stored within the app and uploaded via Wi-Fi or direct transfer. Photos can be used to provide supplemental explanation of audit items as well as aid a variety of participatory action processes such as PhotoVoice [53]. Additionally, using the global positioning system (GPS) within the device, the eCPAT application can collect latitude and longitude coordinates for all activity areas which can then be exported into geographic information systems (GIS) software. Technology features of the eCPAT app were designed to enhance tool functionality and usability as well as support tool reliability and validity which is the focus of this study.

Fig. 1.

eCPAT application screenshots

Study setting and data collection

Data for this study was collected in June 2014 in Greenville County, SC, as part of the eCPAT Project, a randomized control study exploring the use of technology to improve youth empowerment and advocacy for community health promotion efforts. As part of the larger project, 150 youth aged 11–18 years were recruited through existing youth groups and programs to garner a broad cross section of participants. Recruitment methods included distribution of a recruitment flyer through email and hard copies to Greenville County schools, after school groups, and parks and recreation programs, as well as a recruitment booth at a local youth summer park program event. Blocked randomization using a random number generator was used to allocate youth into one of three study conditions that used the paper CPAT tool, the eCPAT application, or a no-audit control group ensuring similar group sizes (approximately 50 per group). Additionally, after serving as the control group, a subsample of youth completed park audits using both the CPAT and eCPAT formats. Data related to technology and youth empowerment and engagement as well as qualitative exploration of youth experiences are addressed in separate manuscripts [38, 54]. This paper reports on validity and reliability data collected from all youth who used the eCPAT application during the course of the larger project.

All youth participants completing park audits attended an hour-long project meeting that included a brief overview of the project and audit tool training (30 min) that consisted of basic instructions, definitions, and an app navigation demonstration. For example, youth were shown how to rate items and take pictures using the app as well as instructed on basic observational audit protocol (e.g., be systematic, avoid taking photos of park visitors). As well, youth completed an on-site practice park audit at an adjacent park (30 min). Youth were then randomly assigned two parks each (paired with a different youth for each park) and asked to complete their park audits independently. Greenville County has a total of 103 parks that vary with respect to size (0.1–293.2 acres), quality, features, neighborhood composition, and geographic dispersion. A sample of 50 parks was chosen to represent a diverse mix of park and neighborhood characteristics while staying within a 30-mile radius from the City of Greenville center to alleviate travel concerns (Fig. 2). Youth were expected to obtain their own transportation to the park, but all park visits were coordinated over the course of a week so youth could be supervised. Project staff were present at each park visit for record keeping and safety/liability purposes and asked to clarify only basic instructions given during training, but not to provide answers to audit items.

Fig. 2.

eCPAT project parks—Greenville County, SC

Quantitative park audit data were captured in each park by youth using the newly developed eCPAT application on Google Nexus 10 tablets provided for them. To examine the validity of youth’s ratings, during the same timeframe, a trained researcher (the lead author) completed a gold standard audit using the eCPAT application in all study parks. All eCPAT park audit data was transferred wirelessly to an encrypted server for data analysis upon audit completion. Study personnel explained details of the study, and written parental consent and youth assent were obtained prior to youth participation in the project. As a thank you for completing all project tasks (pre- and post-surveys, training workshop, two park visits lasting 30–60 min each, and providing own transportation to parks), youth received a $50 gift card for participating in the study. This study occurred in collaboration with Greenville County Parks, Recreation, and Tourism; the City of Greenville Parks and Recreation Department; and LiveWell Greenville and was approved by the Institutional Review Board (IRB) at the University of South Carolina.

Analysis

To examine eCPAT tool validity and reliability, this study utilized data from youth who conducted park audits using the newly developed eCPAT application. Cohen’s kappa [55] and percent agreement [56] statistics were used to examine (i) criterion-related validity when youth audits for a park were compared to those of a gold standard researcher and (ii) inter-rater reliability among paired youth eCPAT ratings of the same park (note: youth were randomly chosen as the validity comparison for each park) [57, 58]. Both kappa and percent agreement are valuable measures for environmental audits because percent agreement statistics are robust when there is little variability in features being rated or ratings by auditors, while kappa statistics account for chance agreement between raters [8, 59]. Further, it has been suggested that reporting the proportion of agreement alongside kappa values could help the reader understand possible prevalence or bias effects in the data [60–62]. Validity and reliability ratings were only calculated for items for which at least three pairs of ratings were available across the sample of parks [63]. Percent agreement statistics were evaluated using the following established criteria: 75–100 % = excellent; 60–74 % = moderate; and less than 60 % = poor [64]. Observed kappa statistics were interpreted using guidelines provided by Landis and Koch: 0.81–1.00 = almost perfect to perfect agreement; 0.61–0.80 = substantial agreement; 0.41–0.60 = moderate agreement; 0.21–0.40 = fair agreement; and 0.00–0.20 = slight or poor agreement [55].

RESULTS

Data from a total of 52 youth were used in the present analyses. Youth participant characteristics are shown in Table 1. Youth ranged from 11 to 18 years of age (M = 14.0, SD = 1.6) and were fairly split between middle and high school grades, with just over half (58 %) in or starting high school. The majority of youth were female (63.5 %), white (63.5 %) or African American (26.9 %), and had a normal body mass index (BMI; 84.0 %). A large portion of youth indicated they were meeting PA recommendations (84.3 %), and just over half of participants indicated they had visited a park in the past 30 days (58.8 %). With respect to technology, almost all youth had access to mobile technology (98.0 %), and over half indicated high levels of regular technology use (56.0 %).

Table 1.

Youth participant characteristics

Characteristic	Number (%)
Total	52 (100.0)
Age
Middle school (12–13 years)	21 (42.0)
High school (14–18 years)	29 (58.0)
Gender
Male	19 (36.5)
Female	33 (63.5)
BMI
Underweight (<5 %)	3 (6.0)
Normal (5–84.99 %)	42 (84.0)
Overweight (85–94.99 %)	2 (4.0)
Obese (≥95 %)	3 (6.0)
Race
White	33 (63.5)
Black	14 (26.9)
Other	1 (1.9)
2 or more races	4 (7.7)
Physical activity
Meets PA recommendations	43 (84.3)
Park use
In the last 30 days	30 (58.8)
Mobile technology access
Access to mobile device	51 (98.0)
Mobile technology usage
High level of technology usage	28 (56.0)

Originally, a sample of 50 parks was selected for this study. However, due to attrition, three parks lacked paired validity ratings and four parks lacked paired inter-rater reliability ratings. This resulted in a final sample of 47 parks with validity ratings and 46 parks with reliability ratings. Selected characteristics of the 47 parks are shown in Table 2. Parks ranged in size from 0.3 to 36.7 acres (M = 9.8, SD = 10.0) and had a diversity of features ranging from 1 to 26 activity areas per park, with an average of almost 6 activity areas per park (M = 5.9, SD = 4.1). Parks were geographically dispersed across five park and recreation districts throughout Greenville County, with just over half (53.2 %) located in the City of Greenville. Parks were located across neighborhoods (census block groups) that were diverse with respect to household income and racial composition. On average, park neighborhoods had a mean household income of $44,900 and were composed of an average of 40.6 % racial minority population.

Table 2.

Study park characteristics

Characteristic	Number (%)
Total	47 (100.0)
Size (acres)
0–4.99	23 (48.9)
5–9.99	5 (10.6)
10–14.99	7 (14.9)
≥15	12 (25.5)
Activity areas per park
1–3	12 (25.5)
4–6	20 (42.5)
≥7	15 (32.0)
Location by district
City of Greenville Parks and Recreation	25 (53.2)
Greenville County Parks, Recreation, and Tourism	14 (29.8)
City of Mauldin Parks and Recreation	4 (8.5)
City of Simpsonville Parks and Recreation	3 (6.4)
City of Greer Parks and Recreation	1 (2.1)
Neighborhood median income (quartiles)a,b
Lowest	12 (25.5)
Second	12 (25.5)
Third	12 (25.5)
Fourth	11 (23.4)
Neighborhood minority population (%)b
0–24	20 (42.6)
25–49	11 (23.4)
50–74	7 (14.9)
75–100	9 (19.1)

^aIncome quartiles ($): 16,321–24,306; 24,307–43,095; 43,096–56,856; 56,857–112,500

^bNeighborhood income and minority proportion are based on data from the American Community Survey 5-year estimates (2008–2012) for block groups containing park area

The eCPAT application collected information regarding 149 distinct items, of which 18 items had an insufficient number of ratings (i.e., fewer than three pairs) for accurate validity or reliability to be determined [63]. Such items often consisted of infrequent or less common park facilities such as fitness stations or skate parks. Further, for 41 items, kappa statistics could not be calculated or were inappropriate due to insufficient item variability (i.e., less than 10 % based on trained researcher audits), in which case percent agreement was used. This resulted in 90 items examined using both Cohen’s kappa and percent agreement, while the remaining 41 items were examined using only percent agreement. Validity and reliability results are shown in Table 3. With respect to criterion validity, over 70 % of items demonstrated fair or moderate to perfect validity. Specifically, kappa statistics between the trained researcher and youth auditors demonstrated moderate to perfect kappas for 40.0 % of items, fair validity for 32.2 % of items, and poor validity for 27.8 % of items. For the 41 items that explored validity using only percent agreement between the trained researcher and the youth auditor, all but two items demonstrated excellent agreement exceeding 75 %, with most items well above 90 %.

Table 3.

eCPAT validity and reliability

	Validity			Reliability
eCPAT Item	Pairs of ratings	Kappa	% Agreement	Pairs of ratings	Kappa	% Agreement
Access and surrounding neighborhood
Can the park be accessed for use?	47	–b	89.4 %	46	–b	84.8 %
Are there signs that state the following?
Park name?	47	0.897	97.9 %	46	0.646	89.1 %
Park hours?	47	0.517	80.9 %	46	0.562	80.4 %
Park contact information?	47	0.003	44.7 %	46	0.203	71.7 %
Park/facility rental information?	47	0.287	76.6 %	46	0.777	95.7 %
Park rules?	47	0.236	61.7 %	46	0.397	69.6 %
Park map?	47	0.486	89.4 %	46	0.238	89.1 %
Park equipment rental?	47	–b	97.9 %	46	–b	95.7 %
Park event/program information?	47	0.082	80.9 %	46	0.179	87.0 %
None present	47	–b	97.9 %	46	–b	93.5 %
How many points of entry does the park have?	47	0.314	51.1 %	46	0.434	63.0 %
Is there a public transit stop within sight of the park?	47	–b	91.5 %	46	–b	84.8 %
What types of parking are available?
None	47	–b	91.5 %	46	–b	87.0 %
Parking lot	47	0.663	85.1 %	46	0.849	93.5 %
On street parking	47	0.732	87.2 %	46	0.284	65.2 %
Bike rack(s)	47	0.555	91.5 %	46	0.330	87.0 %
Are there sidewalks on any roads bordering the park?	47	0.654	83.0 %	46	0.518	76.1 %
Are the sidewalks usable?	23	–b	95.7 %	20	–b	95.0 %
Are there curb cuts?	23	0.058	69.6 %	20	−0.053	60.0 %
Is there an external trail or path connected to the park?	47	0.214	63.8 %	46	0.513	76.1 %
Is the external trail useable?	5	–b	100.0 %	14	–b	92.9 %
Are there bike routes on any roads bordering the park?
Bike lane	47	0.286	85.1 %	46	0.385	89.1 %
Bike route sign	47	–b	95.7 %	46	–b	93.5 %
Share the road signs/markers	47	–b	95.7 %	46	–b	91.3 %
Bike routes none	47	0.376	85.1 %	46	0.328	82.6 %
Are there nearby traffic signals on any roads bordering the park?	47	0.115	55.3 %	46	0.261	63.0 %
What are the main land use(s) around the park?
Residential	47	–b	80.9 %	46	–b	69.6 %
Commercial	47	0.315	78.7 %	46	0.125	76.1 %
Institutional	47	0.588	87.2 %	46	0.246	78.3 %
Industrial	47	0.150	85.1 %	46	0.120	82.6 %
Natural	47	0.231	61.7 %	46	0.391	69.6 %
None present	47	–b	95.7 %	46	–b	91.3 %
Which of the following safety or appearance concerns are present in the neighborhood surrounding the park:
Poor lighting	47	0.157	59.6 %	46	0.386	71.7 %
Graffiti	47	–b	87.2 %	46	–b	84.8 %
Vandalism	47	–b	87.2 %	46	–b	84.8 %
Excessive litter	47	0.084	72.3 %	46	0.072	80.4 %
Heavy traffic	47	0.256	78.7 %	46	0.617	91.3 %
Excessive noise	47	0.301	80.9 %	46	0.281	82.6 %
Vacant or unfavorable buildings	47	0.084	72.3 %	46	0.378	84.8 %
Poorly maintained properties	47	0.138	66.0 %	46	0.275	76.1 %
Lack of eyes on the street	47	0.081	70.2 %	46	0.046	78.3 %
Evidence of threatening persons or behaviors	47	–b	91.5 %	46	–b	97.8 %
None present	47	0.138	66.0 %	46	0.185	60.9 %
Park activity areas
Playground no.	47	0.735	87.2 %	46	0.721	87.0 %
Useable?	30	–b	100.0 %	30	–b	100.0 %
Good condition?	30	0.375	80.0 %	30	0.172	73.3 %
Distinct areas for different age groups?	30	0.315	70.0 %	30	0.068	66.7 %
Colorful equipment?	30	0.444	83.3 %	30	0.375	80.0 %
Shade cover for some (25 %+) of the area?	30	0.348	66.7 %	30	0.267	63.3 %
Benches in/surrounding area	30	0.255	76.7 %	30	0.259	86.7 %
Fence around area?	30	0.645	83.3 %	30	0.648	83.3 %
Separation or distance from road?	30	0.118	70.0 %	30	0.167	73.3 %
Sports field no.	47	0.615	85.1 %	46	0.426	76.1 %
Useable?	4	–b	100.0 %	3	–b	100.0 %
Good condition?	4	1.000	100.0 %	3	0.000	66.7 %
Baseball field no.	47	0.890	93.6 %	46	0.765	89.1 %
Useable?	13	–b	100.0 %	5	–b	80.0 %
Good condition?	13	0.114	53.8 %	5	0.545	80.0 %
Swimming pool no.	47	–b	100.0 %	46	–b	89.1 %
Useablea	0	–a	–a	0	–a	–a
Good condition?a	0	–a	–a	0	–a	–a
Splash pad no.	47	–b	95.7 %	46	–b	93.5 %
Useable?a	0	–a	–a	0	–a	–a
Good condition?a	0	–a	–a	0	–a	–a
Basketball court no.	47	0.702	83.0 %	46	0.720	84.8 %
Useable?	19	–b	94.7 %	18	–b	94.4 %
Good condition?	19	−0.067	47.4 %	18	0.649	83.3 %
Tennis court no.	47	0.629	89.4 %	46	0.776	93.5 %
Useable?a	0	–a	–a	3	–b	100.0 %
Good condition?a	0	–a	–a	3	–b	100.0 %
Volleyball court no.	47	0.791	97.9 %	46	0.657	97.8 %
Useable?a	2	–a	100.0 %	1	–a	100.0 %
Good condition?a	2	–a	50.0 %	1	–a	100.0 %
Trail no.	47	0.329	61.7 %	46	0.605	78.3 %
Useable?	13	–b	100.0 %	11	–b	90.9 %
Good condition?	13	0.755	92.3 %	11	−0.100	81.8 %
Connected to activity areas	10	–b	100.0 %	10	–b	70.0 %
Distance markers/sign	12	0.333	66.7 %	10	−0.015	50.0 %
Benches along trail	12	0.167	58.3 %	10	−0.200	40.0 %
What is the trail surface?	12	0.750	91.7 %	10	0.000	80.0 %
Fitness equipment/station no.	47	–b	97.9 %	46	–b	97.8 %
Useable?a	1	–a	–a	0	–a	–a
Good condition?a	1	–a	–a	0	–a	–a
Skate park no.	47	–b	97.9 %	46	–b	97.8 %
Useable?a	0	–a	–a	0	–a	–a
Good condition?a	0	–a	–a	0	–a	–a
Off-leash Dog Park no.	47	–b	93.6 %	46	–b	93.5 %
Useable?a	0	–a	–a	1	–a	–a
Good condition?a	0	–a	–a	1	–a	–a
Open/GreenSpace no.	47	0.280	48.9 %	46	0.345	54.3 %
Useable?	12	–b	91.7 %	13	–b	92.3 %
Good condition?	12	0.000	58.3 %	13	0.156	61.5 %
Lake no.	47	–b	95.7 %	46	–b	91.3 %
Useable?a	1	–a	–a	0	–a	–a
Good condition?a	1	–a	–a	0	–a	–a
Is there a designated swimming area?a	1	–a	–a	0	–a	–a
Park quality and safety
Are restrooms available?	47	0.786	89.4 %	46	0.778	89.1 %
Useable?	19	0.420	78.9 %	17	0.452	76.5 %
Good condition?	19	0.208	52.6 %	17	0.457	64.7 %
Is there a family restroom?	19	–b	89.5 %	17	–b	82.4 %
Is there a baby change station in any restroom?	19	0.756	89.5 %	17	0.443	76.5 %
Are there drinking fountain(s) at the park?	47	0.692	85.1 %	46	0.671	84.8 %
Drinking fountain no.	15	0.439	73.3 %	13	0.226	61.5 %
Useable?	15	0.348	60.0 %	13	0.500	69.2 %
Good condition?	15	–b	46.7 %	13	–b	69.2 %
Near activity areas?	15	0.082	60.0 %	13	−0.083	61.5 %
Are there bench(es) to sit on in the park?	47	–b	89.4 %	46	–b	87.0 %
Useable?	40	–b	90.0 %	38	–b	78.9 %
Good condition?	40	0.301	67.5 %	38	0.320	65.8 %
Are there picnic table(s) in the park?	47	0.897	97.9 %	46	0.668	91.3 %
Useable?	41	–b	90.2 %	37	–b	86.5 %
Good condition?	41	−0.063	53.7 %	37	0.065	56.8 %
Is there a picnic shelter in the park?	47	0.811	91.5 %	46	0.809	91.3 %
Is there a grill or fire pit in the park?	47	0.744	87.2 %	46	0.696	84.8 %
Are there trash cans in the park?	47	–b	95.7 %	46	–b	93.5 %
Are they overflowing with trash	45	–b	86.7 %	42	–b	83.3 %
Are they near activity areas?	45	−0.158	64.4 %	42	0.156	71.4 %
Are recycling containers provided?	47	0.632	93.6 %	46	0.691	93.5 %
Is there food/vending machines available in the park?	47	0.221	87.2 %	46	0.657	97.8 %
Are fruits and/or vegetables available in the park?a	1	–a	–a	1	–a	–a
If the sun was directly overhead, how much of the park would be shaded?	47	0.413	66.0 %	46	0.531	73.9 %
Are there rules posted about animals in the park?	47	0.595	80.9 %	46	0.660	84.8 %
Is there a place to get dog waste pick up bags in the park?	47	0.636	85.1 %	46	0.710	89.1 %
Are bags available at any of the locations?	10	–b	80.0 %	9	–b	66.7 %
Are there lights in the park?	47	0.422	72.3 %	46	0.419	71.7 %
How much of the park could be lit?	23	0.324	65.2 %	21	0.008	42.9 %
Are the activity areas lit?	23	0.224	52.5 %	21	0.087	42.9 %
Is the park monitored?	47	0.067	72.3 %	46	0.434	87.0 %
Are there any emergency devices in the park?	47	–b	97.9 %	46	–b	97.8 %
From the center of the park, how visible is the surrounding neighborhood?	47	0.243	55.3 %	46	0.461	67.4 %
Are there road(s) of any type through the park?	47	0.269	74.5 %	46	−0.095	58.7 %
Are there traffic control mechanisms on the roads within the park?	4	0.000	50.0 %	2	1.000	100.0 %
Which of the following park quality or safety concerns are present in the park?
Graffiti	47	0.121	72.3 %	46	0.330	87.0 %
Vandalism	47	–b	85.1 %	46	–b	82.6 %
Excessive litter	47	−0.099	66.0 %	46	0.289	80.4 %
Excessive animal waste	47	–b	100.0 %	46	–b	100.0 %
Excessive noise	47	–b	78.7 %	46	–b	80.4 %
Poor maintenance	47	0.714	89.4 %	46	0.354	71.7 %
Evidence of threatening persons or behaviors	47	–b	91.3 %	45	–b	95.6 %
Dangerous spots in the park	47	0.253	78.3 %	45	0.167	77.8 %
Other	47	0.357	68.1 %	46	0.387	69.6 %
What aesthetic features are present in the park?
Evidence of landscaping	47	0.465	72.3 %	46	0.361	69.6 %
Artistic feature	47	0.321	78.7 %	46	0.284	80.4 %
Historical or educational feature	47	0.410	80.9 %	46	0.125	76.1 %
Wooded area	47	−0.062	48.9 %	46	−0.062	47.8 %
Trees throughout the park	47	0.299	66.0 %	46	0.103	56.5 %
Water feature	47	0.670	89.4 %	46	0.548	87.0 %
Meadow	47	–b	78.7 %	46	–b	78.3 %
None present	47	0.128	76.6 %	46	0.243	78.3 %

Kappa: 0.81–1.00 almost perfect; 0.61–0.80 substantial; 0.41–0.60 moderate; 0.21–0.40 fair; 0.00–0.20 poor

^aInsufficient pairs of ratings for accurate validity or reliability to be determined

^bInsufficient item variability across parks to use kappa

With respect to inter-rater reliability between youth auditors, again over 70 % of items demonstrated fair or moderate to perfect reliability. Specifically, kappa analysis demonstrated moderate to perfect degree of reliability for 41.1 % of the items, a fair degree of reliability for 30.0 % of items, and poor reliability for 28.9 % of items. In the remaining 41 tool items explored only by percent agreement between the youth auditors, all but four items demonstrated moderate to excellent reliability exceeding 70 % agreement, with most items well above 80 %.

DISCUSSION

Modifying park and neighborhood environments is a promising strategy for improving community health [6, 7]. A growing body of literature reveals that park characteristics are important predictors of youth park-based PA [11, 65]. Creation of a user-friendly electronic park audit tool can provide a more interactive and hands-on way for youth to engage with their local communities and facilitate participation in park-related health promotion and advocacy efforts [22, 23, 25]. However, understanding the ability of youth to obtain valid and reliable information via technology is an important first step in this process. This study described the development and validity and reliability testing of the eCPAT application for use by youth.

A large portion of the eCPAT items demonstrated fair or moderate to perfect validity and reliability demonstrated by Cohen’s kappa. As well, almost all of the items assessed using percent agreement demonstrated excellent validity and reliability. These findings are similar to those of the original CPAT tool indicating strong inter-rater reliability when tested among a diverse group of community stakeholders [24]. The most consistently valid and reliable items assessed the presence/absence of common activity areas (e.g., playgrounds, baseball fields) and supporting park amenities (e.g., restrooms, drinking fountains). This finding is not surprising as previous research has found that environmental audits have greater accuracy and consistency for items related to the presence or number of park characteristics due to a reduced amount of subjective influence on such ratings [24, 63].

Less than one third of eCPAT items demonstrated poor validity or reliability with youth. Of the 25 items that demonstrated poor validity, 40.0 % were subjective, 28.0 % were abstract or had a potentially difficult definition, 20.0 % were hard to find or rare items, 8.0 % were temporal, and 4.0 % were undefined. Of the 26 items that had poor reliability, 38.5 % were subjective, 30.8 % were abstract or had a potentially difficult definition, 19.2 % were hard to find or rare items, 7.7 % were temporal, and 3.8 % were undefined. A total of 11 items had both poor validity and reliability, the majority of which were subjective.

Our findings are similar to previous environmental audit tool validation studies, in which items that had lower kappa or percent agreement scores tended to be more subjective, difficult to find, temporal, or abstract in nature [39–41]. Subjective items often required youth to make decisions about the adequacy of distances (i.e., Does the playground have separation from the road? Are there drinking fountains near activity areas?). More detailed explanations of ambiguous spatial terms or use of specified distances could improve the precision and accuracy of youth answers. Likewise, some items may have been harder to find or were considered rare in our dataset (i.e., contact or program/event information on park signs, trail distance markers) or were problematic, especially in larger parks. Reiteration of youth following basic audit tips (found on the eCPAT home screen) such as driving, walking, or biking around the entire park may aid discovery of such items. Additionally, items that consisted of abstract concepts (i.e., Are there lack of eyes on the street?) were more difficult for youth to answer accurately. While these items typically included additional cues (e.g., absence of people, no houses or store fronts), the use of a subquestion within the item or instructions (i.e., If you needed help, would someone see/notice you?) might help youth to better understand the concept being assessed. As well, for several items, youth demonstrated a lack of consistency in determining whether something was in “good condition”. This result may have been due to the overall lack of variability in condition among study parks combined with the dichotomous nature of the answer option (yes/no) that potentially encouraged a skewed interpretation of what comprised good condition (i.e., because most of the study park elements were in good condition, youth may have noted very minor differences as being in not good condition). On the other hand, this result may have been indicative of an insufficient operational definition of good condition incorporated into the tool. Future versions of the eCPAT could include an improved system for understanding such concepts, such as standardized relational examples (e.g., guidelines as to what constitutes good/not good conditions across any park) or a discussion of how to interpret condition variability within defined parameters (e.g., within a set of very good parks, x, y, or z should constitute not good conditions). Finally, beyond the aforementioned suggestions regarding improving the validity and reliability of the eCPAT app, enhanced integration of basic tips or reminders about how to correctly conduct observational audits (i.e., review all instructions and examples prior to conducting an audit, direct observation of each item required) within the app could generally improve youth’s ability to assess park characteristics.

Strengths of this study include the use of an innovative mobile technology data collection tool that incorporated answer validation and wireless data submission that ensured complete park audit data and reduction of data entry errors. Additional technology benefits included improved usability, functionality, and the integration of instructions, definitions, and example pictures. As well, data for this study were collected by a diverse group of youth aged 11–18 that were sampled within the context of a larger randomized study which improves generalizability of the tool’s use among other youth populations. Likewise, this study sampled a large number of parks in Greenville County, SC, that represented a diverse mix of park and neighborhood characteristics.

This study also had several limitations. For example, while we employed a variety of recruitment methods, youth study participants were largely white and physically active with a low BMI which may have biased our sample. Future research should seek to specifically target at risk youth for participation in such efforts. As well, the majority of study youth had access to technology devices and over half had visited a park within the last month. While all youth attended the same 1-h project meeting that included a brief tool training, youth characteristics such as technology competency or experience in parks could have influenced the validity or reliability of results. Further, although directions for how to appropriately answer all items were included in the instruction and example photo features of the application, data on whether or not youth accessed these features were not captured in this study. Future evaluation of the eCPAT app should include collection and analysis of touch screen metrics and log files to understand application features accessed to compare against validity and reliability results to ensure that adequate interpretation and operationalization is occurring. If warranted, future versions of the eCPAT tool could incorporate more pronounced reminders of instructional features to ensure their use by youth auditors. It should also be noted that while youth completed a total of three park audits (one practice and two tests), there is potential for reliability and validity to improve with use.

This study included cross-sectional data from only one county in South Carolina. Despite our large sample of parks, for certain items within the eCPAT app, there was insufficient variability across parks to adequately calculate a kappa statistic. Further, certain items (e.g., skate parks, splash pads) did not occur in enough parks (or at all) which prevented collection of an adequate number of pairs of ratings to conduct reliability or validity analyses on those items [63]. Kappa statistics are also limited in their ability to distinguish among various types and sources of agreement, and they are influenced by prevalence and bias making it difficult to compare results across studies or populations [66]. Further, it is possible that kappa statistics may be low even when there are high levels of percent agreement [67]. As such, several researchers note that reporting the proportion of agreement alongside kappa values augments the understanding of results and facilitates enhanced decision making regarding the quality of data [60, 61]. Finally, data for our study was collected during daylight hours in June, which may have influenced the variability of quality and condition items as well as the overall ease of accessing and auditing park environments. Future studies should consider seasonal effects on results.

IMPLICATIONS

The results of this study have several implications for practice, policy, and research. First, there is a growing need for valid and reliable mobile technology tools for use by youth within participatory action research [26, 68]. Our results demonstrated that using the eCPAT app, youth are able to independently reach similar conclusions regarding the availability, usability, and condition of park characteristics that were comparable to those of a trained researcher. Ensuring data quality within participatory action research frameworks is fundamental to understanding community needs and developing environmental action plans [69, 70]. This study serves as a critical step toward validating an environmental audit tool in a largely underrepresented youth population that contributes significantly to the literature in this important field. Further, as youth advocacy and participation in health promotion projects have been identified as the next wave of social change for health, the eCPAT tool provides a valuable resource that has the potential to civically engage youth and provide meaningful participation in healthy community change and obesity prevention efforts [20, 27, 71]. Second, given the proliferation of smartphones, tablets, and other electronic devices among both adolescents and adults across the USA [72], as well as within our study sample, the eCPAT app has potential to be distributed and used widely by youth as well as the general public on their personal mobile devices. For example, the eCPAT app could be utilized to crowdsource environmental park data that could be uploaded in real time to a database interface for others to access and benefit from. Similarly, future practice or research efforts could incorporate eCPAT app data collection into initiatives to link parks with the healthcare system, such as Park Prescription efforts, to improve community awareness of park features and attributes in an effort to increase park-based PA and improve health [73, 74]. Finally, adaptation of the eCPAT app for use by local planning officials could allow agencies to collect and make data-driven decisions based on specific community needs, as well as assist with standardization of aggregated nationwide parks and recreation resource data (a priority identified by diverse agencies across the USA) [75, 76].

CONCLUSION

This study was a part of a broader research project to engage youth in becoming advocates for healthy community design and represents an important next step in ongoing research about the role of technology in youth empowerment for and engagement in health promotion efforts. The eCPAT app is a youth-oriented mobile technology application that provides a comprehensive assessment of park environments. Findings from this study will contribute to the final version of the eCPAT app which will be released in the near future for use by both youth and a variety of other stakeholders (e.g., community members, park and recreation professionals, healthcare providers). Future dissemination of this research will integrate the eCPAT app into youth-led, community-based participatory research projects to advocate for and implement positive park changes in an effort to improve overall community health.

Compliance with ethical standards

This study occurred in collaboration with Greenville County Parks, Recreation, and Tourism; the City of Greenville Parks and Recreation Department; and LiveWell Greenville and was approved by the Institutional Review Board (IRB) at the University of South Carolina.