Validity and Reliability of a School Travel Survey

Abstract

Background

Despite the growing interest in active (ie, nonmotorized) travel to and from school, few studies have explored the measurement properties to assess active travel. We evaluated the criterion validity and test–retest reliability of a questionnaire with a sample of young schoolchildren to assess travel to and from school, including mode, travel companion, and destination after school.

Methods

To assess test–retest reliability, 54 children age 8 to 11 years completed a travel survey on 2 consecutive school days. To assess criterion validity, 28 children age 8 to 10 years and their parents completed a travel survey on 5 consecutive weekdays.

Results

Test–retest reliability of all questions indicated substantial agreement. The questions on mode of transport, where you will go after school, and how you will get there also displayed substantial agreement between parental and child reports.

Conclusions

For this population, a questionnaire completed by school-age children to assess travel to and from school, including mode, travel companion, and destination after school, was reliably collected and indicated validity for most items when compared with parental reports.

Nonmotorized travel to and from school can be a regular source of physical activity for youth.^1,2 This source becomes especially important as time spent on physical activity in and out of school declines.^3,4 Moreover, if physical activity habits track from youth into adulthood, encouragement of physical activity as a mode of transportation during youth would be important. There is a growing interest in the study of active travel (eg, walking or bicycling for transportation purposes) to and from school for surveillance purposes and to understand the correlates, determinants, and interventions that might help increase this source of physical activity. Importance of active travel is reflected in 2 objectives of the US Healthy People 2010 (objectives 22.14 and 22.15) to increase the proportion of trips made by walking and bicycling.⁵

Despite this interest, there are few published works on the development of measures to assess active travel. For example, many surveillance,^6,7 cross-sectional,^1,2,8–27 prospective,^28,29 and intervention or evaluation studies^30–32 rely on self-reported measures of active travel to and from school from youth, reported alone or with parental assistance. These studies include children as young as 5 years old, and most do not report psychometrics of their measures. Thus, the purpose of this study was to evaluate the criterion validity and test–retest reliability of a questionnaire to assess travel to and from school, including mode, travel companion, and destination after school, with a sample of young children. We accomplished this by surveying fourth- and fifth-grade children regarding their travel to and from school and, for the validity sample, asking the same questions of their parents during the same week.

Methods

Participants and Procedures

The evaluation of travel to and from school was conducted as part of the Nonmotorized Travel Study, a pilot intervention to promote active travel to and from school. The study took place in North Carolina, a state with a low prevalence of walking and bicycling to school.⁶ Participants were fourth- or fifth-grade girls and boys and their parents living in the central, or Piedmont, region of North Carolina and attending 1 of 2 elementary schools located in the same school district. Survey administration occurred in April 2004 to explore test–retest reliability and in October 2004 to explore criterion validity. Parents of the children provided written consent to participate, children provided written assent, and this study was approved by the University of North Carolina Institutional Review Board. In addition, separate research applications were approved by the participating school district.

Survey

The daily survey on travel to and from school was developed, pilot tested, and administered by research staff in the classroom, with the questions read aloud by the research staff and completed by the children on the form. The 7-item questionnaire assessed the mode of travel to and from school and also to and from an after-school destination other than home if applicable, with whom travel occurred, and destination after school (see the Appendix for the survey).

Test–Retest Reliability

To examine test–retest reliability, children from 3 classrooms (2 fourth-grade and 1 fifth-grade distributed between the 2 schools) completed the daily travel survey for 2 consecutive days in April 2004. On the first day, students completed the survey in class. Readministration of the identical instrument occurred the following day, allowing for examination of the reliability of the measures. Children were asked to recall the previous day when they completed the second survey. Among 59 students from the 3 classrooms in the 2 schools, 54 completed both survey administrations.

Criterion Validity

In October 2004, children in 5 classrooms at 1 of the same schools were asked to complete the identical daily travel survey for 5 consecutive days (Monday through Friday of 1 school week). To assess criterion validity, the parental report was compared with the child’s report for each of the 5 days. A letter was sent home with previously recruited students requesting parental participation. Parents provided a phone number and time they preferred to be called each day of the week. Parents were required to be in town during the week of data collection. If staff missed contacting a parent 1 day but were able to reach the parent the following day, the staff collected travel information for both days (ie, current day and previous missed day). Parents were called daily, usually in the afternoon or evening. Among the 78 participating students, 73 attended all 5 days of school and completed the surveys each day in the classroom. Among the parents, 29 returned consent forms to participate in the telephone survey, and of those, 26 completed the 5 daily telephone surveys and 2 parents completed the telephone survey on 4 out of the 5 days. The total number of walking trips to and from school in the measurement week was calculated by adding the number of walking trips provided for each day of the week, with a potential range of 0 to 10 trips.

Statistical Analysis

All analyses were conducted using SAS version 9.1 (Cary, NC). To examine test–retest reliability (child–child comparison) and criterion validity (child–parent comparison), percent agreement (calculated as the number of response pairs with exact agreement divided by the total number of response pairs), as well as unweighted kappa coefficients for categorical variables, were calculated. For the continuous measure of the number of walking trips per week, intraclass correlations coefficients (ICC) were calculated. For criterion-validity assessment, because measurement occurred for 5 consecutive days, the percent agreement was calculated as the average over each of the 5 days. Average kappa coefficients³³ were calculated over the 5 days using the frequency procedure (overall kap) in SAS, adjusted for day. This procedure treated the 5 interrater agreements as independent observations. As a rough guide, we followed the ratings suggested by Landis and Koch³⁴ for agreement: poor (0–0.2), fair (0.2–0.4), moderate (0.4–0.6), substantial (0.6–0.8), and almost perfect (0.8–1.0).

Results

Descriptive characteristics of the 2 samples of children are shown in Table 1. For the study samples, children ranged in age from 8 to 11 years, with a median of 10 years for the reliability sample and 9 years for the validity sample. Approximately three-fourths were non-Hispanic white, and two-thirds reported always having an adult at their home on returning from school in the afternoons.

Table 1.

Description of Samples to Examine Reliability and Validity, Based on Child Self-Report

	Reliability samplea (N = 54)		Validity samplea (N = 28)
Survey item	n	%b	n	%b
Are you a …
boy	24	46	13	46
girl	28	54	15	54
missing	2		0
How old are you?
8 y	0	0	1	4
9 y	13	25	24	86
10 y	30	58	3	11
11 y	9	17	0
missing	2		0
Race/Ethnicity
non-Hispanic white	41	76	20	71
other	5	9	5	18
don’t know	8	15	3	11
Is there usually an adult at your home in the afternoon when you return from school?
never	1	2	1	4
sometimes	15	29	5	18
always	34	67	20	71
I don’t go home after school	1	2	2	7
missing	3		0

^aOnly 3 children participated in both the reliability and validity samples.

^bPercents may not add to 100% because of rounding.

For the reliability sample, 48% arrived at school by bus and 50% arrived by other vehicles, with only 1 child reporting walking to school according to the first of 2 surveys (Table 2). For leaving school, 4 children reported walking home. Test–retest reliability of the daily survey indicated almost perfect percent agreement for most items. Percent agreement ranged from 93% to 100%, and the kappa coefficients ranged from .79 to 1.00.

Table 2.

Responses and Test–Retest Reliability of Measures Among the Reliability Sample (N = 54)

	Time 1		Time 2
Survey item	n	%a	n	%a	Kappa coefficients (95% CI)	Percent agreement
(2)b How did you get to school today?					.96 (.90–1.00)	98.1
bus	26	48	24	45
car or truck	27	50	28	53
walked	1	2	1	2
bike, skateboard, scooter, Rollerblade, other	0	0	0	0
missing	3		0
(3) Whom did you travel with? (Mark all that apply.)c
no one, I was by myself	0	0	0	0
mother or father	26	48	27	50	.96 (.89–1.00)	98.1
brother or sister	12	22	10	19	.89 (.73–1.00)	96.3
other relative	1	2	0	0	—d	98.1
other student(s)	25	46	23	43	.93 (.82–1.00)	96.3
other	8	15	9	17	.79 (.56–1.00)	94.4
(4) Where will you go directly after school today?					.88 (.72–1.00)	96.2
home	44	82	42	79
after-school program at school	1	2	2	4
relative’s house	1	2	0	0
friend’s house	5	9	1	2
YMCA/YWCA or boys and girls club	2	4	4	8
sports practice or lessons	1	2	3	6
other	0	0	1	2
(5) How will you get there? (if previous answer was not home)					1.00	100.0
bus	1	9	0	0
car or truck	6	55	7	64
walk	4	36	4	36
bike, skateboard, scooter, Rollerblade, other	0	0	0	0
(6) Whom will you travel with? (Mark all that apply.)c
no one, I will be by myself	1	2	2	4	—	98.1
mother or father	11	20	13	24	.79 (.59–.99)	92.6
brother or sister	10	19	11	20	.94 (.83–1.00)	98.1
other relative	2	4	0	0	—	96.3
other student(s)	39	72	36	67	.87 (.73–1.00)	94.4
other	9	17	7	13	.85 (.66–1.00)	96.3
(7) When you go home, how will you get there?					.96 (.89–1.00)	98.1
bus	33	62	32	60
car or truck	16	30	17	32
walk	4	8	4	8
bike, skateboard, scooter, Rollerblade, other	0	0	0	0
missing	1		0
Number of walk or bike trips that day					1.00	100.0
none	49	91	49	91
1 or 2	5	9	5	9

^aPercents may not add to 100% because of rounding.

^bThe survey item numbers correspond to the survey questions in the Appendix.

^cThirty-nine reported 1 answer, 12 reported 2 answers, and 3 reported 3 answers.

^d— indicates cells too small to calculate kappa.

For the validity sample, none of the children walked or biked to school (Table 3). The question on mode of transport displayed substantial agreement between parental and child reports (kappa = .80). In some cases for both to and from school, the travel-companion questions displayed lower agreement according to the kappa coefficients but higher agreement on the average percent agreement. This seemingly inconsistent finding was caused by the low number of people reporting in some categories; thus, any misclassification made a large difference in the kappa coefficient. The questions on where you will go after school and how you will get there displayed substantial agreement between parental and child reports (kappa ≥ .6). When comparing the number of walking trips to and from school, as reported by the child and the parent, the agreement according to the ICC was 0.55 (95% CI, 0.24–0.76). There were no bicycling trips reported for the week by either the children or the parents.

Table 3.

Mean Responses of Children and Parents Over 5 Days and Comparison of Children’s Responses With Parents’ Responses (N = 28 Pairs)

	Child survey		Parent survey		Agreement between child and parental reports
Survey item	Mean n (rangea)	Mean %b	Mean n (rangea)	Mean %b	Mean kappa coefficient (95% CI)	Mean percent agreement
(2)c How did you get to school today?					.80 (.71–.89)	88.4
bus	10 (8–12)	36.2	10 (9–12)	37.7
car or truck	15 (13–17)	55.8	17 (15–19)	61.6
walked	0 (0–0)	0.0	0 (0–0)	0.0
biked	0 (0–0)	0.0	0 (0–0)	0.0
skateboard, scooter, or   Rollerblade	0 (0–0)	0.0	0 (0–0)	0.0
other	2 (2–3)	8.0	0 (0–1)	0.7
(3) Whom did you travel with? (Mark all that apply.)
no one, I was by myself	0 (0–0)	0.0	0 (0–0)	0.0	1.00	100.0
mother or father	16 (14–17)	58.7	16 (13–19)	59.3	.82 (.73–.92)	89.0
brother or sister	14 (12–15)	49.2	7 (5–11)	25.3	.46 (.33–.58)	73.2
other relative	0 (0–0)	0.0	0 (0–0)	0.0	1.00	100.0
other student(s)	11 (10–13)	41.3	9 (8–11)	34.1	.75 (.63–.86)	87.0
other	2 (2–3)	8.0	1 (1–1)	3.7	—d	91.3
(4) Where will you go directly after school today?					.64 (.34–.98)	86.7
home	20 (19–22)	73.7	21 (20–23)	79.0
after-school program at school	2 (0–3)	4.5	2 (0–1)	2.2
relative’s house	0 (0–1)	0.7	0 (0–1)	0.0
friend’s house	0 (0–1)	0.7	1 (1–1)	0.7
YMCA/YWCA or boys and   girls club	5 (4–5)	17.5	3 (3–4)	11.8
sports practice or lessons	0 (0–1)	0.7	0 (0–1)	0.7
other	1 (0–2)	2.2	2 (0–3)	5.9
(5) How will you get there? (if previous answer was not home)					.61 (.28–.94)	87.0
bus	3 (2–4)	28.4	1 (1–2)	22.4
car or truck	2 (1–2)	17.5	2 (1–3)	35.2
walk	5 (4–5)	54.1	2 (2–3)	42.4
bike	0	0.0	0	0.0
skateboard, scooter, or   Rollerblade	0	0.0	0	0.0
other	0	0.0	0	0.0
(6) Whom will you travel with? (Mark all that apply.)
no one, I will be by myself	2 (1–2)	6.5	1 (1–2)	4.8	—	90.6
mother or father	6 (4–8)	21.7	8 (5 –11)	28.2	.57 (.42–.72)	83.3
brother or sister	11 (10–12)	39.8	4 (2–7)	16.0	.13 (0–.26)	63.0
other relative	0 (0–1)	0.0	0 (0–0)	0.0	1.00	100.0
other student(s)	20 (19–21)	73.2	17 (14–19)	60.1	.64 (.51–.77)	81.9
other	4 (3–4)	13.0	1 (1–1)	3.6	—	86.2
(7) When you go home, how will you get there?					.81 (.70–.90)	88.9
bus	16 (15–18)	59.6	16 (15–18)	59.3
car or truck	10 (8–11)	36.0	11 (8–12)	39.3
walk	1 (1–2)	4.4	1 (1–1)	3.6
bike	0	0.0	0	0.0
skateboard, scooter, or   Rollerblade	0	0.0	0	0.0
other	0	0.0	0	0.0
missing	1		0

^aRanges are based on the 5-day period.

^bPercents do not add up to 100% because they are averaged over the 5 days.

^cThe survey item numbers correspond to the survey questions in the Appendix.

^d— indicates cells too small to calculate kappa.

Discussion

This study indicates that test–retest reliability and criterion validity of the items on travel to and from school were acceptable for this sample. We conjecture that the measurement properties might even be improved for an older group of children. The question regarding with whom the child traveled either to school in the morning or from school to home in the afternoon had the lowest child–parent agreement. It is possible that children who reported during the school day how they planned to go home later that day subsequently changed their mode, destination, or traveling companion, thus possibly accounting for some of the disagreement with the parental reports. It should be noted that during the administration of the questionnaire, some children wanted to record how they usually traveled to and from school, not how they actually traveled on those specific survey days.

Only a few studies have reported reliability for the child self-reported measurement of active travel to and from school. First, in a study of 79 twelve-year-olds, 1-month test–retest reliability of a measure of active commuting to and from school had an ICC of 0.79.³⁵ A second study of approximately 120 youth age 13 to 14 years conducted in England collected self-report data on the mode of travel to school (walking, car, bicycle, bus, train, other).²² Test–retest reliability across a 2-week time span was high (kappa .84–.87). Third, a study in the United States of 480 girls in the sixth and eighth grade asked, “How many days in the past week did you walk, bike or skate to school?” with response options as follows: none, 1 day, 2 or 3 days, 4 days, or every day.²⁴ Over a median of 12 days, test–retest reliability assessed with a weighted kappa coefficient was .60 (95% CI, 0.52–0.67), and overall agreement was 74%. Fourth, in a Belgian study of 33 youth aged 12 to 18 years, 1-week test–retest reliability for a measure of active transport to and from school had 69% agreement and a kappa coefficient of .53.³⁶ The ICC comparing the hours per day of active transport to and from school was 0.84. Generally these 4 studies reported moderate to substantial agreement on test–retest reliability, similar to what we found in this study.

Apart from this study, we are aware of 2 other studies that reported on validity, as well as reliability, of the self-reported child travel measure. First, in a study of 600 children age 9 to 11 years living in rural Nebraska, the children were queried about travel to and from school using a 1-week recall checklist.²⁸ Test–retest reliability was determined by asking the children on Wednesday what mode of transport they used to get to and from school Monday through Wednesday of that week and then comparing with results obtained 2 days later from retesting for that same time recall period. There was 97.0% concordance on the mode of travel between the results obtained from the 2 identical surveys conducted 2 days apart covering the travel for the 3-day period. Validity was evaluated by randomly contacting a 5% sample of parents by phone after the data-collection period on Friday. They found 97.5% concordance between the children’s self-reported method of transport to and from school and the parents’ responses. Second, an Australian study of 5- to 6- and 10- to 12-year-old youth assessed the test–retest reliability of duration and frequency of walking to and from school for each day of the week and compared this with parental self-report.³⁷ The reliability, measured 1 week apart from the child’s report, as indicated by the ICC, ranged from 0.86 to 0.94 for frequency of walking to school and 0.69 to 0.90 for duration of walking to school. Among the 10- to 12-year-old children, there was 87.5% agreement compared with their parents on the measure of walking to school.

From this and previous studies, it appears that assessment of travel to and from school collected from children is useful. What is not known is the number of days and times per year of data collection needed to ascertain travel patterns. More work is needed to clarify the best measures to use for surveillance purposes. Although we found that the frequency of walking to school differed from the frequency of walking home from school, an additional question to ascertain these differences might not be needed for surveillance purposes.

Other methods used in other studies to assess active travel to and from school, beyond child self-report, include parental report,^35,38–50 a combination of parental and child report,^{8–11,13,15,51} asking the children for a show of hands in the classroom regarding travel,³¹ diaries,^32,52–54 indirect assessment through accelerometry,^1,2,20,22,44 counting the number of bicycles in racks,⁵⁵ and direct observation of student travel to and from school.^56,57 Benefits of the observation system include the elimination of selection bias, recall errors, and low response rates, but observation is only able to address mode and not questions regarding with whom school children were traveling and their after-school destination. In our experience, challenges of observation also include the cost of multiple staff needed to accurately observe and not double-count students. Also, in our experience it was not always clear whether an observed child was an older student attending the elementary school or a sibling picking up or dropping off his or her brother or sister. Observation also requires accurate counts of the total census of the school and the number of absences in a given day to determine a denominator for percent of children walking or bicycling. It also requires long hours of the observer because of before-school breakfast and after-school programming. It would be useful to triangulate the observed data with those collected from students and parents.

Although this study contributes to the measurement development of transport to and from school, several limitations should be discussed. Both the student and parental measures relied on self-report, so errors from these methods might be correlated. For the assessment of validity, the kappa coefficients were averaged over 5 days, ignoring the potential correlation between pairs. This study did not evaluate perceived intensity or duration of active travel, which are other important components of physical activity. These measures should be evaluated in other, more diverse populations with respect to such characteristics as more diverse travel modes, age, geography, and neighborhood socioeconomics to verify the generalizability of the findings. Finally, it should be noted that this survey did not ascertain mode of transport if the child went home directly after school.

Although not part of a primary goal of our study, the responses from this small survey population show that one cannot assume that walking or bicycling to school will provide the same frequencies as walking or bicycling home from school. This study also indicates that a questionnaire to assess travel to and from school of fourth-and fifth-grade schoolchildren, including mode, travel companion, and destination after school, can be reliably self-reported by the schoolchildren themselves, and results indicated agreement between child and parental reports.

Appendix