Abstract
Background
Few studies have reported the accuracy of measures used to assess sun-protection practices. Valid measures are critical to the internal validity and use of skin cancer control research.
Objectives
We sought to validate self-reported covering-up practices of pool-goers.
Methods
A total of 162 lifeguards and 201 parent/child pairs from 16 pools in 4 metropolitan regions in the United States completed a survey and a 4-day sun-habits diary. Observations of sun-protective behaviors were conducted on two occasions.
Results
Agreement between observations and diaries ranged from slight to substantial, with most values in the fair to moderate range. Highest agreement was observed for parent hat use (κ = 0.58–0.70). There was no systematic pattern of over- or under-reporting among the 3 study groups.
Limitations
Potential reactivity and a relatively affluent sample are limitations.
Conclusion
There was little over-reporting and no systematic bias, which increases confidence in reliance on verbal reports of these behaviors in surveys and intervention research.
Keywords: concurrent validity, measurement, observation, self-report assessment, sun protection
The incidence and mortality from skin cancer, specifically cutaneous melanoma, has increased rapidly in the past few decades.1,2 Behavioral recommendations for the prevention of skin cancer aim to reduce exposure to ultraviolet radiation by limiting time spent in the sun, seeking shade particularly during periods of peak ultraviolet radiation, using sunscreen with a sun-protection factor of 15 or higher, wearing protective clothing (hat, shirt, pants) and sunglasses, and making sun safety a regular habit.3
Valid measures of behavioral outcomes are critical to the internal validity and use of skin cancer control research. Measurement of sun-protection behaviors most often involves self-report of habitual sun-protection practices.4,5 Relatively few studies have reported how accurately their survey items can assess sun-protection practices, which leaves some ambiguities regarding the veracity of reported findings.
Direct observation is a recognized procedure to assess the validity of self-report sun-protection habits.5 Studies that have reported the validity of their measures have been undertaken among schoolchildren6 and adult populations including outdoor workers7,8 and beachgoers.9 Findings from these studies have revealed substantial agreement between the two measures among adult populations for all measures: head wear (κ = 0.60–0.77), upper body (κ = 0.64–0.71), lower body (κ = 0.63–0.83), sunglasses (κ = 0.60–0.76), and footwear (κ = 0.63). Fair agreement was obtained among children for upper (κ = 0.34) and lower (κ = 0.35) body coverage.6 Fair agreement for use of footwear (κ = 0.23) and slight agreement for sunglasses (κ = 0.11) was obtained among beachgoers.9
Validation of self-report is an important step toward understanding the strengths and limitations of measures of sun-protection behavior. Using a multimethod research design, we conducted a large measurement validation study among lifeguards, parents, and children at swimming pools to quantify the association between self-reported sun-protection habits and available objective measures. The aim of this article was to evaluate the concurrent validity of self-reported covering-up practices of lifeguards, parents, and children compared with observations. An ancillary aim was to descriptively examine the extent of systematic error found in self-reports of using hats, shirts, and sunglasses in various subgroups in the study.
METHODS
Setting and recruitment
The study was conducted in the summer of 2006. Sixteen pools in 4 metropolitan regions were selected from a larger sample of 245 pools that were participating in the parent study, Diffusion of an Effective Skin Cancer Prevention Program (Pool Cool).10,11 Regions were stratified according to latitude (north or south) and study arm (basic or enhanced). Pools were required to have a minimum of 15 lifeguards on staff and the ability to recruit at least 15 parents and their children, 5 to 10 years of age, who were taking swim lessons at that pool.
Lifeguards and parent/child pairs were enrolled in the study at each pool on the Wednesday before the start of data collection on either Thursday or Friday of that week. Lifeguards were approached as they arrived at the pool or when they were not on duty. Parents were approached when they brought their children to the pool for swim lessons. Study procedures were explained to lifeguards, parents, and children at the time of recruitment, and those who agreed to participate in the study were asked to sign consent forms and complete a baseline survey. Verbal assent was obtained from the children. After completing the enrollment process, each person was given an instruction sheet, a timeline and description of all the activities to be conducted throughout the study, and a small sling bag to thank them for participating.
Procedure
Each participant was asked to come to the pool for data collection on one weekday (Thursday or Friday) and one weekend day (Saturday or Sunday). Participants were asked to arrive at the pool for the first day of data collection at 9:00 AM, or as early as possible thereafter, and completed a sun-habits survey shortly after arrival. Research staff observed and recorded participants' visible sun-protection practices on a pre-coded form on each of the 2 days of data collection. Subjects were also asked to fill out diaries of their activities for each of the 4 days they were involved. They were contacted in the afternoon (via telephone calls, e-mail, or text messages) as reminders to complete the diary each day. At the conclusion of data collection, each lifeguard and parent was given a $25 gift card to thank them for participating. This approach was pilot tested in Georgia and Hawaii before implementation.12
Measures
Self-report measures
Sun-habits surveys were completed at the time of enrollment into the study. There were two separate surveys: one for parents and children and one for lifeguards/aquatic instructors. Parents answered for both themselves and their child. Surveys included questions on sun-protection habits, skin cancer risk factors, sunburn history, ultraviolet exposure, and background demographics. Measures were selected or adapted from previously published studies and tools used in earlier studies conducted by the project team.10,13,14 Demographic information collected included sex, age, race/ethnicity, education, income level, marital status, number of children, and job title (for lifeguards). Sun-protection habits were assessed by questions asking about the frequency of usually practicing 5 protective behaviors when outside on a sunny day (wearing a shirt with sleeves, wearing sunglasses, staying in the shade or under an umbrella, wearing sunscreen, and wearing a hat) on 4-point ordinal scales ranging from 1 (rarely or never) to 4 (always).
The sun-habits diary was used to record daily sun-exposure and -protective behavior, and was adapted from a diary developed for earlier skin cancer prevention research.15 All participants were instructed to complete the diary for 4 consecutive days (2 weekdays and 2 weekend days), which is considered sufficient to estimate weekly sun exposure and sun protection.15 Parents were instructed to fill out separate diaries for themselves and their children, although this was sometimes undertaken in consultation with their children. For each hour when they were outside between 10 AM and 4 PM, individuals were asked to indicate their sun-protection practices such as: wearing a hat, a shirt with sleeves, sunscreen, sunglasses, and staying in the shade or under and umbrella.
Observation measures
Direct observation was undertaken by research staff to record the behavior of each participant on data-collection days (one weekday and one weekend day). A pre-coded form, adapted from previous research,9 was used to record the observations. For clothing, researchers observed participants' use of head wear (nothing, cap/visor, brimmed hat/legionnaire); upper body clothing (nothing, bikini top, one-piece suit, tank top/tankini, short sleeves, long sleeves); and use of sunglasses (yes or no) at the time of data collection. Interrater reliability for these categories was excellent (sunglasses: κ = 0.86; head wear: κ = 0.87; and upper body: κ = 0.90).
Data preparation and statistical analysis
Recoding of observation data was necessary to improve consistency with categories used in the survey and diary. The 3 observed sun-protection practices were dichotomized. Hat use was recoded to classify the use of any type of hat (yes = cap/visor, brimmed hat/legionnaire; and no = nothing). Upper body clothing was recoded from 6 categories to identifying the use of a shirt with sleeves (yes = short or long sleeves; and no = nothing, bikini top, one-piece suit, tank top/tankini). The use of sunglasses (yes or no) remained unchanged.
Covering-up sun-protection practices reported in the diary were calculated as a percentage of time that each behavior was performed while outside, for example, each of the covering-up sun-protection practices (wearing a hat, covering up, and wearing sunglasses). To compare diary records with survey data, the proportion of time spent using various practices during the 4 days of data collection was calculated. Covering-up practices reported on or around the time that the two observations were conducted, which was undertaken on one weekday (Thursday or Friday) and one weekend day (Saturday or Sunday), were compared with direct visual observation for wearing a hat, use of shirt, and use of sunglasses.
To compare direct observation with survey responses, the observed use of each practice (wearing a hat, shirt with sleeves, and use of sunglasses) for each of the two occasions were assigned a score: 0 = did not observe and 1 = observed. By summing across the two observation periods, a score ranging from 0 to 2 was obtained (0 = was not observed using at all; 1 = was observed on at least one occasion; and 2 = was observed on both occasions).
Statistical analysis
The primary aim of this study was to evaluate the concurrent validity of self-reported covering-up practices of parents, lifeguards, and children. The ancillary aim was to examine the extent of systematic error found in self-reports of using hats, shirts, and sunglasses in various subgroups in the study. To address the primary aim, the observations were compared with the two self-report measures using two approaches. The first approach involved using cross-tabs procedures and kappa statistics to assess agreement between the observed shirt, hat, and sunglasses and the matching self-reported (weekday and weekend) diary measures among parents, lifeguards, and children. Kappa coefficients were categorized as: poor (κ < 0.0); slight (κ = 0.0–0.2); fair (κ = 0.2–0.4); moderate (κ = 0.4–0.6); substantial (κ = 0.6–0.8); and almost perfect (κ = 0.8–1.0).16 The second approach to assessing concurrent validity involved comparing each of the 3 data collection methods via correlations. Spearman coefficients were used to estimate correlations for the weekday and weekend mornings of observations and for the total week.
To address the ancillary aim, the cross-tabs procedure was also used to examine the distribution of accurate and over- or under-self-reporting of each clothing category relative to observations. Descriptive and agreement statistics were computed separately within lifeguards, parents, and children, for subgroups defined by sex, latitude, Pool Cool intervention arm, and skin cancer risk level.
RESULTS
Participant characteristics
A total of 993 eligible participants were approached across the 16 pools; 631 (64%) consented to participate in the study. Complete data were obtained from 89% (n = 564) of participants resulting in a completion rate of 90% (n = 201) for parent/child pairs and 87.6% (n = 162) for lifeguards. Rates of participation and completion were similar across regions.
Most of the parent participants were female (95%), the child's mother (91.0%), and reported being white (83.5%), well educated (65.5% had ≥ college graduate), and of moderate to high income (78.4% with >$50,000 household income/year). Children had a mean age of 7.7 years (SD = 1.7) and were slightly more often male (52.3%) than female. The lifeguards were 59.3% female, mostly white (89.9%), unmarried (98.1%), averaged 19.4 years old (SD = 5.6), and high-school students or graduates (51.3%) or college students (41%).
Sun-protection coverage by observation and self-report measures
Overall, a large proportion of participants were observed wearing a shirt with sleeves (60.3% weekday, 76.6% weekend). There were no significant differences between the weekday and weekend observations (χ2 (2) = 3.42, P = .18). Although hat use was observed for 12% of participants on the weekday and 8% on the weekend, it was not statistically significant (χ2 (2) = 2.76, P = .25). Lifeguards were significantly more likely (χ2 (2) = 15.85, P <.001) on the weekday to have been observed wearing a hat (20.6%) compared with parents (11.9%) or children (5.1%). However, there were no significant differences between participants for the weekend assessment (χ2 (2) = 3.34, P = .19). Sunglasses were observed on 28.4% of participants on the weekdays and 26.6% on the weekend. Children were significantly less likely to be observed wearing sunglasses (1.0% weekday; 2.0% weekend) when compared with parents (43.8% weekday; 37.2% weekend) and lifeguards (45.6% weekday; 46.8% weekend).
Sun-habits diaries completed during a 4-day period (two weekdays and two weekend days) showed that a large proportion of participants reported wearing a shirt with sleeves (37.9% weekday; 35.3% weekend). There were no significant between-group differences for wearing a shirt on weekdays (χ2 (2) = 0.33, P = .85) or weekend days (χ2 (2) = 2.52, P = .28), or between the two reporting times (χ2 (2) =.40, P = .52). Significantly more people (χ2 (2) = 40.6, P = .001) reported wearing sunglasses on weekdays (44.0%) compared with the weekend (25.9%). Children were significantly less likely on both days (χ2 (2) = 97.68, P < .001; χ2 (2) = 69.82, P < .001) to report wearing sunglasses (5.0% weekday; 2.0% weekend) when compared with parents (52.2% weekday; 29.9% weekend) and lifeguards (82.1% weekday; 50.6% weekend). Approximately 16% of participants on the weekday and 8% on the weekend reported wearing any type of hat. In addition, lifeguards (35.2% weekday; 16.7% weekend) were significantly more likely to report wearing a hat on both days (χ2 (2) = 47.03, P < .001; χ2 (2) = 7.69, P = .02) than parents (13.4% weekday; 3.5% weekend) or children (3.5% weekday; 6.5% weekend).
Agreement between measures
Percent agreement between the direct observations and reported clothing coverage at time of observation (by diary entry) ranged from 48% for children wearing shirts on the weekend to 96% for parents wearing hats on the weekend (Table I). Agreement between the two methods for the lifeguard group ranged from 53% to 83%. Overall, agreement was relatively good across the 3 participant categories with the kappa value ranging over the clothing types for children (0.12–0.45) followed by lifeguards (0.14–0.43) and then parents (0.21–0.70). Children wearing sunglasses was excluded from the analysis because of the small number of children who reported and were observed wearing sunglasses.
Table I.
Agreement between self-report (diary) and direct observation of sun-protective clothing
| Observation and diary discrepancy |
|||||
|---|---|---|---|---|---|
| Observation and diary agreement n (%) | Positive observation and negative diary entry n (%) | Negative observation and positive diary entry n (%) | κ (95% CI) | ||
| Hat use | |||||
| Weekday | Lifeguard | 122 (76.3) | 7 (4.4) | 31 (19.4) | 0.43 (0.27, 0.59) |
| Parent | 180 (93.2) | 6 (3.1) | 7 (3.6) | 0.70 (0.54, 0.85) | |
| Child | 184 (93.9) | 8 (4.1) | 4 (2.0) | 0.22 (0.01, 0.51) | |
| Weekend | Lifeguard | 130 (83.4) | 9 (5.8) | 17 (10.9) | 0.32 (0.11, 0.52) |
| Parent | 190 (95.9) | 7 (3.5) | 1 (.5) | 0.58 (0.32, 0.84) | |
| Child | 186 (93.5) | 6 (3.0) | 7 (3.5) | 0.45 (0.19, 0.70) | |
| Shirt with sleeves | |||||
| Weekday | Lifeguard | 96 (59.2) | 50 (30.9) | 16 (9.9) | 0.22 (0.05, 0.39) |
| Parent | 124 (62.9) | 68 (34.5) | 5 (2.5) | 0.33 (0.18, 0.48) | |
| Child | 112 (56.5) | 59 (29.8) | 27 (13.6) | 0.15 (0.01, 0.30) | |
| Weekend | Lifeguard | 83 (53.2) | 66 (42.3) | 7 (4.5) | 0.14 (0.01, 0.36) |
| Parent | 107 (53.7) | 88 (44.2) | 4 (2.0) | 0.21 (0.04, 0.39) | |
| Child | 96 (48.0) | 98 (49.0) | 6 (3.0) | 0.12 (0.01, 0.33) | |
| Sunglasses | |||||
| Weekday | Lifeguard | 96 (60.1) | 3 (1.9) | 61 (38.1) | 0.24 (0.05, 0.44) |
| Parent | 144 (74.3) | 16 (8.2) | 34 (17.5) | 0.49 (0.37, 0.61) | |
| Weekend | Lifeguard | 96 (62.4) | 26 (16.9) | 32 (20.8) | 0.25 (0.09, 0.40) |
| Parent | 143 (71.9) | 35 (17.6) | 21 (10.6) | 0.37 (0.24, 0.51) | |
CI, Confidence interval.
Correlations among the measures of sun-protection clothing use are reported in Table II. Overall agreement between measures ranged from slight to moderate for parents (0.15–0.60), children (0.10–0.52), and lifeguards (0.10–0.55). Although comparisons between the survey and weekly diary reports were similar, observation/survey comparisons generally produced the lowest coefficients (κ = 0.10–0.47).
Table II.
Level of agreement among observation, diary, and survey measures of clothing coverage
| Observation and survey |
||||
|---|---|---|---|---|
| Clothing coverage (n = 564) | Weekday r | Weekend r | Diary and survey r | |
| Hat use | Lifeguard | 0.37 | 0.28 | 0.55 |
| Parent | 0.35 | 0.24 | 0.42 | |
| Child | 0.31 | 0.10* | 0.27 | |
| Shirt with sleeves | Lifeguard | 0.10* | 0.21† | 0.46 |
| Parent | 0.15† | 0.29 | 0.49 | |
| Child | 0.15† | 0.15† | 0.52 | |
| Sunglasses | Lifeguard | 0.23 | 0.23 | 0.18† |
| Parent | 0.47 | 0.45 | 0.60 | |
All coefficients statistically significant at P < .01 except as noted.
Not significant.
P < .05.
Systematic errors by subgroup
The distribution of accurate and over- or under-reporting was examined for each participant category and for the stratification variables (sex, risk level, latitude, and study group) within each clothing category. Overall, among the parent, child, and lifeguard groups there were no apparent systematic patterns of under- and over-reporting. Differences in over-reporting ranged from 2.5% (n = 3) between parent and lifeguard weekend shirt use to 20.6% (n = 27) between lifeguard and parent weekday sunglasses use. For lifeguards, parents, and children separately, the proportion of those over- and under-reporting, was similar among the clothing types, with the exception of wearing shirts. As can be seen from Table I, all 3 groups under-reported shirt use and this was consistent across groups. There did not appear to be any significant patterns in level of agreement based on the stratification variables. Examination of reporting for parents, lifeguards, and children based on these variables all confirmed that differences in under- and over-reporting were minimal (1%–20%) and consistent with the overall reporting percentages (ie, over- and under-reporting was not dependent on sex, latitude, study arm, or skin cancer risk group).
DISCUSSION
The purpose of this study was to examine the validity of self-reported covering-up practices using direct observation and to report on the extent of systematic error, if any, found in reported measures of using hats, shirts with sleeves, and sunglasses. We found fair to moderate agreement between direct observation and diary records of protective coverage worn at the time the observations were undertaken. Overall, good levels of agreement were obtained between 4-day diary records and survey reports of usual sun-protective coverage practices. These findings were consistent across the sample regardless of sex, latitude, Pool Cool intervention arm, and skin cancer risk level. Given that the diary resulted in consistently higher correlations with the observation (0.16–0.70) than survey, the diary method appears to be somewhat more valid than the survey method. However, because of participant time and effort involved in diary completion, these validity results indicate that a survey may be adequate.
This study is distinct from previous validation studies undertaken in that it was a multisite assessment undertaken at 16 swimming pools among communities in 4 metropolitan areas in the United States. These pools were already participating in the Pool Cool skin cancer prevention diffusion trial.11,17
Agreement between diary and observations for sun-protective coverage has been previously reported for schoolchildren,6 outdoor workers,7 postal workers,8 and beachgoers.9 Although overall moderate to substantial levels of agreement between the two measures were reported, some variations have been observed that may have been a result of a number of issues: the population being assessed, the setting in which the assessment is being undertaken, and/or the measures being used to make the assessment. Findings from this study showed that although there was one measure that resulted in substantial agreement (hat use for parents on the weekday), the overall agreement across groups and clothing categories was slightly lower than the previously reported studies. One possible explanation pertains to the difficulty of extracting records from the diary regarding the clothing worn at the exact time the direct observation was undertaken, as a result contributing to a lower level of agreement between diary and observation data. For example, this may explain the under-reporting of shirts with sleeves, as participants may have worn a shirt with sleeves during the assessment, but removed them to go for a swim immediately after the assessment occurred.
Few studies have reported comparisons between different types of self-report measures (ie, diary and survey measures) or concurrent validity, a practice that is common in other fields of health behavior such as physical activity and nutrition.5 Our study found that overall, there was a good level of agreement between 4-day diary records and survey records. With the exception of the sunglass use by lifeguards (r = 0.18) and hat use by children (r = 0.27), all other protective practices among participants were correlated at levels between 0.42 and 0.60. This finding is very encouraging, as it indicates that a 4-day diary, including two weekdays and two weekend days, provides not only an adequate estimate of behavior throughout the week,15 but a satisfactory estimate of usual or habitual sun-protective behavior for the participants of this study.
Interpretation of the findings reported in this study should be tempered by the possibility that involvement in this study may have resulted in reactivity among participants. However, given the level of agreement between diary records and the survey completed before involvement in the study, it appears that being involved in the study was a minimal disruption to their usual sun-protective coverage practices. Also, although several observations throughout the 4-day period would have been desired, this would have been cost prohibitive. The sample was relatively affluent and all were swimming pool patrons, which may limit the generaliz-ability of these findings. Finally, although some of the parents completed the survey in consultation with their children, there was no statistically significant difference between how parents completed the diary and their covering-up practices.
This study contributes to the growing number of publications that have reported the validity of measures being used to assess sun-protective practices. Future studies should evaluate the validity of their verbal reporting methods of measuring covering-up behavior on subsamples to ensure quality measures in various study contexts. This will increase confidence in the accuracy of outcome measures of sun-protection behaviors and enable the field of skin cancer prevention research to continue to advance.5
Acknowledgments
We would like to thank the pool managers for allowing us to undertake this study; parents, children, and pool lifeguards for their involvement; Maria Fawzy, Dawn Hall, Nancy Marencin, and Nicole Dubruiel for their assistance in recruitment and data collection; and Tom Elliott for his assistance with staff training and project coordination.
Supported by grant CA 92505-S1 from the National Cancer Institute.
Footnotes
Conflicts of interest: None declared.
REFERENCES
- 1.Jemal A, Devesa SS, Fears TT, Hartge P. Cancer surveillance series: changing patterns of cutaneous malignant melanoma mortality rates among whites in the United States. J Natl Cancer Inst. 2000;92:811–8. doi: 10.1093/jnci/92.10.811. [DOI] [PubMed] [Google Scholar]
- 2.Jemal A, Devesa SS, Hartge P, Tucker MA. Recent trends in cutaneous melanoma incidence among whites in the United States. J Natl Cancer Inst. 2001;93:678–83. doi: 10.1093/jnci/93.9.678. [DOI] [PubMed] [Google Scholar]
- 3.American Cancer Society . Cancer prevention and early detection facts and figures. American Cancer Society; Atlanta (GA): 2007. [Google Scholar]
- 4.Creech LL, Mayer JA. Ultraviolet radiation exposure in children: a review of measurement strategies. Ann Behav Med. 1997;19:399–407. doi: 10.1007/BF02895159. [DOI] [PubMed] [Google Scholar]
- 5.Glanz K, Mayer JA. Reducing ultraviolet radiation exposure to prevent skin cancer methodology and measurement. Am J Prev Med. 2005;29:131–42. doi: 10.1016/j.amepre.2005.04.007. [DOI] [PubMed] [Google Scholar]
- 6.Girgis A, Sanson-Fisher RW, Tripodi D, Golding T. Evaluation of interventions to improve solar protection in primary schools. Health Educ Q. 1993;20:275–87. doi: 10.1177/109019819302000217. [DOI] [PubMed] [Google Scholar]
- 7.Girgis A, Sanson-Fisher RW, Watson A. A workplace intervention for increasing outdoor workers' use of solar protection. Am J Public Health. 1994;84:77–81. doi: 10.2105/ajph.84.1.77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Oh SS, Mayer JA, Lewis EC, Slymen DJ, Sallis JF, Elder JP, et al. Validating outdoor workers' self-report of sun protection. Prev Med. 2004;39:798–803. doi: 10.1016/j.ypmed.2004.03.011. [DOI] [PubMed] [Google Scholar]
- 9.O'Riordan DL, Lunde KB, Steffen AD, Maddock JE. Validity of self-report sun habits of beachgoers. Arch Dermatol. 2006;142:1304–11. doi: 10.1001/archderm.142.10.1304. [DOI] [PubMed] [Google Scholar]
- 10.Glanz K, Geller AC, Shigaki D, Maddock JE, Isnac MR. A randomized trial of skin cancer prevention in aquatics settings: the pool cool program. Health Psychol. 2002;21:576–87. [PubMed] [Google Scholar]
- 11.Glanz K, Steffen A, Elliott T, O'Riordan DL. Diffusion of an effective skin cancer prevention program: design, theoretical foundations, and first-year implementation. Health Psychol. 2005;24:477–87. doi: 10.1037/0278-6133.24.5.477. [DOI] [PubMed] [Google Scholar]
- 12.O'Riordan DL, Glanz K, Gies P, Elliott T. A pilot study of self-reported ultraviolet radiation exposure and sun protection practices among lifeguards, parents and children. Photochem Photobiol. 2008;84:774–8. doi: 10.1111/j.1751-1097.2007.00262.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Glanz K, Chang L, Song V, Silverio R, Muneoka L. Skin cancer prevention for children, parents, and caregivers: a field test of Hawaii's SunSmart program. J Am Acad Dermatol. 1998;38:413–7. doi: 10.1016/s0190-9622(98)70498-0. [DOI] [PubMed] [Google Scholar]
- 14.Glanz K, Lew RA, Song V, Cook VA. Factors associated with skin cancer prevention practices in a multiethnic population. Health Educ Behav. 1999;26:344–59. doi: 10.1177/109019819902600305. [DOI] [PubMed] [Google Scholar]
- 15.Glanz K, Silverio R, Farmer A. Diary reveals sun protection practices. Skin Cancer Found J. 1996;14:27–9. [Google Scholar]
- 16.Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977;33:363–74. [PubMed] [Google Scholar]
- 17.Escoffery C, Glanz K, Elliott T. Process evaluation of the pool cool diffusion trial for skin cancer prevention across 2 years. Health Educ Res. 2007;23:732–43. doi: 10.1093/her/cym060. [DOI] [PMC free article] [PubMed] [Google Scholar]