Abstract
Introduction
To determine whether a web-based survey was an acceptable method of data collection for a clinic-based case-control study of adult brain cancer, the authors compared the reliability of paired responses to a main and resurvey for participants completing surveys by telephone (N=74) or self-administered on the web (N=465) between 2003 and 2006.
Methods
Recruitment of cases was performed at the Evanston Northwestern Healthcare Kellogg Cancer Care Center and the Duke University Medical Center Cancer Control division, and controls were friends and siblings of cases. Twenty-five variables were examined including smoking, oral contraceptive and residential histories, water sources, meat preparation, fruit and vegetable consumption and pesticide use. Weighted and simple kappas were estimated for categorical and binary variables, respectively.
Results
The number of concordant paired responses were summed for use in linear regression. Respondents were 97 percent White and 85 percent had post-secondary education. Kappas for individual questions ranged from 0.31 (duration of residence in a single family house) to 0.96 (ever smoked) with a median of 0.57 (95 percent CI: 0.47, 0.64). The median number of concordant responses was 16.2 (range: 5–22). Reliability was greater for controls than cases, web-based vs. telephone responders, females and higher income responders. Frequency of email and internet use was not associated with reliability.
Conclusions
A self-administered, web-based survey was a feasible and appropriate mode of interview in this study. The comparable reliability of web compared to telephone responses suggest that web-based self-interviews could be a cost-effective alternative to traditional modes of interview.
Keywords: Reproducibility of results, brain tumors, online data collection, risk factors, kappa
INTRODUCTION
According to the Pew Internet and American Life Project, Internet usage in the United States has risen over time in all age categories, with 71 percent of American adults currently using the Internet at least occasionally. Though this percentage is higher among younger adults, 65 percent of Americans aged 50–64 currently use the Internet and 93 percent of adults living in a household with an income greater than $75,000 per year are online1. Web-based technology has been used extensively to collect data in the areas of public opinion and commercial research, but has been underutilized as a method of data collection for epidemiologic studies2, especially in the exploration of the role of human exposures in the causation of cancer. Underutilization of this technology may be due a concern that general population studies would include individuals who are not regular Internet users, and may not be comfortable completing a lengthy exposure assessment questionnaire online.
While it has been recommended to keep the length of web-administered surveys to less than 20 minutes3, risk factor questionnaires tend to be considerably longer, especially when studying a disease with a complex and largely unknown etiology. Since using the web for exposure assessment in health research is so new, it is important to assess the reliability of responses obtained from this mode of data collection. Our pilot case-control study of brain cancer provided an opportunity to compare the reliability of paired responses to a questionnaire administered either by telephone or via the Internet.
Between 2003 and 2006, we collected data from brain tumor cases and their friend and sibling controls regarding their personal histories and self-reported exposures to substances that have been demonstrated as animal neurocarcinogens, including occupational, environmental, and food exposures. Participants had the option of completing the interview either by telephone or self-administered over the internet. The sub-study described here examined the reliability of responses to a subset of items on the main questionnaire. This reliability sub-study had three primary goals: 1) to determine whether participants would choose the web self-interview or would opt for the more traditional telephone interview; 2) to determine whether the reliability of responses from the web-based survey were comparable to the those from the telephone-administered survey; and 3) to determine whether reliability was impacted by case-control status and other participant characteristics. To our knowledge, this is the first reliability study to be performed for a comprehensive risk factor questionnaire in a sample of brain tumor cases and their friend and sibling controls.
MATERIALS AND METHODS
Research design overview
Brain cancer patients were ascertained at two specialty clinics (Duke University Medical Center in North Carolina and Evanston Northwestern Healthcare in Illinois) that obtain patient referrals from a broad geographic region. The Duke clinic serves brain tumor patients of all ages with approximately one-third coming from North Carolina, South Carolina and southern Virginia, and the rest coming from other areas of the United States or other countries. The Evanston Northwestern healthcare clinic draws 93 percent of its patient referrals from Illinois, with the remainder primarily from other Midwest states such as Wisconsin and Indiana. Cases were ascertained between 2003 and 2006. To be eligible, cases must (1) have had a histologically confirmed diagnosis of a first primary brain tumor (ICDO-3 sites C70.0–C72.9 and C75.1–C75.3) with the following histologies: glioblastoma (ICDO-3 histology codes 9440–9442), astrocytoma grades 2 and 3 (9400–9411 and 9420), or oligodendroglial (9382 and 9450–9451); (2) be over 18 years of age, (3) speak English and (4) reside within the United States. Cognitive function of cases was assessed by the doctor at the time consent was obtained.
Since it was not possible to identify and draw a sample of controls representing the underlying referral population for the two clinics, patients were asked to provide names and contact information for up to two siblings and three friends that might agree to serve as study controls. Eligible sibling and friend controls had to be at least 18 years old, residents of the United States, and have no history of brain cancer. IRB approval for this study was obtained from the University of Illinois at Chicago (UIC), Duke University, and Evanston Northwestern Healthcare IRBs. Participant recruitment and consent to participate in each section of the study was performed in person with cases at the clinic and by telephone or mail for controls. Case and control interviews were conducted between 2003 and 2006.
Choice of survey format
Participants were encouraged to complete the interview via the internet but were offered a telephone-administered interview if they preferred. Patients who chose to complete the web-based version were provided with a packet containing a confidential and unique username and password and a set of detailed instructions. The home page for the web-based survey listed the different sections of the instrument and the survey had a status bar on each page to show the participant what percentage of the survey questions was completed. Participants completing the web-based survey had the opportunity to complete it at their leisure and at multiple locations, to log in and out multiple times as needed or desired, and to call the toll-free support line if they had questions or problems. Participants who chose the telephone-administered survey were also given the opportunity to complete the survey during more than one call if needed. While the questions and response choices were identically worded for the web- and telephone-administered versions of the survey, some transition phrases were added to the telephone version to facilitate the segue from one section to the next.
A total of 679 cases were eligible to participate in the study, 359 of which consented to the main survey. 269 cases completed the main survey, 49 by telephone and 220 over the internet, which resulted in an overall response rate of 40 percent for cases in the main survey. A total of 651 controls were eligible to participate in the study, 532 of which consented to the main survey. 400 controls completed the main survey, 48 by telephone and 353 over the internet, which resulted in an overall response rate of 61 percent for controls in the main survey.
Reliability sub-study
Several weeks after completing the main survey, all participants were re-contacted and asked to complete a short resurvey. The resurvey consisted of a subset of 25 questions from the main survey. Brain tumor cases and their sibling and friend controls were included in this analysis if they completed both the main survey and the brief resurvey using the same survey mode for each, either by telephone or self-administered on the web.
Of the 269 cases and 400 controls who completed the main survey, 222 (83%) and 337 (84%) also completed a resurvey, respectively. Twenty participants were excluded from the reliability study because they completed the main survey and resurvey via different modes. Of the 539 remaining, 74 completed both surveys via telephone and 465 completed both online. The median time between completion of the main survey and the resurvey was 36 days via telephone and 48 days via the web.
The following 25 questions from the main survey were asked during the re-survey: where did the respondent stay as a child (home, relative’s home, daycare); primary source of drinking water as a child (city water, household well, bottled water); ever smoked cigarettes, average number of cigarettes smoked per day (continuous); frequency of adult dental x-rays (Never/only as a child, At least once a year, 2–3 years, 4–5 years, or less often); ever used oral contraceptives; age at which the respondent began using oral contraception (continuous); and duration of use (continuous). There were separate questions on duration of residence (0 years, 1–9 years, 10–19 years and 20 or more years) in each of the following housing structures: a mobile home or trailer, one-family house, one-family house attached to one or more houses, a building with 1 to 9 apartments, a building with 10 to 49 apartments, and a building with 50 or more apartments. Respondents were asked separately how long they had ever lived near an industrial facility or a gas station, using the same scale. There were also separate questions about duration of residence (0 years, 1–9 years, 10–19 years and 20 or more years) with the following domestic water sources: public or commercial water system; private well; and cistern. Questions were asked about the frequency of consumption (never, rarely, sometimes, often) of broiled food; grilled or barbecued food; and food charred or blackened by burning; as well as the frequency of consumption of home-grown fruits and vegetables during the summertime. Finally, there were questions on the duration of residence on a farm and the number of years residing in a place where pesticides were professionally applied indoors (each with response categories of 0 years, 1–9 years, 10–19 years and 20 or more years).
Statistical Analysis
We calculated the percentage of participants choosing the web-based survey, overall and by case/control status, within categories of age, gender, race, education, income, and frequency of internet and email use. Chi-square tests of association were performed overall and by case/control status to determine whether choice of survey mode was significantly associated with these characteristics (alpha=0.05). In addition, a separate logistic regression model predicting choice of survey mode was fit for each characteristic, including terms for case/control status and the interaction between case/control status and the characteristic, to determine whether the relationship between survey mode and the characteristic differed by case/control status. The p-value from the Type 3 test was reported.
We also examined how survey process characteristics varied by survey mode. These included the perceived level of difficulty completing the survey, perceived length of the survey, whether the respondent received help in completing the survey, and the total number of minutes, sessions and days to complete the survey.
To examine the reliability of responses to individual questions, we recoded the continuous variables, number of cigarettes smoked per day (1–4, 5–9, 10–19, 20 or more), the number of years oral contraceptives were used (1 or less, 2, 3–5, 6–10, more than 10 years) and the participant’s age when she started oral contraceptives (less than 20, 20–29, 30–39, 40–49, 50 or older), into 4–6 categories and estimated simple and weighted kappas for binary and ordinal categorical variables, respectively, along with their 95 percent asymptotic confidence limits4–6. Estimated kappa values for each question were graphed in ascending order. We then re-estimated and plotted kappas after stratifying on survey mode, case-control status, gender, and income level.
In addition, we summed the number of concordant responses between the main survey and resurvey for each participant across the 22 questions, after excluding the three questions on oral contraceptive histories that were not asked of men. The result was an ordinal variable that was approximately normally-distributed, with a possible range between 0 and 22. We used this variable as a dependent variable in linear regression analyses in order to examine participant-level predictors of reliability. First, the dependent variable was regressed one at a time with each independent variable in order to screen for variables to include in a final model. The only variables that were associated with reliability were gender, case-control status, survey mode, and household income, and these were included together in the final multivariate linear regression model of reliability.
RESULTS
Respondents to the main survey were 97 percent White, with 85 percent having post- secondary education (data not shown). Controls and participants who were younger, college-educated and who had higher incomes were more likely to choose the web-based survey over the telephone-administered survey. There was a monotonic increase in choice of web-based survey with decreasing age, increasing household income, and increasing frequency of email and internet use (Table 1). There was no difference in the relationship between survey mode and participant characteristics for cases compared to controls.
Table 1.
Percentage of respondents who chose the web-based survey format, overall and by case/control status, and associations between the choice of survey mode (web vs. telephone-administered) and participant characteristics
| Total Cases Choosing Web | Total Controls Choosing Web | Total Participants Choosing Web |
|||
|---|---|---|---|---|---|
| Characteristic | No. | % | p value^ | ||
| Case-control status | 209 | 330 | |||
| Case | 209 | -- | -- | 83 | |
| Sibling control | 153 | -- | -- | 90 | |
| Friend control | 177 | -- | -- | 88 | |
| P-value* | -- | -- | 0.15 | -- | |
| Age | |||||
| <40 | 143 | 95 | 96 | 96 | |
| 40–49 | 125 | 84 | 88 | 86 | |
| 50–59 | 159 | 77 | 89 | 85 | |
| 60+ | 107 | 67 | 80 | 76 | |
| P-value* | 0.002 | 0.02 | 0.0001 | 0.79 | |
| Gender | |||||
| Male | 267 | 84 | 92 | 88 | |
| Female | 275 | 82 | 86 | 84 | |
| P-value* | 0.67 | 0.09 | 0.19 | 0.37 | |
| Race/ethnicity | |||||
| White | 517 | 85 | 89 | 87 | |
| NonWhite | 16 | 82 | 100 | 88 | |
| P-value* | 0.81 | 0.43 | 0.97 | 0.97 | |
| College educated | |||||
| No | 74 | 61 | 72 | 68 | |
| Yes | 465 | 87 | 91 | 89 | |
| P-value* | 0.0006 | 0.0003 | 0.0001 | 0.94 | |
| Household Income | |||||
| <25,000 | 35 | 75 | 53 | 63 | |
| 25–50,000 | 72 | 73 | 88 | 83 | |
| 50–75,000 | 103 | 73 | 89 | 86 | |
| 75–100,000 | 90 | 79 | 94 | 88 | |
| >100,000 | 229 | 90 | 92 | 91 | |
| P-value* | 0.24 | <0.0001 | 0.0002 | 0.16 | |
| Have an email account | |||||
| No | 49 | 55 | 59 | 57 | |
| Yes | 490 | 86 | 91 | 89 | |
| P-value* | 0.0005 | <0.0001 | 0.0001 | 0.52 | |
| Check email daily | |||||
| Daily | 315 | 92 | 93 | 93 | |
| Few time a week | 111 | 83 | 90 | 87 | |
| Weekly | 46 | 81 | 90 | 87 | |
| Less/Never | 18 | 44 | 69 | 44 | |
| P-value* | <0.0001 | 0.02 | 0.0001 | 0.87 | |
| Use the internet daily | |||||
| Daily | 204 | 91 | 96 | 94 | |
| Few time a week | 160 | 88 | 91 | 89 | |
| Weekly | 71 | 86 | 95 | 92 | |
| Less/Never | 104 | 60 | 64 | 63 | |
| P-value* | <0.0001 | <0.0001 | 0.0001 | 0.57 | |
p-value for the chi-square test of the association between survey choice and characteristics, among cases only, controls only, and overall
p-value for the interaction term between case/control status and the characteristic, from a logistic regression model predicting survey mode
Web respondents were more likely than telephone respondents to report receiving help from another person while completing the survey. In addition, web respondents were more likely to have more than one login session and complete the main survey during a span of more than one day, when compared to telephone respondents (Table 2).
Table 2.
Distribution of interview process characteristics and associations between the choice of survey mode (web vs. telephone-administered) and interview characteristics
| Telephone | Web | p value | ||
|---|---|---|---|---|
|
|
||||
| N | (%) | (%) | ||
| Questionnaire too difficult | ||||
| No | 496 | 93 | 92 | |
| Yes | 39 | 7 | 8 | 0.85 |
| Questionnaire too long | ||||
| No | 407 | 88 | 82 | |
| Yes | 85 | 12 | 18 | 0.23 |
| Received help | ||||
| No | 440 | 91 | 81 | |
| Yes | 97 | 9 | 19 | 0.04 |
| Minutes to complete | ||||
| <=30 | 16 | 0 | 6 | |
| 31–60 | 193 | 57 | 57 | |
| 61–90 | 80 | 37 | 22 | |
| >90 | 47 | 7 | 15 | 0.036 |
| Sessions to complete | ||||
| 1 | 252 | 65 | 44 | |
| 2 | 145 | 20 | 28 | |
| 3 | 78 | 8 | 15 | |
| 4+ | 64 | 7 | 13 | 0.008 |
| Days to complete | ||||
| Within a day | 454 | 94 | 83 | |
| Longer than a day | 81 | 6 | 17 | 0.016 |
Across all 25 questions the overall median value of kappa was 0.57 (range 0.31 to 0.96). Kappa values were generally higher among web-based compared to telephone-based respondents. Questions on oral contraceptive use and smoking had the highest kappa values overall, ranging from 0.75 to 0.96. The four questions on frequency of dietary habits had kappa values ranging from 0.40 to 0.50, while the questions related to residential histories had a broad range of kappa values from 0.31 to 0.81 (Table 3).
Table 3.
Kappa values and 95% Confidence Intervals for the 25 binary and ordinal variables included in both the main survey and resurvey, overall and by survey mode, ordered by increasing kappa value overall
| Overall | Web | Telephone | |||||
|---|---|---|---|---|---|---|---|
| # | Variable | Kappa | 95% CI | Kappa | 95% CI | Kappa | 95% CI |
| 1 | Duration of residence in a single family home | 0.31 | 0.18,0.44 | 0.34 | 0.20,0.48 | 0.14 | −0.22,0.50 |
| 2 | Duration of residence in a mid-sized apartment building | 0.39 | 0.31,0.47 | 0.41 | 0.32,0.50 | ** | **** |
| 3 | Duration of drinking water from cistern | 0.40 | 0.23,0.57 | 0.46 | 0.27,0.65 | ** | **** |
| 4 | Frequency of consuming broiled food | 0.40 | 0.34,0.46 | 0.42 | 0.35,0.48 | 0.32 | 0.14,0.50 |
| 5 | Frequency of consuming charred food | 0.43 | 0.37,0.49 | 0.38 | 0.32,0.45 | 0.67 | 0.55,0.80 |
| 6 | Where stayed as a child | 0.47 | 0.34,0.60 | 0.48 | 0.34,0.61 | ** | **** |
| 7 | Frequency of consuming grilled food | 0.48 | 0.41,0.55 | ** | **** | 0.40 | 0.21,0.58 |
| 8 | Duration of residence in a small apartment building | 0.49 | 0.42,0.56 | 0.52 | 0.44,0.59 | 0.31 | 0.10,0.52 |
| 9 | Frequency of consuming fresh fruits and vegetables | 0.50 | 0.44,0.55 | 0.49 | 0.43,0.55 | 0.55 | 0.41,0.68 |
| 10 | Duration of residence near an industrial facility | 0.52 | 0.38,0.66 | 0.52 | 0.36,0.67 | 0.54 | 0.24,0.84 |
| 11 | Frequency of dental x-rays in adulthood | 0.52 | 0.46,0.58 | 0.49 | 0.42,0.56 | 0.71 | 0.58,0.84 |
| 12 | Duration of using professional exterminator | 0.55 | 0.49,0.60 | 0.55 | 0.49,0.62 | ** | **** |
| 13 | Duration of drinking water from a public source | 0.57 | 0.50,0.64 | 0.59 | 0.52,0.67 | 0.38 | 0.11,0.65 |
| 14 | Duration of residence near a gas station | 0.58 | 0.49,0.66 | 0.59 | 0.50,0.68 | 0.52 | 0.30,0.74 |
| 15 | Duration of residence in a large apartment building | 0.60 | 0.52,0.69 | 0.64 | 0.55,0.73 | 0.30 | −0.01,0.61 |
| 16 | Duration of residence in an attached home | 0.61 | 0.53,0.68 | 0.62 | 0.55,0.70 | 0.49 | 0.30,0.69 |
| 17 | Source of drinking water as a child | 0.63 | 0.53,0.73 | 0.64 | 0.54,0.75 | ** | **** |
| 18 | Duration of residence on a farm | 0.64 | 0.56,0.72 | 0.65 | 0.57,0.74 | 0.57 | 0.36,0.77 |
| 19 | Duration of drinking water from a private well | 0.72 | 0.67,0.77 | 0.76 | 0.71,0.80 | 0.52 | 0.35,0.68 |
| 20 | Age started oral contraceptives | 0.75 | 0.68,0.83 | 0.80 | 0.72,0.87 | 0.53 | 0.25,0.82 |
| 21 | Years used oral contraceptives | 0.78 | 0.73,0.82 | 0.77 | 0.72,0.82 | 0.79 | 0.68,0.90 |
| 22 | Duration of residence in a mobile home | 0.81 | 0.74,0.88 | 0.82 | 0.74,0.90 | 0.76 | 0.59,0.94 |
| 23 | Cigarettes smoked per day | 0.90 | 0.88,0.93 | 0.90 | 0.87,0.93 | 0.91 | 0.86,0.96 |
| 24 | Ever used oral contraceptives | 0.92 | 0.85,0.98 | 0.96 | 0.91,1.00 | 0.76 | 0.53,0.98 |
| 25 | Ever smoked cigarettes | 0.96 | 0.94,0.99 | 0.97 | 0.95,0.99 | 0.92 | 0.83,1.00 |
Kappa not computed because zero participants chose that value on the other survey.
Kappa values for individual questions tended to be higher for web respondents, controls, women and higher income respondents when compared to telephone respondents, cases, men and lower income respondents, respectively (Figure 1). Within cases, kappa values did not appear to be different for patients with more (high grade tumor or a glioblastoma) versus less severe disease, though data were sparse (data not shown).
Figure 1.
Kappa values for individual questions, ordered from Lowest to highest Kappa value, by Interview Mode, Case-Control Status, Gender, Age, Income, and Education
The mean number of concordant responses across the 22 questions that were common to both men and women was 16.2 (median = 16), ranging from 5 to 22. Table 4 shows that average concordance was 0.56 questions higher in web respondents vs. telephone respondents (p value =0.07), and 0.40 questions higher in controls vs. cases (p value =0.06). Average concordance was also higher in women versus men and in participants with higher incomes (Table 4). Neither age, educational attainment, frequency of email use, nor frequency of internet use were associated with number of concordant responses. The association between survey mode and the number of concordant responses did not differ by case control status, given that an interaction term added to the final model was not statistically significant (p=0.75) (data not shown).
Table 4.
Bivariate and multivariate associations between survey mode (web vs. telephone-administered), interview process characteristics, demographics, and the number of concordant responses in the main survey and resurvey*
| Bivariate |
Multivariate |
||||
|---|---|---|---|---|---|
| Variable | # | Beta | p value | Beta | p value |
| Status | |||||
| Case (ref) | 209 | ||||
| Control | 330 | 0.437 | 0.038 | 0.40 | 0.061 |
| Survey Mode | |||||
| Telephone (ref) | 74 | ||||
| Web | 465 | 0.711 | 0.017 | 0.56 | 0.068 |
| Gender | |||||
| Male (ref) | 264 | ||||
| Female | 275 | 0.369 | 0.073 | 0.40 | 0.051 |
| Income | |||||
| Ordinal, 5 categories | 529 | 0.300 | 0.0001 | 0.29 | 0.0002 |
| Age | |||||
| Years (ordinal) | 534 | 0.0011 | 0.89 | ||
| Any college | |||||
| Yes (ref) | 465 | ||||
| No | 74 | 0.197 | 0.51 | ||
| Email Frequency | |||||
| Ordinal, 4 categories | 490 | −0.075 | 0.53 | ||
| Internet Frequency | |||||
| Ordinal, 4 categories | 539 | −0.057 | 0.40 | ||
Number of concordant responses was measured on a scale from 1-22, excluding 3 questions for oral contraceptive use, which only applied to women.
DISCUSSION
Results from this reliability study suggest that a self-administered, web-based survey is a feasible and appropriate method for collecting data about environmental and other exposure conditions in case-control studies of malignant brain cancer. Upon encouragement to do so, participants overwhelmingly chose the web-based survey mode over the telephone-administered survey, and the reliability of responses via the web exceeded the reliability of responses via telephone, even after controlling for the demographics of participants and frequency of email and internet use. Increased reliability among web-based respondents may have been due in part to their ability to complete the main survey over several sessions and a longer period of time, which may have allowed them more opportunity for retrospection, verification, and seeking help from others when recalling events. Alternatively, since participants were not randomized to either the web-based or telephone-administered survey mode, the apparently greater reliability for web-based respondents may be due to a tendency for more reliable responders to choose the web-based survey. While we attempted to control for differences in socioeconomic status and level of comfort with the internet, residual confounding may still explain the greater reliability observed for web-based responses. The relatively low response rate in this study may have inflated estimates of reliability for both modes of interview, if non-responders tended to also be less reliable reporters. Nonetheless, our results suggest that the reliability of web-based responders is at least comparable to those via telephone.
Studies in other areas of health research have shown better or equal reliability for web-based surveys compared with telephone-administered or paper-based surveys for exposures such as alcohol intake7–9, general health status, and smoking cessation7. Most of these studies randomized participants into one survey mode or the other and were performed in younger populations, such as college students, or in populations of Internet users recruited through websites. Our clinic-based sample of brain tumor patients was very different from these populations. Among our healthy adult controls, who are more comparable to populations previously studied, the number of concordant responses between the main survey and the resurvey was significantly higher for the web-based survey compared to the telephone survey (data not shown).
As anticipated, survey responses were more reliable for controls than for brain cancer cases, consistent with the cognitive decline affecting memory and attention that often occurs in brain tumor cases as a result of their disease10 or treatment for their disease11. Contrary to expectation, however, we did not find differentially lower reliability among cases with glioblastomas and other high grade tumors, for whom cognitive decline might be expected to be more pronounced than those with lower grade tumors. This may have been the result of data sparseness due to stratifying within cases, which limited our ability to detect effects.
We found that the traditional epidemiologic risk factors had higher reliability than the dietary and environmental measures. Consistent with prior research12–15, reliability for smoking and oral contraceptive histories was very high in our study. Agreement for the four dietary history items in our study was found to be “moderate”, according to the cutoffs established by Landis and Koch16, and comparable to those from nutrient intake assessments in studies of other diseases17–21. The reliability of responses for the environmental exposures items such as housing, residence on a farm, pesticide exposure and water source were also generally modest. Because these individual food and environmental items will be used to construct participant-specific indices of potential neurocarcinogen exposure, the modest reliability for these responses could lead to exposure misclassification, which may substantially attenuate any associations between exposure and disease.
Prior research suggests that respondents are less likely to under-report sensitive issues in the context of a self-interview such as a paper-based survey, audio computer-assisted interview (ACASI), or a web-based survey9, 22, 23. While the interview for the current study did not focus on sensitive issues, recent evidence has suggested that exposure to marijuana may be related to the development of brain tumors24, so sensitive topics such as illicit drug use may be important in future data collection efforts in brain tumor research.
There are advantages to a computer-based self-interview when compared to other traditional paper-based self-interviews. The questionnaire can be programmed with logic so that questions are skipped automatically when not applicable, and error or warning messages can appear if responses do not meet a valid range of responses. These aspects of computer-assisted modes can reduce errors in respondent reports. There are additional advantages to web-based over computer-based interviews. First, respondents have the option of completing the interview at a time and place of their choosing, and over as many sessions and as long a period of time as needed. This enables respondents to choose moments when they are best able to focus on the topic at hand, which may make them less likely to rush their answers. Since they can start and stop the interview at any time, it also provides opportunities for soliciting information from relatives and others who are knowledgeable about the respondents’ exposure history in order to verify their own responses.
Web-based interviews are also more rapid and cost efficient than other interview modes9, 25, 26. While there are fees associated with programming the questionnaire online, other costs are saved, such as paper and postage for paper-based surveys and interviewer training and personnel time for telephone-based surveys. Also, since participants directly enter pre-coded data into the database, data entry costs are saved and data are available immediately for analysis27.
The comparable reliability and cost-efficiency of web-based versus telephone interviews in our study suggest that web-based self-interviews should be considered more often as the primary interview mode when planning epidemiologic studies in populations with internet access,, especially when the study sample is widely distributed geographically24, 27.
Future studies should focus on the feasibility and reliability of web-based surveys within studies of other populations and different disease states in order to fill the knowledge gaps in this area and describe the potential impact of survey mode on resulting measures of association between exposure and disease.
Acknowledgments
FINANCIAL SUPPORT:
National Cancer Institute Specialized Programs of Research Excellence in Brain Cancer grant [#1P20CA96890-01]; National Cancer Institute [Grant # 5 P50 CA 106743].
This study would not have been possible without the cooperation and hard work of the research staff at the Evanston Northwestern Healthcare Kellogg Cancer Care Center, Neuro-Oncology Program and the Duke University Medical Center, Cancer Control, Detection and Prevention Research Program, in addition the dedicated participants who have volunteered their time and energy to complete the activities of the study.
References
- 1.Demographics of Internet Users (Table). Updated 2007. [Accessed September 20, 2007]. Available at: http://www.pewinternet.org/trends/User_Demo_4.26.06.htm.
- 2.Tourangeau R. Survey research and societal change. Annu Rev Psychol. 2004;55:775–801. doi: 10.1146/annurev.psych.55.090902.142040. [DOI] [PubMed] [Google Scholar]
- 3.Umbach PD. Web surveys: Best practices. In: Porter SR, editor. Overcoming Survey Research Problems: New Directions for Institutional Research. 121. New Jersey: Jossey Bass; 2004. pp. 23–38. [Google Scholar]
- 4.Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37–46. [Google Scholar]
- 5.Fleiss JL, Cohen J, Everitt B. Large sample standard errors of kappa and weighted kappa. Pysch Bull. 1969;72:323–327. [Google Scholar]
- 6.Fleiss JL, Levine B, Paik MC. Statistical Methods for Rates and Proportions. 3. New Jersey: Wiley-Interscience; 2003. [Google Scholar]
- 7.Graham AL, Papandonatos GD, Bock BC, Cobb NK, Baskin-Sommers A, Niaura R, et al. Internet- vs. telephone-administered questionnaires in a randomized trial of smoking cessation. Nicotine and Tob Res. 2006;8(Suppl 1):S49–S57. doi: 10.1080/14622200601045367. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Miller ET, Neal DJ, Roberts LJ, Baer JS, Cressler SO, Metrik J, et al. Test-retest reliability of alcohol measures: Is there a difference between internet-based assessment and traditional methods? Psychol of Addict Behav. 2002;16( 1):56–63. [PubMed] [Google Scholar]
- 9.Parks KA, Pardi AM, Bradizza CM. Collecting data on alcohol use and alcohol-related victimization: A comparison of telephone and web-based survey methods. J Stud Alcohol. 2006;67:318–323. doi: 10.15288/jsa.2006.67.318. [DOI] [PubMed] [Google Scholar]
- 10.Tucha O, Smely C, Preier M, Lange KW. Cognitive deficits before treatment among patients with brain tumors. Neurosurgery. 2000;47( 2):324–333. doi: 10.1097/00006123-200008000-00011. [DOI] [PubMed] [Google Scholar]
- 11.Correa DD, DeAngelis LM, Shi W, Thaler HT, Lin M, Abrey LE. Cognitive functions in low-grade gliomas: Disease and treatment effects. J Neurooncol. 2007;81( 2):175–184. doi: 10.1007/s11060-006-9212-3. [DOI] [PubMed] [Google Scholar]
- 12.Karatela S, Purdie DM, Green AC, Webb PM, Whiteman DC. Repeatability of self-reported information for population-based studies of cancer. Asian Pacific J Cancer Prev. 2006;7:303–308. [PubMed] [Google Scholar]
- 13.Band PR, Spinelli JJ, Threlfall WJ, Fang R, Le ND, Gallagher RP. Identification of occupational cancer risks in British Columbia. part I: Methodology, descriptive results, and analysis of cancer risks, by cigarette smoking categories of 15,463 incident cancer cases. Jof Occup Environ Med. 1999;41( 4):224–232. doi: 10.1097/00043764-199904000-00004. [DOI] [PubMed] [Google Scholar]
- 14.Gartner CE, Battistutta D, Dunne MP, Silburn PA, Mellick GD. Test-retest repeatability of self-reported environmental exposures in parkinson's disease cases and healthy controls. Parkinsonism Relat Disord. 2005;11( 5):287–295. doi: 10.1016/j.parkreldis.2005.04.002. [DOI] [PubMed] [Google Scholar]
- 15.Reider CR, Hubble JP. Test-retest reliability of an epidemiological instrument for Parkinson's disease. J Clin Epidemiol. 2000;53( 8):863–865. doi: 10.1016/s0895-4356(00)00229-8. [DOI] [PubMed] [Google Scholar]
- 16.Landis JR, Koch G. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed] [Google Scholar]
- 17.Friis S, Kruger Kjaer S, Stripp C, Overvad K. Reproducibility and relative validity of a self-administered semiquantitative food frequency questionnaire applied to younger women. J Clin Epidemiol. 1997;50( 3):303–311. doi: 10.1016/s0895-4356(96)00379-4. [DOI] [PubMed] [Google Scholar]
- 18.Schaffer DM, Coates AO, Caan BJ, Slattery ML, Potter JD. Performance of a shortened telephone-administered version of a quantitative food frequency questionnaire. Ann Epidemiol. 1997;7( 7):463–471. doi: 10.1016/s1047-2797(97)00079-3. [DOI] [PubMed] [Google Scholar]
- 19.Parr CL, Veierod MB, Laake P, Lund E, Hjartaker A. Test-retest reproducibility of a food frequency questionnaire (FFQ) and estimated effects on disease risk in the Norwegian women and cancer study (NOWAC) Nutr J. 2006;5:4. doi: 10.1186/1475-2891-5-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Date C, Fukui M, Yamamoto A, Wakai K, Ozeki A, Motohashi Y, et al. Reproducibility and validity of a self-administered food frequency questionnaire used in the JACC study. J Epidemiol. 2005;15( Suppl 1):S9–23. doi: 10.2188/jea.15.S9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Margetts B, Nelson M, editors. Design Concepts in Nutritional Epidemiology. 2. USA: Oxford University Press; 1997. [Google Scholar]
- 22.Galesic M, Tourangeau R, Couper MP. Complementing random-digit-dial telephone surveys with other approaches to collecting sensitive data. Am J Prev Med. 2006;31( 5):437–443. doi: 10.1016/j.amepre.2006.07.023. [DOI] [PubMed] [Google Scholar]
- 23.Tourangeau R, Smith TW. Asking sensitive questions - the impact of data collection mode, question format, and question context. Public Opin Q. 1996;60( 2):275–304. [Google Scholar]
- 24.Eysenbach G, Wyatt J. Using the internet for surveys and health research. Journal of Medical Internet Research. 2002;4( 2):E13. doi: 10.2196/jmir.4.2.e13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Rhodes SD, Bowie DA, Hergenrather KC. Collecting behavioural data using the world wide web: Considerations for researchers. J Epidemiol Community Health. 2003;57( 1):68–73. doi: 10.1136/jech.57.1.68. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Mavis B, Doig K. The value of noncognitive factors in predicting students' first-year academic probation. Acad Med. 1998;73( 2):201–203. doi: 10.1097/00001888-199802000-00021. [DOI] [PubMed] [Google Scholar]
- 27.Van Selm M, Jankowski NW. Conducting online surveys. Qual Quant. 2006;40:435–456. [Google Scholar]