Abstract
Objective
The aim of this study is to investigate differences in responses related to (mental) health and behavior between two methods of data collection: web-based (web) and paper-and-pencil (p&p).
Study Design
Within each participating school all third-grade classes (mainly 14–15-year-old pupils) were randomly assigned to either the Internet condition (n=271) or the paper-and-pencil condition (n=261).
Principal Findings
Significant but small differences were found for the strengths and difficulties subscales “emotional symptoms” (p&p>web) and “prosocial behavior” (p&p>web), and carrying a weapon (web>p&p). Perceived level of privacy and confidentiality did not differ between the two modes.
Conclusions
The findings suggest that in a controlled school setting, web-based administration of health indicators yields almost the same results as paper-and-pencil administration. To generalize these findings, we recommend repeated studies in other populations and settings.
Keywords: Methodology, computerized questionnaire, preventive youth health care, adolescents, SDQ
Because of the many advantages, computerized questionnaires are used more often in youth surveys, replacing the more traditional paper-and-pencil (p&p) questionnaires. Advantages of computerized questionnaires are, for instance, the possibility to check directly for missing answers and consistency, automatic branching, cost advantages, and the possibility to give computerized tailored feedback on health and health-related behavior, which may be more effective than generic (paper) health advices (Brug, Campbell, and van Assema 1999). As the method of data collection can affect the answers that are obtained, especially for sensitive questions (Tourangeau and Smith 1996; Bowling 2005), it is important to determine whether responses to computerized questionnaires are comparable to those obtained by paper-and-pencil method.
So far, findings from previous studies among school-aged children and adolescents are inconclusive. Several studies did not find major differences between computerized and paper-and-pencil questionnaires (Truman et al. 2003; Hays and McCallum 2005; Mangunkusumo et al. 2005, 2006; McCabe et al. 2005), whereas other studies showed that adolescents disclose more sensitive information in computerized questionnaires than in paper-and-pencil conditions (Paperny et al. 1990; Turner et al. 1998; Wright, Aquilino, and Supple 1998; Wang et al. 2005) or found (gender-related) differences for some topics (Beebe et al. 1998; Webb et al. 1999; Hallfors et al. 2000; Vereecken and Maes 2006). Of these studies, a few were web-based (Mangunkusumo et al. 2005, 2006; McCabe et al. 2005; Wang et al. 2005).
Several factors may be responsible for the differences found in these studies. It is known that the validity of self-reports may be affected by cognitive as well as situational factors. Factors considered especially influential include the presence of others while responding to questions and the respondents’ perceptions of privacy or confidentiality (Brener, Billy, and Grady 2003; Bowling 2005). A perceived lack of privacy or confidentiality could cause response bias because of a fear of reprisal. In particular, behavior that is illegal, stigmatized, or laden with moral implications may be underreported because of this concern (Brener et al. 2003). As the setting of computerized administration of questionnaires at schools is likely to be different from the administration of paper-and-pencil questionnaires, it is important to investigate respondents’ perceived level of privacy and confidentiality as a possible source of response bias. However, little is known about adolescents’ perceived level of privacy and confidentiality in surveys. Because of this and the inconsistent pattern that appears from the literature, the following issues were examined in the present study: (a) (gender-related) differences in perceived level of privacy and confidentiality between web-based and paper-and-pencil questionnaires and (b) (gender-related) differences in the report of indicators of (mental) health and behavior between web-based and paper-and-pencil questionnaires when taking into account pupils’ perceived level of privacy and confidentiality.
Method
Study Design and Procedure
The present study was conducted as part of the Youth Health Monitor Rotterdam (YMR), a longitudinal youth health surveillance system. The YMR monitors the general health, well-being, behavior, and related factors of youth aged 0–19 years living in Rotterdam and its surroundings (the Netherlands) in order to supply information for youth policy at the school, neighborhood, and municipality level. The YMR is incorporated in the regular check-ups of the preventive youth health care system.
Five secondary schools with various educational levels were approached (and all agreed) for participation in this study. Within each school, all third-grade classes1 (n=26) were randomly assigned to either the Internet condition or the paper-and-pencil condition (p&p). Pupils assigned to the Internet condition completed the questionnaire in the school computer lab.
Four specially trained school nurses of the Municipal Public Health Service carried out the administration of the YMR questionnaire during regular class hours. Pupils were assured that all information provided was strictly confidential. It was stated (verbal and on paper) that completed questionnaires would “never be shown to parents, teachers, police, or anyone else, except to the school nurse.” Pupils were also informed that they could be invited for a health examination by the school nurse later on in the school year. After this introduction, pupils in the paper-and-pencil condition were given a questionnaire with their name printed on the front page. Pupils in the Internet condition were given an instruction sheet with their name, a username, a login code, and the name of the website (http://www.jeugdmonitorrotterdam.nl).
The two versions of the questionnaire were identical in terms of the questions asked, their wording, and their order of presentation in the survey. The computerized version differed from the paper-and-pencil version in the way the answering categories were presented. In the computerized version, all answering categories were presented below each question, whereas in the paper-and-pencil version, the answering categories of some questions were presented next to the question (in order to save printing space). Per-screen multiple questions of the same topic were presented. Questions that were not relevant to the pupil were not displayed. Logging out after completing the questionnaire was allowed only after answering all items.
Parents received an information letter and were given the opportunity to refuse their child's participation. In the Netherlands, for this kind of research (as part of a routine health examination offered to all children), informed consent is legally required.
The questionnaire was filled out in the classroom in about 1 hour in the presence of a teacher (for keeping order in the classroom) and a school nurse. A researcher was present when the Internet version was administered (for technical problems). Data were collected in November and December 2005.
Subjects
Overall response rate was 90 percent and did not differ by administration mode. Reasons for absence were mainly illness. A total of 532 pupils completed the questionnaire, 271 in the Internet condition and 261 in the paper-and-pencil condition. Because the data of one pupil (Internet condition) were not reliable, the data of 531 cases were analyzed.
Sociodemographic characteristics of the sample were equally distributed across Internet mode and paper-and-pencil mode (p≥.05, Table 1).
Table 1.
Sociodemographic Characteristics of the Study Population, by Mode of Administration
| Internet | Paper-and-Pencil | |||||
|---|---|---|---|---|---|---|
| n | % | n | % | χ2 (df) | p | |
| Sex | ||||||
| Boys | 152 | 56.3 | 127 | 48.7 | 3.10 (1) | .08 |
| Girls | 118 | 43.7 | 134 | 51.3 | ||
| Age | ||||||
| 14 years and younger* | 150 | 55.6 | 133 | 51.2 | 1.17 (2) | .56 |
| 15 years | 100 | 37.0 | 108 | 41.5 | ||
| 16 years and older | 20 | 7.4 | 19 | 7.3 | ||
| Level of education | ||||||
| Basic prevocational training | 119 | 44.1 | 118 | 45.2 | 0.15 (3) | .99 |
| Theoretical prevocational training | 20 | 7.4 | 20 | 7.7 | ||
| General secondary education | 75 | 27.8 | 72 | 27.6 | ||
| Preuniversity education | 56 | 20.7 | 51 | 19.5 | ||
| Ethnicity | ||||||
| Dutch | 113 | 41.9 | 121 | 46.7 | 3.03 (4) | .55 |
| Surinamese/Dutch Antillean/Aruban | 39 | 14.4 | 33 | 12.7 | ||
| Moroccan | 33 | 12.2 | 28 | 10.8 | ||
| Turkish | 36 | 13.3 | 25 | 9.7 | ||
| Other | 49 | 18.2 | 52 | 20.1 | ||
Five pupils were 13-years old.
Measures
(Mental) Health and Problem behavior
The Dutch self-report version of the Strengths and Difficulties Questionnaire (SDQ) was used (van Widenfelt et al. 2003). The SDQ comprises of five subscales of five items each: emotional symptoms, conduct problems, hyperactivity-inattention, peer problems, and prosocial behavior. Self-esteem was measured by the Dutch version of the Rosenberg Self-Esteem Scale (Rosenberg 1965; van der Linden, Dijkman, and Roeder 1983). Psychological well-being was measured with nine items about feelings and moods, derived from the Child Health Questionnaire (CHQ-CF87) (Landgraf, Abetz, and Ware 1996; Raat et al. 2002). Perceived health was assessed by means of one question in which the respondents were asked to rate their health (CBS 2001). Suicidal ideation was assessed with one item: “In the last twelve months, have you thought about putting an end to your life?” (Garnefski and Diekstra 1993). Suicide attempt was measured by the question: “Did you ever seriously attempt putting an end to your life?”
Furthermore, the questionnaire included four items about aggressive behavior and six items about delinquent behavior, developed and tested by the scientific research department of the Ministry of Justice (Junger-Tas, van der Laan, and Kruisink 1992): one question about carrying a weapon, one question about playing truant, and one question about bullying.
Health Risk Behavior and Sexual behavior
Questions on smoking, alcohol consumption, and cannabis use were based on a questionnaire used by the Netherlands Institute of Mental Health and Addiction (De Zwart 1999). Smoking habits were measured by the question: “Have you ever smoked cigarettes?” For alcohol consumption, we used the question concerning frequency of consumption in the past 4 weeks. Cannabis use was measured by the question “How often have you used cannabis in the past four weeks?” Sexual behavior was measured by the question: “Have you ever had sexual intercourse?” This question was not further specified.
Perceived Level of Privacy and Confidentiality
Perceived level of privacy and confidentiality was measured by two questions: “I had sufficient privacy when completing this questionnaire” and “I trust that the Municipal Health Service will treat my data confidentially” (1=fully agree, 4=fully disagree). A sum score of these two questions was calculated (Cronbach's α 0.66).
Data Analysis
Differences in responses on the health indicators between the two modes of administration were analyzed by two-way analysis of covariance (ANCOVAs). Condition (Internet, paper-and-pencil), sex (boy, girl), and interaction between condition and sex were entered in the analysis as independent variables. Perceived level of privacy and confidentiality was entered into the analyses as covariate in order to control for this factor as a source of bias. Before carrying out these analyses, we tested one of the assumptions for ANCOVAs, e.g., that the relationship between perceived level of privacy and confidentiality and the dependent variables is the same across the two modes of administration. These results showed no interaction effect between perceived level of privacy and confidentiality and the health indicators. Missing data were <5 percent and were excluded. Effect sizes were computed, considering f=0.10 as a small, f=0.25 as a medium, and f=0.40 as a large difference (Cohen 1988).
Results
No significant differences in perceived level of privacy and confidentiality between web-based and paper-and-pencil questionnaires were found (Table 2), and this did not differ by gender (data not shown).
Table 2.
Perceived Level of Privacy and Confidentiality, by Mode of Administration
| Internet (n=270) (%) | Paper-and-Pencil (n=261) (%) | p-value* | |
|---|---|---|---|
| I had sufficient privacy when completing this questionnaire | |||
| Fully agree | 30.7 | 33.7 | .56 |
| Agree | 51.9 | 49.4 | |
| Disagree | 14.1 | 12.9 | |
| Fully disagree | 3.3 | 4.0 | |
| I trust that the Municipal Health Service will treat my data confidentially | |||
| Fully agree | 47.4 | 43.4 | .26 |
| Agree | 44.1 | 45.0 | |
| Disagree | 4.8 | 8.0 | |
| Fully disagree | 3.7 | 3.6 | |
Mann-Whitney U-test.
Table 3 shows the means (SD) of the health indicators, by mode of administration and sex, and the results of the ANCOVA. For two subscales of the SDQ, differences were found. SDQ emotional symptoms were reported less frequently in the Internet condition. Pupils in the paper-and-pencil condition reported more prosocial behavior compared with pupils in the Internet condition. For both differences, effect sizes reached Cohen's level of “small.” For all indicators of well-being, no differences were found. Of the problem behavior items, main effects of condition were found for “carrying a weapon.” Pupils in the Internet condition reported significantly more often carrying a weapon compared with pupils in the paper-and-pencil questionnaire (small-effect size). No differences were found for smoking, use of alcohol, use of marijuana, and sexual behavior.
Table 3.
Means (standard deviation [SD]) by Sex and Condition; Results, Two-Way Analysis of Covariance (ANCOVA): Main Effect Condition, Interaction Effect Condition and Sex, and Main Effect Confidentiality
| Boys | Girls | ANOVA | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Internet (n=152) | p&p (n=127) | Internet (n=118) | p&p (n=134) | Condition | Condition Sex | Perceived Privacy and Confidentiality | ||||||||
| Measure | Mean | SD | Mean | SD | Mean | SD | Mean | SD | F | f | F | f | F | f |
| (Mental) health and behavior | ||||||||||||||
| Strengths and Difficulties Questionnaire | ||||||||||||||
| Total difficulties (sum score 0–40)†† | 8.7 | 3.9 | 9.7 | 5.2 | 10.3 | 5.0 | 10.0 | 5.0 | 0.60 | 1.32 | 4.78*,¶ | 0.10 | ||
| Emotional symptoms (sum score 0–10) | 1.6 | 1.4 | 2.1 | 1.9 | 3.0 | 2.1 | 3.3 | 2.2 | 3.92*,† | 0.09 | 0.07 | 0.00 | ||
| Conduct problems (sum score 0–10) | 2.0 | 1.3 | 2.0 | 1.4 | 1.6 | 1.4 | 1.6 | 1.3 | 0.03 | 0.04 | 0.77 | |||
| Hyperactivity-inattention (sum score 0–10) | 3.6 | 2.3 | 3.8 | 2.5 | 3.8 | 2.6 | 3.5 | 2.3 | 0.30 | 1.02 | 9.40**,¶ | 0.14 | ||
| Peer problems (sum score 0–10) | 1.5 | 1.3 | 1.8 | 1.6 | 1.8 | 1.6 | 1.7 | 1.5 | 0.86 | 2.15 | 1.47 | |||
| Prosocial behavior (sum score 0–10) | 6.3 | 1.9 | 6.7 | 1.9 | 7.8 | 1.7 | 8.2 | 1.5 | 8.56*,† | 0.13 | 0.20 | 13.3***,∥ | 0.16 | |
| Self-esteem (sum score 0–10) | 8.7 | 1.6 | 8.4 | 1.9 | 7.8 | 2.6 | 8.0 | 2.3 | 0.00 | 1.68 | 2.33 | |||
| Psychological well-being CHQ (sum score 0–100) | 80.5 | 10.3 | 81.0 | 12.4 | 73.0 | 16.6 | 75.0 | 15.9 | 1.53 | 0.16 | 0.14 | |||
| Perceived health (1–5) | 1.8 | 0.6 | 1.8 | 0.8 | 2.0 | 0.7 | 2.0 | 0.7 | 0.01 | 0.08 | 0.06 | |||
| Suicidal ideation (1–5), past year | 1.1 | 0.4 | 1.2 | 0.5 | 1.5 | 0.9 | 1.4 | 0.8 | 0.51 | 1.64 | 0.03 | |||
| Suicide attempt (1–3), ever | 1.0 | 0.1 | 1.1 | 0.3 | 1.1 | 0.4 | 1.1 | 0.4 | 0.02 | 0.40 | 0.35 | |||
| Aggressive behavior (sum score 0–16), past year | 1.5 | 2.3 | 1.5 | 2.5 | 0.6 | 1.4 | 0.5 | 1.1 | 0.75 | 0.01 | 2.15 | |||
| Vandalism and stealing (sum score 0–24), past year | 1.3 | 2.1 | 1.1 | 2.2 | 0.7 | 1.8 | 0.4 | 1.0 | 2.75 | 0.01 | 2.71 | |||
| Carrying a weapon (1–3) | 1.1 | 0.4 | 1.1 | 0.3 | 1.1 | 0.3 | 1.0 | 0.1 | 4.87*,‡ | 0.10 | 0.03 | 0.69 | ||
| Playing truant (1–6), past 4 weeks | 1.2 | 0.6 | 1.1 | 0.4 | 1.1 | 0.5 | 1.2 | 0.5 | 0.01 | 3.56 | 0.13 | |||
| Bullying (1–5), past 4 weeks | 1.3 | 0.6 | 1.5 | 0.8 | 1.1 | 0.5 | 1.2 | 0.5 | 3.46 | 0.98 | 4.82*,¶ | 0.10 | ||
| Health risk behavior | ||||||||||||||
| Smoking (1–5) | 1.6 | 0.9 | 1.6 | 1.1 | 1.7 | 1.1 | 1.6 | 1.0 | 0.38 | 0.72 | 0.00 | |||
| Use of alcohol (1–6), past 4 weeks | 1.6 | 0.9 | 1.6 | 1.0 | 1.6 | 0.9 | 1.4 | 0.8 | 0.62 | 1.30 | 0.36 | |||
| Use of marijuana (1–6), past 4 weeks | 1.1 | 0.4 | 1.2 | 0.6 | 1.1 | 0.6 | 1.1 | 0.4 | 0.03 | 0.34 | 0.01 | |||
| Sexual behavior (1–4) | 1.5 | 1.0 | 1.3 | 0.8 | 1.2 | 0.7 | 1.2 | 0.6 | 1.61 | 0.42 | 1.23 | |||
p<.05.
p<.01.
p<.001.
p&p>Internet.
Internet>p&p.
Boys, Internet>p&p.
High score<privacy and confidentiality.
High score>privacy and confidentiality.
The scale range for answers to each of the variables/sum scores are presented in parentheses after the variable name.
A total difficulties score on the SDQ is derived by summing four subscales relating to emotional symptoms, conduct problems, hyperactivity-inattention, and peer problems.
SD, standard deviation; ANOVA, analysis of variance; p&p, paper-and-pencil; CHQ, Child Health Questionnaire; SDQ, Strengths and Difficulties Questionnaire.
Finally, as can be seen from Table 3, a high score on SDQ total difficulties, SDQ hyperactivity-inattention, and bullying (indicating more problems) was associated with less privacy and confidentiality. A high score on SDQ prosocial behavior was associated with more perceived privacy and confidentiality.
Discussion
For most indicators of (mental) health and behavior, no significant (gender-related) differences between the two modes were found, which is in agreement with previous research reporting no or very few differences between computerized and paper-and-pencil studies (Truman et al. 2003; Hays and McCallum 2005; Mangunkusumo et al. 2005, 2006; McCabe et al. 2005). Significant but small differences between the two modes of data collection were found for the SDQ subscales “emotional symptoms” (p&p>web) and “prosocial behavior” (p&p>web), and carrying a weapon (web>p&p). Furthermore, no difference in perceived level of privacy and confidentiality between the two conditions was found.
Our results do not confirm the finding from other studies (Paperny et al. 1990; Turner et al. 1998; Wright et al. 1998; Wang et al. 2005) that adolescents disclose more sensitive information in computerized questionnaires. For sexual behavior, which can be regarded as the most sensitive topic in our survey, no differences between web-based and paper-and-pencil mode were found. Also, for other sensitive topics like use of alcohol, use of marijuana, vandalism, and stealing no differences were found. The only exception is found for carrying a weapon, which is reported more often in the web-based condition.
Although no differences in perceived level of privacy and confidentiality between the two modes were found, the percentage of pupils who do not agree with the statement “I had sufficient privacy when completing this questionnaire,” is high, i.e., about 17 percent in both conditions. For confidentiality assurances, this percentage is somewhat lower (about 10 percent), but still relatively high, despite the efforts we made to assure the confidentiality of the survey, e.g., training school nurses for the administration and closed boxes to put the questionnaires in after the paper-and-pencil administration. Given these results, it seems likely that some pupils were not sufficiently convinced of the confidentiality of the study. In addition, the knowledge that they could be invited for a health examination might have influenced their answer on the confidentiality question. As no comparable data are available, it is difficult to interpret these findings. Furthermore, the association of perceived level of privacy and confidentiality and the reporting of (sensitive) topics appeared to be limited, except for a few topics. Moreover, the association was not in the direction we expected. For example, a high SDQ total difficulties score was associated with less perceived privacy and confidentiality. This may be due to the order of the questions (Serdula et al. 1995; Dilman 2007), because questions about privacy and confidentiality were asked at the end of the questionnaire. Perhaps these questions reflect pupils’ need of privacy and confidentiality more than their actual sense of confidentiality. Further research is needed to investigate which factors play a role in pupils’ concern regarding privacy and confidentiality.
Limitations
A few limitations should be mentioned. The findings are applicable to the school setting as the use of web-based versus paper-and-pencil questionnaires in clinical settings and in other age groups was not studied. To generalize these findings, we recommend repeated studies in other populations and settings.
Concerning the study design, classrooms were randomly allocated to one of the two conditions. A disadvantage of this design is that within-pupils’ variance was not taken into account. A randomized crossover design would be more applicable. However, crossover designs may induce a carry-over effect, i.e., administration during the first period may carry over into the second administration period.
Implications
In the Netherlands, one of the main legal tasks of preventive youth health care is to monitor youth health and detect health risks. In Rotterdam, this is performed both individually and collectively. For third-grade pupils, the traditional paper-and-pencil questionnaire of the YMR (SDQ total score) is used as the first part of a two-step screening in order to preselect pupils for a visit to the school nurse. In the second step, the school nurse assesses the need for further referral, often with clinical screening measures. This two-step screening procedure is used because it is known that several clinical screening measures profit significantly in positive predictive power when used in groups that were preselected by a general screening measure (Loeber, Dishion, and Patterson 1984; Rothman and Greenland 1998). Our results show that this general screening procedure, as the SDQ total difficulties is concerned, is also possible by using computerized questionnaires in a school setting.
In conclusion, our study provides further evidence that in a controlled school setting, web-based administration of health indicators yields almost the same results as paper-and-pencil administration. However, this study also demonstrates that in general (regardless of the method of data collection), pupils’ sense of privacy and confidentiality is associated with some of their answers. Therefore, researchers should pay attention to guarantee sufficient privacy and confidentiality in classroom settings.
Acknowledgments
The authors thank Joop Duisterhout, M.Sc. (Department of Medical Informatics, Erasmus Medical Centre) for adjusting the Internet tool and his advice. We also thank the school nurses of the Municipal Public Health Service for administering the questionnaires, and the participating pupils for filling out the questionnaire.
Disclaimers and disclosures: The authors have no disclaimers or disclosures to report.
Note
Third-grade classes in the Netherlands are comparable with ninth-grade classes in the United States.
Supplementary material
The following supplementary material for this article is available online:
Appendix SA1: Author matrix.
This material is available as part of the online article from
http:// www.blackwell-synergy.com/doi/abs/10.1111/j.1475-6773.2008.00860.x
(this link will take you to the article abstract).
Please note: Blackwell Publishing is not responsible for the content or functionality of any supplementary materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.
REFERENCES
- Beebe TJ, Harrison PA, McRae JA, Jr., Anderson RE, Fulkerson JA. An Evaluation of Computer-Assisted Self-Interviews in a School Setting. Public Opinion Quarterly. 1998;62(4):623–32. [Google Scholar]
- Bowling A. Mode of Questionnaire Administration Can Have Serious Effects on Data Quality. Journal of Public Health (Oxford) 2005;27(3):281–91. doi: 10.1093/pubmed/fdi031. [DOI] [PubMed] [Google Scholar]
- Brener ND, Billy J O G, Grady WR. Assessment of Factors Affecting the Validity of Self-Reported Health-Risk Behavior among Adolescents: Evidence from the Scientific Literature. Journal of Adolescent Health. 2003;33(6):436–57. doi: 10.1016/s1054-139x(03)00052-1. [DOI] [PubMed] [Google Scholar]
- Brug J, Campbell M, van Assema P. The Application and Impact of Computer-Generated Personalized Nutrition Education: A Review of the Literature. Patient Education and Counseling. 1999;36(2):145–56. doi: 10.1016/s0738-3991(98)00131-1. [DOI] [PubMed] [Google Scholar]
- CBS. Jeugd, cijfers en feiten. Voorburg: CBS; 2001. [Google Scholar]
- Cohen J. In Statistical Power Analysis for the Behavior Sciences (Revised Edition) New York: Academic; 1988. [Google Scholar]
- De Zwart R. Nationaal Scholierenonderzoek 1996. Deel 2: Gedrag en Gezondheid van Scholieren. Utrecht, the Netherlands: NIBUD; 1999. [Google Scholar]
- Dilman DA. Mail and Internet Surveys: The Tailored Design Method. 2d Edition. Hoboken, NJ: John Wiley & Sons Inc; 2007. [Google Scholar]
- Garnefski N, Diekstra R F W. Scholierenonderzoek 1992: Landelijke Uitkomsten Rond Een Onderzoek Naar Gedrag en Gezondheid van ca. 11.000 Scholieren van het Voortgezet Onderwijs. Leiden, the Netherlands: Rijksuniversiteit Leiden; 1993. [Google Scholar]
- Hallfors D, Khatapoush S, Kadushin C, Watson K, Saxe L. A Comparison of Paper vs Computer-Assisted Self Interview for School Alcohol, Tobacco, and Other Drug Surveys. Evaluation and Program Planning. 2000;23(2):149–55. [Google Scholar]
- Hays S, McCallum RS. A Comparison of the Pencil-and-Paper and Computer-Administered Minnesota Multiphasic Personality Inventory-Adolescent. Psychology in the Schools. 2005;42(6):605–13. [Google Scholar]
- Junger-Tas J, van der Laan PH, Kruisink M. Ontwikkeling van de Jeugdcriminaliteit en de Justitiële Jeugdbescherming: Periode 1980–1990. Arnhem, the Netherlands: Gouda Quint; 1992. [Google Scholar]
- Landgraf JM, Abetz L, Ware JE. Child Health Questionnaire (CHQ): A User's Manual. Boston: The Health Institute, New England Medical Center; 1996. [Google Scholar]
- Loeber R, Dishion T, Patterson G. Multiple Gating: A Multistage Assessment Procedure for Identifying Youths at Risk for Delinquency. Journal of Research on Crime and Delinquency. 1984;21:7–32. [Google Scholar]
- Mangunkusumo R, Moorman P, Berg van den-Ruiterde A, Lei van der J, Koning de H, Raat H. Internet-Administered Adolescent Questionnaires Compared with a Paper Version in a Randomized Study. Journal of Adolescent Health. 2005;36(1):70. doi: 10.1016/j.jadohealth.2004.02.020. [DOI] [PubMed] [Google Scholar]
- Mangunkusumo RT, Duisterhout JS, De Graaff N, Maarsingh EJ, Koning de HJ, Raat H. Internet versus Paper Mode of Health and Health Behavior Questionnaires in Elementary Schools: Asthma and Fruit as Examples. Journal of School Health. 2006;76(2):80–6. doi: 10.1111/j.1746-1561.2006.00072.x. [DOI] [PubMed] [Google Scholar]
- McCabe SE, Boyd CJ, Young A, Crawford S, Pope D. Mode Effects for Collecting Alcohol and Tobacco Data among 3rd and 4th Grade Students: A Randomized Pilot Study of Web-Form versus Paper-Form Surveys. Addictive Behavior. 2005;30(4):663–71. doi: 10.1016/j.addbeh.2004.08.012. [DOI] [PubMed] [Google Scholar]
- Paperny DM, Aono JY, Lehman RM, Hammar SL, Risser J. Computer-Assisted Detection and Intervention in Adolescent High-Risk Health Behaviors. Journal of Pediatrics. 1990;116(8):456–62. doi: 10.1016/s0022-3476(05)82844-6. [DOI] [PubMed] [Google Scholar]
- Raat H, Landgraf JM, Bonsel GJ, Gemke R J B J, Essink Bot ML. Reliability and Validity of the Child Health Questionnaire-Child Form (CHQ-CF87) in a Dutch Adolescent Population. Quality of Life Research. 2002;11(6):575–81. doi: 10.1023/a:1016393311799. [DOI] [PubMed] [Google Scholar]
- Rosenberg M. Society and the Adolescent Self-Image. Princeton, NJ: Princeton University Press; 1965. [Google Scholar]
- Rothman KJ, Greenland S. Modern Epidemiology. Philadelphia: Lippincott-Raven Publishers; 1998. [Google Scholar]
- Serdula MK, Mokdad AH, Pamuk ER, Williamson DF, Byers T. Effects of Question Order on Estimates of the Prevalence of Attempted Weight Loss. American Journal of Epidemiology. 1995;142(1):64–7. doi: 10.1093/oxfordjournals.aje.a117546. [DOI] [PubMed] [Google Scholar]
- Tourangeau R, Smith TW. Asking Sensitive Questions: The Impact of Data Collection Mode, Question Format, and Question Context. Public Opinion Quarterly. 1996;60(2):275–304. [Google Scholar]
- Truman J, Robinson K, Evans AL, Smith D, Cunningham L, Millward R, Minnis H. The Strengths and Difficulties Questionnaire: A Pilot Study of a New Computer Version of the Self-Report Scale. European Child and Adolescent Psychiatry. 2003;12(1):9–14. doi: 10.1007/s00787-003-0303-9. [DOI] [PubMed] [Google Scholar]
- Turner CF, Ku L, Rogers SM, Lindberg LD, Pleck JH, Sonenstein FL. Adolescent Sexual Behavior, Drug Use, and Violence: Increased Reporting with Computer Survey Technology. Science. 1998;280(5365):867–73. doi: 10.1126/science.280.5365.867. [DOI] [PubMed] [Google Scholar]
- van der Linden FJ, Dijkman TA, Roeder P J B. Schoolgaande Jongeren, Hun Leefwereld en Zelfbeleving. Nijmegen, the Netherlands: Hoogveldt Instituut; 1983. [Google Scholar]
- van Widenfelt BM, Goedhart AW, Treffers P D A, Goodman R. Dutch Version of the Strengths and Difficulties Questionnaire (SDQ) European Child and Adolescent Psychiatry. 2003;12(6):281–9. doi: 10.1007/s00787-003-0341-3. [DOI] [PubMed] [Google Scholar]
- Vereecken CA, Maes L. Comparison of a Computer-Administered and Paper-and-Pencil-Administered Questionnaire on Health and Lifestyle Behaviors. Journal of Adolescent Health. 2006;38(4):426–32. doi: 10.1016/j.jadohealth.2004.10.010. [DOI] [PubMed] [Google Scholar]
- Wang YC, Lee CM, Lew-Ting CY, Hsiao CK, Chen DR, Chen WJ. Survey of Substance Use among High School Students in Taipei: Web-Based Questionnaire versus Paper-and-Pencil Questionnaire. Journal of Adolescent Health. 2005;37(4):289–95. doi: 10.1016/j.jadohealth.2005.03.017. [DOI] [PubMed] [Google Scholar]
- Webb PM, Zimet GD, Fortenberry JD, Blythe MJ. Comparability of a Computer-Assisted versus Written Method for Collecting Health Behavior Information from Adolescent Patients. Journal of Adolescent Health. 1999;24(6):383–8. doi: 10.1016/s1054-139x(99)00005-1. [DOI] [PubMed] [Google Scholar]
- Wright DL, Aquilino WS, Supple AJ. A Comparison of Computer-Assisted Paper-and-Pencil Self-Administered Questionnaires in a Survey on Smoking, Alcohol, and Drug Use. Public Opinion Quarterly. 1998;62(3):331–53. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.