Skip to main content
NCBI home page
Preventive Medicine Reports logoLink to Preventive Medicine Reports
. 2023 Oct 9;36:102462. doi: 10.1016/j.pmedr.2023.102462

Comparison of the responses of cross-sectional web- and paper-based surveys on lifestyle behaviors of Japanese adolescents

Yuichiro Otsuka a,⁎, Aya Kinjo b, Yoshitaka Kaneita a, Osamu Itani a, Yuki Kuwabara b, Ruriko Minobe c, Hitoshi Maesato c, Susumu Higuchi c, Hideyuki Kanda d, Hisashi Yoshimoto e, Maki Jike f, Hideaki Kasuga g, Teruna Ito h, Yoneatsu Osaki b
PMCID: PMC10583168  PMID: 37860161

Highlights

  • •

    We compared the value of paper- and web-based surveys among Japanese adolescents.

  • •

    Alcohol use and poor mental health were lower among the web (vs. paper) group.

  • •

    Short sleep duration and college plans were higher among the web (vs. paper) group.

  • •

    In Japan, paper-based surveys are currently more effective than web-based surveys.

  • •

    Whether behavioral differences are owing to selection bias or method is unclear.

Keywords: Adolescent, Japan, Response rate, Web-based survey, Paper-based survey, Smoking, Alcohol

Abstract

Objective

In Japan, paper-based surveys are currently more effective than web-based surveys. This study compared the response rates and lifestyle behaviors between paper- and web-based surveys conducted among Japanese junior and senior high schools.

Methods

In total, there were 42 and 64 junior and senior high schools, respectively, for the web-based surveys and 20 and 27 junior and senior high schools, respectively, for the paper-based surveys. The questionnaire covered lifestyle behaviors (e.g., alcohol consumption, smoking, exercise, eating, and sleep status), mental health, and plans to attend college. School- and student-level response rates by survey method were assessed, and so was the effect on the reporting of each lifestyle behavior using logistic regression models.

Results

The school response rates were 16.0% and 38.3% for web- and paper-based surveys, respectively. The student response rates were 88.7% and 77.2%, respectively. The web-based group had significantly more female participants and lower response rates for higher grades in senior high schools. The odds of lifetime and current alcohol consumption and poor mental health were lower, whereas those of lifetime use of conventional cigarettes, shorter sleep duration, and plans to continue attending college were higher among web-based (vs. paper-based) participants.

Conclusions

The school response rate was poor in the web-based survey. However, whether the differences in lifestyle behavior are attributed to selection bias or the survey method remains unclear.

1. Introduction

School-based surveys are used globally to evaluate adolescents’ health conditions, lifestyles, and risk behaviors (Kim et al., 2016, Smit et al., 2002). Paper-based surveys have long been applied in school surveys. However, recently, web-based surveys have replaced paper-based surveys in small but growing populations in developed countries, such as the Monitoring the Future project in the US and the Korea Youth Risk Behavior Web-based Survey (Kim et al., 2016, Miech et al., 2018, Miech et al., 2021).

Web-based surveys have several advantages over paper-based ones, including convenience, low cost, faster response, design flexibility, and fewer missing values and data entry errors (Denniston et al., 2010, Hamann et al., 2016, Kim et al., 2016, Revilla and Ochoa, 2015, Zeiler et al., 2021). However, web-based surveys show lower response rates and require digital devices, Internet access, and technological proficiency (Akl et al., 2005, Couper, 2000, Eaton et al., 2010, Fricker and Schonlau, 2002). Recent meta-analyses and review studies have demonstrated that web- and paper-based surveys were generally comparable (Alfonsson et al., 2014, Campbell et al., 2015, Gwaltney et al., 2008). However, the results of reports on lifestyle and risk behaviors among adolescents across survey methods remain controversial. Some studies have reported no differences between these methods regarding health-related behaviors (Akl et al., 2005, Baier, 2018, Colasante et al., 2019, Eaton et al., 2010, Mangunkusumo et al., 2005, Miech et al., 2021). A cross-sectional study of Italian adolescents showed that health risk behavior—drug, alcohol, and tobacco use—prevalence was generally equivalent between paper-based and computerized surveys (Colasante et al., 2019). Others have demonstrated that the percentages of sensitive information were higher in web- than in paper-based surveys (Turner et al., 1998, Wang et al., 2005) and that participants in paper-based surveys reported more restraint than those in web-based surveys (Šmigelskas et al., 2019).

In Japan, nationwide surveys of lifestyle behaviors targeting adolescents have mostly been conducted using paper-based methods. Since 1996, the Japanese Ministry of Health, Labour and Welfare (Ministry of Health, 2022) has conducted the nationwide Lifestyle Survey of Adolescents every three or four years to collect comparable data on alcohol and tobacco consumption among junior and senior high school students and guide future health policies. However, no representative survey of adolescents has compared web- and paper-based surveys. The impact of the COVID-19 pandemic in 2020 led to school closures across Japan and demanded the rapid development of information and communication technology (ICT) for home-based learning, precipitating an urgent need to develop new survey methods for the post-COVID-19 era. Additionally, from 1996 to 2017, the response rates for the Lifestyle Survey of Adolescents gradually declined from 65.6 % to 49 % and 67.0 % to 64.0 % among junior and senior high school students, respectively (Fujii et al., 2021). Therefore, the 2021 Lifestyle Survey of Adolescents developed a web-based method and used it along with the conventional paper-based method. Considering the past evidence, we hypothesized that there would be no significant difference between survey methods in adolescents’ responses regarding lifestyle behaviors (Ministry of Health, 2022).

2. Material and methods

This school-based survey study used a single stratified single-stage standard cluster sampling procedure (Nulty, 2008). Proportional probability sampling was used, wherein the sampling probability changed in proportion to the number of students in the school. This method involved dividing Japan into regional blocks and randomly selecting schools from each block (Osaki and Minowa, 1996).

2.1. Participants

The target population was restricted to 7–12-grade students in junior and senior high schools across Japan, which were selected using the National School Directory (Fig. 1). All students enrolled in the sampled schools were included in the target population. Owing to survey budget limitations, the number of schools varied in each survey, causing sample size changes across surveys. We randomly sampled 91 junior and 62 senior high schools. Of these, 64 junior and 42 senior high schools and 27 junior and 20 senior high schools were assigned to web- and paper-based methods, respectively, by computer-generated randomization. We used the 2:1 allocation ratio because of lower response rates for web-based (vs. paper-based) surveys in past research (Daikeler et al., 2020, Nulty, 2008).

Fig. 1.

Fig. 1

Open in a new tab

Flow chart of data collection.

2.2. Procedure

This survey—conducted between May and August 2021—was anonymous. Schools could opt to receive a feedback report that included school-level and overall results. Informed written consent was obtained from all participants and their parents/guardians. This study was approved by the ethics review board of Tottori University (No. 20A099) and conducted in accordance with the Declaration of Helsinki.

A letter was sent to the principal of each selected school requesting cooperation, along with a questionnaire form, an explanation of the survey for parents/guardians, and an envelope for each enrolled student. Then, the class teachers informed students of study details and privacy protection and indicated that the completed questionnaires would not be seen by teachers. The teachers explained the nature of confidentiality and voluntary participation to all students. The paper-based questionnaire takes approximately 20–40 min to answer, and is generally conducted during homeroom time under the supervision of a teacher; students are not allowed to take the questionnaire home. The completed paper-based questionnaires were returned to the working group in sealed envelopes.

For the web-based questionnaires, the survey implementation procedure contained log-in information, including the URL, QR code, and access key. The students accessed the URL via their computers/tablets in the school. The working group used anonymized questionnaires to ensure participant confidentiality and privacy. Response data were stored on a secure server to which only research group representatives had access.

2.3. Measures

The questionnaire included the following categories: lifestyle, including alcohol consumption, smoking, exercise, eating behavior, and sleep status, including insomnia symptoms and sleep duration; mental health status; demographic variables, including sex, grade, type of school (junior/senior high school), and plans to attend college (yes/no).

The web-based version was designed to be, visually, as similar as possible to the paper-based version. In the web-based version, the system did not allow responses to be sent until all questions had been answered. This is because Colasante et al.’s (2019) study, mentioned in the Introduction, showed more incomplete questionnaires and less valid responses in web-based surveys (vs. paper-based) when blank answers were accepted. It was not possible to confirm questionnaire completeness in the paper-based version.

The following variables had a multiple-choice format: lifetime alcohol and tobacco consumption (having consumed alcohol or tobacco at least once), current consumption (having consumed alcohol or tobacco at least once during the past 30 days), tobacco categories (conventional cigarettes, heated tobacco products, and e-cigarettes) (Kuwabara et al., 2020); exercise (the number of days participants engaged in physical activity for 30 min or longer, except for physical education class, during the past 30 days—0, 1–2, 3–5, 6–9, 10–19, and 20–29 days or daily—with 10 days or more considered as habitual exercise); eating behaviors—skipping breakfast (rarely eating breakfast on weekdays) and skipping vegetables (rarely eating vegetables); insomnia symptoms (frequent or constant difficulty initiating and maintaining sleep, and waking up early at least once in the past 30 days) (Otsuka et al., 2021), and sleep duration (<5h, 5–6 h, 6–7 h, 7–8 h, 8–9 h, and ≥ 9 h—with < 6 h considered as “short sleep duration”).

Mental health was assessed using the two-item General Health Questionnaire owing to the limited space available in the survey questionnaire. Each item was rated from 0 to 1, with a total score ranging from 0 to 2. Scores of 1 or 2 were regarded as poor mental health; this scale’s specificity and sensitivity were 87.0 % and 85.1 %, respectively (Suzuki et al., 2011).

2.4. School information

This study collected the following information using the National School Directory and from each school’s homepage: school type (public/private), school size (small/medium/large, separating junior high and high schools), whether junior high schools or senior high schools, regions (Hokkaido, Tohoku, Kanto, Chubu, Kinki, Chugoku, Shikoku, Kyushu-Okinawa), city size (large cities—20 government-decreed cities and 23 wards in Tokyo—or other cities/towns), and college-going rate—only for senior high schools (preparatory schools, over 80 %; career-diverse schools, 20–79 %; and non-preparatory schools, less than 20 %).

2.5. Statistical analyses

First, school- and student-level response rates were calculated in the total group and sub-groups by survey method. Second, school- and student-level differences in response or non-response to school information were examined using Fisher’s exact tests. Third, participants’ demographic characteristics extracted with each survey method were examined through chi-squared tests. Fourth, lifestyle behaviors, mental health, and plans to attend college were compared by performing chi-squared tests. Fifth, survey method effects on the reporting of each lifestyle behavior were examined using separate logistic regression models. Each model was adjusted for sex, age, school, and school type (junior or senior). Significance was set at p <.05. This study adopted pairwise deletion to handle missing data. SPSS 22.0 (IBM) was used for all data analyses.

3. Results

In total, 22 paper- and 4 web-based questionnaires were excluded owing to missing/inconsistent data. In the web-based version, inconsistent data were responses recorded outside of school hours. In the paper-based version, answers that did not include grade/sex were defined as missing, and inconsistent data were questionnaires with inconsistent answers (e.g., containing details of alcohol consumption and alcohol beverage type for those below the legal drinking age; Table 1).

Table 1.

Detailed information of the 153 Japanese schools that participated in the 2021 Lifestyle Survey of Adolescents.

School type Method Number of invited schools Number of participating schools School response rate Number of students in the schools that were invited Number of students in the schools that accepted the invitation Number of students who could respond to the survey Number of participating students Students’ response rate
Junior high school Paper 27 9 33.3 14,951 5,029 5,029 4,626 92.0
Web 64 9 14.1 30,274 4,127 4,127 3,648 88.4
Total 91 18 19.8 45,225 9,156 9,156 8,274 90.4
Senior high school Paper 20 9 45.0 15,055 7,350 7,350 4,935 67.1
Web 42 8 19.0 35,268 4,364 2,963 2,643 89.2
Total 62 17 27.4 50,323 11,714 10,313 7,578 73.5
Total Paper 47 18 38.3 30,006 12,379 12,379 9,561 77.2
Web 106 17 16.0 65,542 8,491 7,090a 6,291 88.7
Total 153 35 22.9 95,548 20,870 19,469 15,852 81.4
Open in a new tab

Note.aThe principals of three schools that participated in the web-based survey restricted student participation to 10-grade students.

School location was a significant factor in school-level response rate, and there was a tendency for public and medium-sized schools and the Tohoku and Chubu regions to have high participation rates in both methods—with non-significant differences. In the Kanto region, response rates were low and high for the web- and paper-based versions, respectively. In the Hokkaido and Kyushu-Okinawa regions, both response rates tended to be low. Regarding student-level response rates, only high school students had a significantly lower response rate than junior high school students. There were no significant factors related to the response rates regarding school information, such as school management (Table 2; Supplemental Table 1).

Table 2.

Factors associated with the school-level response rate of the 153 Japanese schools that participated in the 2021 Lifestyle Survey of Adolescents.

Total (N = 153)
 p Web (n = 106)
 p Paper (n = 47)
 p
Target Response, % Target Response, % Target Response, %
School type
Public 125 32 25.6 0.134 86 15 17.4 0.519 39 17 43.6 0.130
Private 28 3 10.7 20 2 10.0 8 1 12.5
School size
Small 50 10 20.0 0.089 23 2 8.7 0.449 7 2 28.6 0.879
Medium 51 17 33.3 23 5 21.7 7 3 42.9
Large 52 8 15.4 18 2 11.1 13 4 30.8
Combined junior and high school
No 128 29 22.7 1.000 87 12 13.8 0.183 41 17 41.5 0.384
Yes 25 6 24.0 19 5 26.3 6 1 16.7
Region
Hokkaido 6 0 0.0 0.346 4 0 0.0 0.157 2 0 0.0 0.820
Tohoku 12 5 41.7 8 3 37.5 4 2 50.0
Kanto 44 9 20.5 31 2 6.5 13 7 53.8
Chubu 29 10 34.5 19 6 31.6 10 4 40.0
Kinki 27 6 22.2 19 4 21.1 8 2 25.0
Chugoku 10 2 20.0 8 1 12.5 2 1 50.0
Shikoku 6 1 16.7 3 0 0.0 3 1 33.3
Kyushu and Okinawa 19 2 10.5 14 1 7.1 5 1 20.0
Size of the city in which the school is located
Twenty-one large cities 37 3 8.1 0.024 29 1 3.4 0.060 8 2 25.0 0.493
City 105 30 28.6 71 15 21.1 34 15 44.1
Municipality 11 2 18.2 6 1 16.7 5 1 20.0
Open in a new tab

Note. p-values were calculated by performing Fisher’s exact test.

The web-based group (vs. paper-based) had significantly more female students and less 11- and 12-grade students (Table 3).

Table 3.

Characteristics of the 15,832 adolescent participants from the 153 Japanese schools that participated in the 2021 Lifestyle Survey of Adolescents.

Participants Paper-based (%) Web-based (%) χ2 p
N = 15,832 n = 9,545 n = 6,287
Sex
Male 49.4 51.6 46.0 46.54 < 0.001
Female 50.6 48.4 54.0
Grade
Junior high school 366.69 < 0.001
7 17.7 16.0 20.2
8 17.6 16.0 20.2
9 16.7 16.0 17.6
Unknown 0.2 0.4 0.0
Senior high school
10 18.7 17.2 20.9
11 14.5 17.0 10.8
12 14.5 17.3 10.3
Unknown 0.1 0.1 0.0
Open in a new tab

Note. p-values were calculated by performing χ2 tests.

The web-based group (vs. paper-based) showed a significantly higher prevalence of shorter sleep duration and a lower prevalence of lifetime and current alcohol consumption, current use of conventional cigarettes, exercise habits, skipping vegetables, insomnia, and poor mental health. Similarly, the web-based group (vs. paper-based) showed significantly higher odds ratios (ORs) for lifetime use of conventional cigarettes, shorter sleep duration, and having plans to attend college; and significantly lower ORs for lifetime alcohol consumption, current alcohol consumption, habitual exercise, skipping vegetables, insomnia, and poor mental health (Table 4).

Table 4.

Prevalence of lifestyle behaviors by survey method and adjusted odds ratios of lifestyle behaviors of the 15,832 adolescent participants from the 153 Japanese schools that participated in the 2021 Lifestyle Survey of Adolescents.

Paper-based (%) Web-based χ2 pa ORb 95 % CI Wald p
n = 9,539 n = 6,287
Lifetime alcohol consumption 9,441 11.8 6,287 9.8 15.66 < 0.001 0.86 0.78 0.96 7.74 0.005
Current alcohol consumption 9,508 2.5 6,287 1.7 8.85 0.002 0.77 0.61 0.98 4.62 0.032
Lifetime use of conventional cigarettes 9,458 2.1 6,287 2.4 1.52 0.218 1.26 1.01 1.56 4.24 0.039
Current use of conventional cigarettes 9,465 0.5 6,287 0.3 5.16 0.023 0.61 0.34 1.09 2.84 0.092
Lifetime use of HTPs 9,500 1.2 6,287 1.1 0.25 0.614 1.01 0.74 1.37 0.00 0.972
Current use of HTPs 9,460 0.5 6,287 0.3 1.76 0.184 0.77 0.45 1.32 0.90 0.344
Lifetime use of e-cigarettes 9,487 1.2 6,287 1.1 0.92 0.337 0.93 0.69 1.27 0.20 0.652
Current use of e-cigarettes 9,470 0.4 6,287 0.3 2.13 0.144 0.77 0.43 1.37 0.79 0.374
Exercise habits 9,455 59.7 6,287 57.8 6.06 0.014 0.89 0.83 0.95 11.88 0.001
Skipping breakfast (weekday) 9,407 4.7 6,287 4.3 1.57 0.210 0.98 0.84 1.15 0.08 0.784
Skipping vegetables 9,440 2.3 6,287 1.6 8.88 0.003 0.89 0.83 0.95 11.85 0.001
Insomnia 9,479 21.4 6,287 20 4.50 0.034 0.92 0.85 0.99 4.22 0.040
Short sleep duration 9,474 25.9 6,287 32.6 83.23 < 0.001 1.56 1.45 1.68 139.00 < 0.001
Poor mental health 9,449 56.9 6,287 49.5 83.12 < 0.001 0.74 0.69 0.79 83.30 < 0.001
Plans to continue attending college 9,480 41.6 6,287 41.4 0.05 0.823 1.26 1.17 1.35 40.95 < 0.001
Open in a new tab

Note. HTPs: heated tobacco products; OR: odds ratio.

a

p-values were calculated by performing the χ2 test.

b

ORs were adjusted for age, sex, school, and school type (junior/senior).

4. Discussion

This is the first nationwide survey to compare the response rates and differences in lifestyle behaviors between web- and paper-based surveys among Japanese adolescents. School-level response rates were lower for web-based surveys (vs. paper-based); student-level response rates were similar for both methods. Among web-based participants (vs. paper-based), the odds of lifetime and current alcohol consumption, exercise habits, and poor mental health were lower; the odds were higher for lifetime use of conventional cigarettes, short sleep duration, and plans to continue attending college.

Regarding school response rates, the paper- and web-based surveys combined had a lower response rate (22.9 %) than the previous paper-based Lifestyle Survey of Adolescents in Japan (approximately 50–70 %) (Otsuka et al., 2021). Several factors may explain this decline: schools’ strict measures to safeguard school and personal information; law differences by survey method, which mandate consent from both student and parent/guardian; decline in the prevalence of adolescent smoking and alcohol consumption (Otsuka et al., 2021), which might have made these behaviors less important for school principals; lack of time to respond to surveys because of school closures owing to the pandemic; inadequate ICT environment in Japanese schools. Previous studies associated response rate differences with an inadequate ICT environment (i.e., lack of personalized computers/tablets) in schools (Šmigelskas et al., 2019, Yetter and Capaccioli, 2010). More than 90 % of students in China, Denmark, and Singapore were enrolled in schools with an effective online learning support platform (Ikeda, 2020). However, in Japan, less than 30 % of students had access to such a platform (Ikeda, 2020), and the class time spent using digital devices was the shortest among Organisation for Economic Co-operation and Development (OECD) member countries in 2018 (PISA, 2019). Moreover, the proportion of teachers with the necessary skills to effectively support student learning using digital technologies was 35 % in Japan (OECD average: 67 %; PISA, 2019). In the web-based survey group, there were fewer 11- and 12-grade participants because some schools in this group only permitted 10-grade students to participate. This is possibly owing to the limited ICT environment and curricula for higher grades, which focus on entrance examination preparation for higher education or job search. Furthermore, schools may have found providing instructions for web-based surveys inconvenient.

Student-level response rates (vs. school-level) were higher for both the web- and paper-based versions. The response rate was high among junior high school and high school students for the paper- and web-based surveys, respectively. This may have been because some schools limited participation to 10-grade students and most high school students are familiar with Internet surfing—corroborating evidence that young people tend to prefer web-based surveys (Miller et al., 2002, Shih and Fan, 2008). Student-level results indicated that only school type (junior/senior) was associated with non-response. Non-response may occur for two reasons (Monseur, 2005): a student within a participating school fails to complete the questionnaire (e.g., absence) or lack of parental consent. Systematic reviews showed that effective strategies to increase response rates include monetary incentives, short questionnaires (approximately 10 min), personalized questionnaires and letters, contacting participants before sending the questionnaires (email prenotification/invitation and two reminders), and semi-automatic log-in (Bryan et al., 2022, Edwards et al., 2002). Although researchers cannot obtain personal/contact information in school-based settings, creating shorter questionnaires may be feasible. These findings imply that the biggest challenge for survey administrators is convincing schools to participate, as high student participation rates may usually be easily achieved. Government agencies must provide more support and ask schools to cooperate in surveys. Additionally, asking the principal about survey responses in advance and looking for another school if the principal refuses may be useful (Bryan et al., 2022).

Some school-based surveys showed inconsistent results regarding differences in reported lifestyle behaviors between web- and paper-based surveys (Akl et al., 2005, Colasante et al., 2019, Eaton et al., 2010, Mangunkusumo et al., 2005, Miech et al., 2021, Monseur, 2005, Turner et al., 1998, Wang et al., 2005), and others demonstrated generally equivalent comparability of web- and paper-based surveys across methods (Alfonsson et al., 2014, Campbell et al., 2015, Gwaltney et al., 2008). This study’s multivariable regression analysis showed that paper-based survey participants (vs. web-based) tended to drink more alcohol, experience more insomnia, and have poorer mental health. However, web-based survey participants (vs. paper-based) tended to have less habitual exercise, less sleep, and plans to attend college. The multiple differences in prevalence for each lifestyle behavior may be a result of survey method differences. First, sample properties may have differed; while the web-based survey featured three high schools with a university entrance rate of 80 % or higher, there were no such schools in the paper-based survey. As in other Asian countries (Bapat et al., 2017, Chen et al., 2014), lower sleep duration and exercise habits may be a result of increasing academic demand and pressure on higher education students. Second, questionnaire style differences between survey methods may have resulted in different responses. In the web-based survey, the questionnaire could not be sent until all questions had been answered, whereas response deficits could occur in the paper-based survey. Therefore, differences in answers may have occurred owing to web-based survey participants being forced to answer questions they did not wish to answer. However, a previous study that conducted paper- and web-based surveys for the same participants found no significant differences in the responses (Braekman et al., 2018). Therefore, future studies could confirm response differences by method with the same participants in a pilot survey. Third, survey method differences may influence the effect of some normative and cognitive mechanisms (Dillman, 2000). Normative mechanisms encompass factors like social desirability (respondents choose socially acceptable responses), acquiescence (tendency to agree rather than disagree), question-order effects (respondents seek consistency with previous answers when responding to later questions), and primacy or recency effects (preference for selecting the first or last offered response). Cognitive mechanisms pertain to whether survey respondents receive uniform stimuli through the survey methods. These encompass differences in visual and neurological processing preferences.

Furthermore, among junior high school students, the prevalence of smoking-related behaviors in the 2021 survey was similar to that in the 2017 survey, which used similar paper-based methods (Kuwabara et al., 2020). The prevalence of current alcohol consumption was also less than half of that in 2017 (Otsuka et al., 2021). Additionally, the prevalence of skipping breakfast was 14.7 % in 2017—significantly lower than the current result (Otsuka et al., 2021). A previous nationwide school-based study reported a decline in the prevalence of adolescent smoking and alcohol consumption (Otsuka et al., 2021). However, considering sleep-related behaviors, the prevalence of short sleep duration and insomnia symptoms increased since 2017 (Otsuka et al., 2021). The percentage of those with college plans was almost the same as in 2017 (Otsuka et al., 2021). These past studies have been age-adjusted and weighted. Hence, current and past results could not be compared. However, as the present findings suggested that students display health-conscious behaviors, whether differences can be attributed to the participating population or method of administration remains uncertain.

While our study provides valuable insights into web- and paper-based survey differences among Japanese adolescents, it is essential to consider the broader context when interpreting our findings, and a major consideration relates to generalizability. The low school-level response rate in our sample is an important factor in this regard, and schools with varying ICT environments and policies regarding survey participation may yield different results. Thus, generalizations should be made with caution. To enhance the external validity of our findings, researchers are urged to replicate our research with larger and more diverse adolescent samples. Comparative studies conducted in other countries could also provide valuable insights into the generalizability of our findings at the global scale. Systematic reviews and meta-analyses comparing web- and paper-based surveys targeting adolescents are also warranted; for example, there are many studies on adolescent sleep problems, and it may be useful to examine these problems by survey method.

Web-based surveys are a desirable alternative to paper-based surveys. The present survey had few missing values and data entry errors and had easy data transmission for analysis. However, to transition from paper- to web-based surveys, the ICT environment in schools in Japan must be improved. Until all junior and senior high schools in Japan are adequately equipped, schools should be provided with a choice between web- and paper-based surveys, to avoid selection bias. In both methods, permission from the principals is necessary, and researchers must have sufficient communication with them to ensure their cooperation.

This study had several limitations. First, the survey response rate among schools was low, and selection bias should be considered when comparing web- and paper-based surveys. However, a cross-sectional study in the US found that the estimates obtained using a national survey of college student engagement remained reliable, even with a 5–10 % response rate and a sample size of over 500 (Fosnacht et al., 2017). Thus, low response rates do not necessarily induce bias. Nonetheless, future studies should conduct not only paper-based but also web-based surveys to continue monitoring adolescents’ lifestyle behaviors while keeping in mind the issue of selection bias. Second, the present results may have underestimated the unfavorable lifestyle behaviors of adolescents, compared with previous results (i.e., healthy-user bias) (Otsuka et al., 2021). Reportedly, non-response is associated with a lower socioeconomic status, unfavorable lifestyle behaviors, and psychopathological problems (Brownstein et al., 2011, Dunne et al., 1997). A cross-sectional survey of Dutch adolescents showed that non-response contributed to underestimating various health risks (Cheung et al., 2017). In web-based surveys, some school principals may be confident that their students’ lifestyle behaviors are better than those of students from other schools and/or educate students about favorable lifestyle behaviors. Third, the same participants did not respond to both the paper- and web-based questionnaires. Therefore, whether the survey method affected the difference in response contents remains unclear. A more optimal design would be providing both surveys to the same participants/schools with a small interval period. However, this might be difficult, costly, and impractical. Future studies need to increase the response rate to ensure participant homogeneity across survey methods.

5. Conclusion

These findings suggest that conventional paper-based surveys are effective and can reduce selection bias more effectively (vs. web-based) in Japanese schools. These results could be useful in developing more effective survey strategies. However, the low school-level response rate is an emergent issue that undermines nationally representative surveys’ validity and reliability. Therefore, strategies to encourage responses must be considered.

CRediT authorship contribution statement

Yuichiro Otsuka: Writing – original draft, Formal analysis. Aya Kinjo: Supervision, Methodology, Conceptualization. Yoshitaka Kaneita: Conceptualization, Methodology. Osamu Itani: Software, Validation. Yuki Kuwabara: Data curation. Ruriko Minobe: Methodology. Hitoshi Maesato: Methodology, Investigation. Susumu Higuchi: Methodology, Investigation. Hideyuki Kanda: Resources, Conceptualization. Hisashi Yoshimoto: Resources. Maki Jike: Resources. Hideaki Kasuga: Investigation. Teruna Ito: Investigation. Yoneatsu Osaki: Methodology, Project administration, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

We thank Editage (www.editage.jp) for English language editing.

Funding

This work was supported by a grant for comprehensive research on lifestyle-related diseases including cardiovascular diseases and diabetes mellitus by the Ministry of Health, Labour and Welfare, Health Science Research Fund in Japan [grant number 20FA1003]. The funding body did not have any role in the design of the study; collection, analysis, and interpretation of data; in the writing of the manuscript; in the decision to submit the article for publication.

Data statement

The datasets analyzed in this survey is not publicly available.

Footnotes

Appendix A

Supplementary data to this article can be found online at https://doi.org/10.1016/j.pmedr.2023.102462.

Appendix A. Supplementary data

The following are the Supplementary data to this article:

Supplementary data 1
mmc1.docx (22.4KB, docx)

Data availability

The authors do not have permission to share data.

References

  1. Akl E.A., Maroun N., Klocke R.A., Montori V., Schünemann H.J. Electronic mail was not better than postal mail for surveying residents and faculty. J. Clin. Epidemiol. 2005;58:425–429. doi: 10.1016/j.jclinepi.2004.10.006. [DOI] [PubMed] [Google Scholar]
  2. Alfonsson S., Maathz P., Hursti T. Interformat reliability of digital psychiatric self-report questionnaires: A systematic review. J. Med. Internet Res. 2014;16:e268. doi: 10.2196/jmir.3395. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Baier D. Computer-assisted versus paper-and-pencil self-report delinquency surveys: Results of an experimental study. Eur. J. Criminol. 2018;15:385–402. doi: 10.1177/1477370817743482. [DOI] [Google Scholar]
  4. Bapat R., van Geel M., Vedder P. Socio-economic status, time spending, and sleep duration in Indian children and adolescents. J. Child Fam. Stud. 2017;26:80–87. doi: 10.1007/s10826-016-0557-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Braekman E., Berete F., Charafeddine R., Demarest S., Drieskens S., Gisle L., Molenberghs G., Tafforeau J., Van der Heyden J., Van Hal G., Alamian A. Measurement agreement of the self-administered questionnaire of the Belgian Health Interview Survey: Paper-and-pencil versus web-based mode. PLoS One. 2018;13(5) doi: 10.1371/journal.pone.0197434. e0197434. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Brownstein N., Kalsbeek W.D., Tabor J., Entzel P., Daza E., Harris K.M. Carolina Population Center, University of North Carolina; Chapel Hill: 2011. Non-response in Wave IV of the national longitudinal study of adolescent health. https://doi.org/10.17615/C6ZQ0Q. [Google Scholar]
  7. Bryan L.N., Smith-Grant J., Brener N., Kilmer G., Lo A., Queen B., Underwood J.M. Electronic versus paper and pencil survey administration mode comparison: 2019 Youth Risk Behavior Survey. J. Sch. Health. 2022;92:804–811. doi: 10.1111/josh.13184. [DOI] [PubMed] [Google Scholar]
  8. Campbell N., Ali F., Finlay A.Y., Salek S.S. Equivalence of electronic and paper-based patient-reported outcome measures. Qual. Life Res. 2015;24:1949–1961. doi: 10.1007/s11136-015-0937-3. [DOI] [PubMed] [Google Scholar]
  9. Chen T., Wu Z., Shen Z., Zhang J., Shen X., Li S. Sleep duration in Chinese adolescents: Biological, environmental, and behavioral predictors. Sleep Med. 2014;15:1345–1353. doi: 10.1016/j.sleep.2014.05.018. [DOI] [PubMed] [Google Scholar]
  10. Cheung K.L., Ten Klooster P.M., Smit C., de Vries H., Pieterse M.E. The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health. BMC Public Health. 2017;17:276. doi: 10.1186/s12889-017-4189-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Colasante E., Benedetti E., Fortunato L., Scalese M., Potente R., Cutilli A., Molinaro S., Zeeb H. Paper-and-pencil versus computerized administration mode: Comparison of data quality and risk behavior prevalence estimates in the European school Survey Project on Alcohol and other Drugs (ESPAD) PLoS One. 2019;14(11) doi: 10.1371/journal.pone.0225140. e0225140. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Couper M.P. Web surveys: A review of issues and approaches. Public Opin. Q. 2000;64:464–494. doi: 10.1086/318641. [DOI] [PubMed] [Google Scholar]
  13. Daikeler J., Bošnjak M., Lozar Manfreda K. Web versus other survey modes: An updated and extended meta-analysis comparing response rates. J. Surv. Stat. Methodol. 2020;8:513–539. doi: 10.1093/jssam/smz008. [DOI] [Google Scholar]
  14. Denniston M.M., Brener N.D., Kann L., Eaton D.K., McManus T., Kyle T.M., Roberts A.M., Flint K.H., Ross J.G. Comparison of paper-and-pencil versus Web administration of the Youth Risk Behavior Survey (YRBS): Participation, data quality, and perceived privacy and anonymity. Comput. Hum. Behav. 2010;26:1054–1060. doi: 10.1016/j.chb.2010.03.006. [DOI] [PubMed] [Google Scholar]
  15. Dillman D.A. John Wiley & Sons; New York: 2000. Mail and Internet Surveys: The Tailored Design Method. [Google Scholar]
  16. Dunne M.P., Martin N.G., Bailey J.M., Heath A.C., Bucholz K.K., Madden P.A., Statham D.J. Participation bias in a sexuality survey: Psychological and behavioural characteristics of responders and non-responders. Int. J. Epidemiol. 1997;26:844–854. doi: 10.1093/ije/26.4.844. [DOI] [PubMed] [Google Scholar]
  17. Eaton D.K., Brener N.D., Kann L., Denniston M.M., McManus T., Kyle T.M., Roberts A.M., Flint K.H., Ross J.G. Comparison of paper-and-pencil versus web administration of the Youth Risk Behavior Survey (YRBS): Risk behavior prevalence estimates. Eval. Rev. 2010;34:137–153. doi: 10.1177/0193841X10362491. [DOI] [PubMed] [Google Scholar]
  18. Edwards P., Roberts I., Clarke M., Diguiseppi C., Pratap S., Wentz R., Kwan I. Increasing response rates to postal questionnaires: Systematic review. BMJ. 2002;324:1183. doi: 10.1136/bmj.324.7347.1183. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Fosnacht K., Sarraf S., Howe E., Peck L.K. How important are high response rates for college surveys? Rev. Higher Educ. 2017;40:245–265. doi: 10.1353/rhe.2017.0003. [DOI] [Google Scholar]
  20. Fricker R.D., Schonlau M. Advantages and disadvantages of internet research surveys: Evidence from the literature. Field Methods. 2002;14:347–367. doi: 10.1177/152582202237725. [DOI] [Google Scholar]
  21. Fujii M., Kuwabara Y., Kinjo A., Imamoto A., Jike M., Otsuka Y., Itani O., Kaneita Y., Minobe R., Maesato H., Higuchi S., Yoshimoto H., Kanda H., Osaki Y. Trends in the co-use of alcohol and tobacco among Japanese adolescents: Periodical nationwide cross-sectional surveys 1996–2017. BMJ Open. 2021;11(8) doi: 10.1136/bmjopen-2020-045063. e045063. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Gwaltney C.J., Shields A.L., Shiffman S. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value Health. 2008;11:322–333. doi: 10.1111/j.1524-4733.2007.00231.x. [DOI] [PubMed] [Google Scholar]
  23. Hamann C., Schultze-Lutter F., Tarokh L. Web-based assessment of mental well-being in early adolescence: A reliability study. J. Med. Internet Res. 2016;18:e138. doi: 10.2196/jmir.5482. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Ikeda M. Were Schools Equipped to Teach – And Were Students Ready to Learn – Remotely? Paris: PISA in Focus 108. OECD Publishing. 2020 doi: 10.1787/4bcd7938-en. [DOI] [Google Scholar]
  25. Kim Y., Choi S., Chun C., Park S., Khang Y.H., Oh K. Data resource profile: The Korea youth risk behavior web-based survey (KYRBS) Int. J. Epidemiol. 2016;45 doi: 10.1093/ije/dyw070. 1076-1076e. [DOI] [PubMed] [Google Scholar]
  26. Kuwabara Y., Kinjo A., Fujii M., Imamoto A., Osaki Y., Jike M., Otsuka Y., Itani O., Kaneita Y., Minobe R., Maezato H., Higuchi S., Yoshimoto H., Kanda H. Heat-not-burn tobacco, electronic cigarettes, and combustible cigarette use among Japanese adolescents: A nationwide population survey 2017. BMC Public Health. 2020;20:741. doi: 10.1186/s12889-020-08916-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Mangunkusumo R.T., Moorman P.W., Van Den Berg-de Ruiter A.E., Van Der Lei J., De Koning H.J., Raat H. Internet-administered adolescent health questionnaires compared with a paper version in a randomized study. J. Adolesc. Health. 2005;36:70.e1–70.e6. doi: 10.1016/j.jadohealth.2004.02.020. [DOI] [PubMed] [Google Scholar]
  28. Miech R.A., Couper M.P., Heeringa S.G., Patrick M.E. The impact of survey mode on US national estimates of adolescent drug prevalence: Results from a randomized controlled study. Addiction. 2021;116:1144–1151. doi: 10.1111/add.15249. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. R.A. Miech L.D. Johnston P.M. O’Malley J.G. Bachman J.E. Schulenberg M.E. Patrick Monitoring the Future National Survey Results on Drug Use, 1975–2017: I 2018 Institute for Social Research, University of Michigan, Ann Arbor Secondary School Students 10.3998/2027.42/146530.
  30. Miller E.T., Neal D.J., Roberts L.J., Baer J.S., Cressler S.O., Metrik J., Marlatt G.A. Test-retest reliability of alcohol measures: Is there a difference between Internet-based assessment and traditional methods? Psychol. Addict. Behav. 2002;16:56–63. doi: 10.1037/0893-164x.16.1.56. [DOI] [PubMed] [Google Scholar]
  31. Ministry of Health, Labour and Welfare, Japan. Healthy Japan 21. https://www.mhlw.go.jp/stf/seisakunitsuite/bunya/kenkou_iryou/kenkou/kenkounippon21.html (accessed 22 October 2022).
  32. Monseur C. An exploratory alternative approach for student non response weight adjustment. Stud. Educ. Eval. 2005;31:129–144. doi: 10.1016/j.stueduc.2005.05.006. [DOI] [Google Scholar]
  33. Nulty D.D. The adequacy of response rates to online and paper surveys: What can be done? Assess. Eval. Higher Educ. 2008;33:301–314. doi: 10.1080/02602930701293231. [DOI] [Google Scholar]
  34. Osaki Y., Minowa M. Cigarette smoking among junior and senior high school students in Japan. J. Adolesc. Health. 1996;18:59–65. doi: 10.1016/1054-139X(95)00110-E. [DOI] [PubMed] [Google Scholar]
  35. Otsuka Y., Kaneita Y., Spira A.P., Mojtabai R., Itani O., Jike M., Higuchi S., Kanda H., Kuwabara Y., Kinjo A., Osaki Y. Trends in sleep problems and patterns among Japanese adolescents: 2004 to 2017. Lancet Reg. Health West. Pac. 2021;9 doi: 10.1016/j.lanwpc.2021.100107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Pisa . OECD Publishing; 2019. Insights and Interpretations. [Google Scholar]
  37. Revilla M., Ochoa C. What are the links in a web survey among response time, quality, and auto-evaluation of the efforts done? Soc. Sci. Comput. Rev. 2015;33:97–114. doi: 10.1177/0894439314531214. [DOI] [Google Scholar]
  38. Shih T.-H., Fan X. Comparing response rates from web and mail surveys: A meta-analysis. Field Methods. 2008;20:249–271. doi: 10.1177/1525822X08317085. [DOI] [Google Scholar]
  39. Šmigelskas K., Lukoševičiūtė J., Vaičiūnas T., Mozūraitytė K., Ivanavičiūtė U., Milevičiūtė I., Žemaitaitytė M. Measurement of health and social behaviors in schoolchildren: Randomized study comparing paper versus electronic mode. Zdr. Varst. 2019;58:1–10. doi: 10.2478/sjph-2019-0001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Smit F., de Zwart W., Spruit I., Monshouwer K., van Ameijden E. Monitoring substance use in adolescents: School survey or household survey? Drugs Educ. Prev. Policy. 2002;9:267–274. doi: 10.1080/09687630210129529. [DOI] [Google Scholar]
  41. Suzuki H., Kaneita Y., Osaki Y., Minowa M., Kanda H., Suzuki K., Wada K., Hayashi K., Tanihata T., Ohida T. Clarification of the factor structure of the 12-item General Health Questionnaire among Japanese adolescents and associated sleep status. Psychiatry Res. 2011;188:138–146. doi: 10.1016/j.psychres.2010.10.025. [DOI] [PubMed] [Google Scholar]
  42. Turner C.F., Ku L., Rogers S.M., Lindberg L.D., Pleck J.H., Sonenstein F.L. Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science. 1998;280:867–873. doi: 10.1126/science.280.5365.867. [DOI] [PubMed] [Google Scholar]
  43. Wang Y.C., Lee C.M., Lew-Ting C.Y., Hsiao C.K., Chen D.R., Chen W.J. Survey of substance use among high school students in Taipei: Web-based questionnaire versus paper-and-pencil questionnaire. J. Adolesc. Health. 2005;37:289–295. doi: 10.1016/j.jadohealth.2005.03.017. [DOI] [PubMed] [Google Scholar]
  44. Yetter G., Capaccioli K. Differences in responses to Web and paper surveys among school professionals. Behav. Res. Methods. 2010;42:266–272. doi: 10.3758/BRM.42.1.266. [DOI] [PubMed] [Google Scholar]
  45. Zeiler M., Peer S., Philipp J., Truttmann S., Wagner G., Karwautz A., Waldherr K. Web-based versus paper-pencil assessment of behavioral problems using the youth self-report. Eur. J. Psychol. Assess. 2021;37:95–103. doi: 10.1027/1015-5759/a000585. [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary data 1
mmc1.docx (22.4KB, docx)

Data Availability Statement

The authors do not have permission to share data.

Articles from Preventive Medicine Reports are provided here courtesy of Elsevier

RESOURCES