Development and Validation of a Short-Form Internet Overuse Screening Questionnaire for Adults

Abstract

Aims

The aim of the current study was to develop and validate a short-form of the internet overuse screening questionnaire (IOS-Qs).

Methods

A total of 571 adults were recruited from a representative, stratified, and multistage cluster sample. Among participants, 188 and 383 were used in the development and validation of the IOS-Qs, respectively.

Results

Experts' ratings and Rasch model analyses led to the selection of 8 items from the IOS-Qs; latent-class analysis using these 8 items revealed an estimated prevalence of 8.6% (33 out of 383) of problematic internet over-users. Problematic internet over-users were positively associated with a 1-year prevalence rate of any mental disorder (OR 3.08, p = 0.008), mood disorder (OR 7.11, p = 0.003), and depressive disorder (OR 5.22, p = 0.016). The receiver operating characteristic curves identified an optimal cutoff score of 9.5 for differentiating problematic internet over-users from unproblematic internet users with 94% sensitivity and 94% specificity.

Conclusion

The results suggest that the IOS-Qs was valid, and items including social isolation were crucial to the brief distinction of at-risk internet users. Because of its brevity, the questionnaire can be effectively administered as a large-scale survey.

Introduction

Internet usage has continuously and substantially increased since its advent. The International Telecommunication Union reported that, in 2018, 51.2% of people around the world could use the internet; this percentage has more than tripled since 2005 (15.8%) [1]. Of note, in 2017, 99.5% of individuals in South Korea had access to the internet, which was the highest percentage among OECD members [2]. However, as availability of the internet has increased, so too has internet overuse. According to an internet usage survey conducted by the Ministry of Science, the Information and Communication Technology, and the Korea Internet and Security Agency in 2017, 8.5% of the respondents reported that they use the internet for 35 h or more per week (5 h a day) [3].

Internet overuse is a serious issue, in part because it may be accompanied by one or more problems (e.g., loneliness [4], sleep disorders [5], other psychiatric disorders [6, 7, 8]). The lives of those with an addiction to the internet may be very disabled and accompanied by a higher frequency of suicidal-ideation than those not addicted to the internet [9, 10]. In addition, heavy internet usage corresponds to poor academic proficiency [11] and lower academic scores [12]. Since identifying individuals with an addiction to the internet is a crucial step in intervening, we aimed to develop a brief and sensitive screening tool that can be easily administered to a large sample to find at-risk internet over-users.

Various questionnaires related to internet addiction have been developed [13, 14]. Some questionnaires deal with specific types of excessive internet use (e.g., games [15, 16, 17], searches [18], social networking services) [19]. To screen for general internet over-users, >20 questionnaires have been developed; however, most have limitations (e.g., improper cut-offs, irrelevant items, lack of diagnostic criteria) [20]. Importantly, Lee et al. [22] noted that: (i) the Internet Addiction Test may not be suitable for those under a certain age (i.e., “youth”) and does not support diagnosis, (ii) the Chinese Internet Addiction Scale [21] is not thoroughly validated, and (iii) the Korean Scale for Internet Addiction includes ambiguous items.

To overcome these limitations, Lee et al. [22] developed the internet overuse screening-questionnaire ­(IOS-Q) to screen for at-risk internet over-users (Lee et al. [22]). It is worth noting that the IOS-Q was developed and validated to screen for Korean internet over-users. However, the IOS-Q is composed of 28 items, a potentially excessive number of questions to identify internet over-users, thus making it a lengthy tool that is difficult to broadly administer. Therefore, developing a shorter version of the questionnaire is more applicable in the clinical field due to its shorter administration time. A shorter version would help to screen problematic internet overuse in a brief and concise manner.

The Current Study

In the current study, a short form of the IOS-Qs for adults was developed and validated. Based on the results of an a priori sample size calculation in the G-power software [23], we divided the sample set for the development and the validation of this new tool. The development process was based on expert ratings for the importance and adequacy of each item, and the Rasch model was applied to evaluate the items' fitness and relative difficulty of the items to select effectual questions. Items were determined through group discussions with experts in psychiatry, psychopathology, and psychology as well as the results from Rasch modeling. In the validation process, psychometric properties were firstly checked via reliability, factor structure, inter-item, and item-total correlations. Concurrent validity was checked via correlations with mental distress [24, 25], sleeping problems [5], low quality in family relationships [26], psychiatric disorders, depressive mood disorders [9, 27], and anxiety disorders [28]. Since there was not a direct question to ask for the quality of participants' family relationships, a question about perceived importance of family was used instead.

We also performed a latent-class analysis using the items selected in the development process to categorize adults who used the internet problematically and those who did not. The latent-class analysis has been adapted to classify users at risk of problematic Internet overuse and those with no risk [29, 30]. Next, potential psychiatric comorbidities were assessed. Moreover, the optimal cutoff score of the IOS-Qs was suggested from the ROC curve.

Methods

Sample

The sample was derived from the “Survey of Mental Disorders in Korea.” The survey was conducted between April 2016 and ­November 2016. The target population included all eligible residents 18 years and older who were listed in the 2010 population census of the community registry offices. A total of 5,102 households were selected using a stratified, multi-stage, and cluster sampling method. For the selection of a person per selected household, the “last birthday” method was administered. One person who had the most recent birthday was chosen. Only the day and the month of the birth date, not the birth year, was considered. Of the 5,102 participants randomly selected, 571 were told to answer the ­IOS-Qs. Because the “Survey of Mental Disorders in Korea” was mainly for examining the prevalence of mental disorders, questionnaires about internet usage were administered to a relatively small sample (n = 571) due to the time and cost. The flow chart of the studied sample is presented in online supplementary Figure 1 (for all online suppl. material, see www.karger.com/doi/10.1159/000506629).

Among the 571 participants, 188 were randomly selected and assigned to the development of the IOS-Qs, while the remaining 383 were assigned to scale validation. Random assignments were determined using random number generation in Microsoft Excel. The G-power software [23] revealed that 383 participants were required for regression analysis, thus this number of ­participants were assigned for the validation process. Demographics for each sample are presented in online supplementary Table 1.

Measures

Internet Overuse Screening Questionnaire

The IOS-Qs was developed and validated to identify at-risk internet users through their self-reported internet usage habits [31]. The questionnaire is composed of 28 items with 4-point Likert scales ranging from 1 (Never), 2 (Sometimes), 3 (Frequently), and 4 (Always) and total scores ranging from 28 to 112. The 28 items selected were identified through expert evaluation and panel discussions.

Mental Distress

Mental distress was measured by an item of “Circle the number (0–10) that best indicates the degree of mental distress you experienced in the past year” with an 11-point Likert scale (0 = not distressed at all, 10 = extremely distressed).

Sleeping Problems

Sleeping problems were measured using the item of measuring sleeping problems in “Depression and dysthymia” of the Diagnostic and Statistical Manual of Mental Disorders criteria (DSM) IV. The question such that “Did you have difficulty sleeping almost every night, for at least 2 weeks, when you were depressed/lost interest/lacked energy in your lifetime? In other words, did you have trouble falling asleep, woke up in the middle of the night, or woke up too early for at least 2 weeks?” was asked to participants with the answer choices of “Yes” or “No”.

Importance of Family

The perceived importance of family was measured by an item of “How important are the items below in your life?: Family” with a 4-point Likert scale (1: not important at all. 4: extremely important).

Diagnostic and Statistical Manual of Mental Disorders-IV

We used the Korean version of the Composite International Diagnostic Interview (K-CIDI) to diagnose psychiatric disorders using the DSM-IV. Cho et al. [32] validated the K-CIDI according to the World Health Organization guidelines [33, 34]. K-CIDI is a fully structured, easily administered diagnostic interview. A 12-month prevalence of DSM-IV psychiatric disorders to evaluate comorbidities of internet overuse was examined.

Experts Ratings

Eight experts in the fields of psychiatry (2 professors, 3 clinical researchers), psychology (1 professor, 1 post-doctoral researcher), and clinical pathology (1 medical laboratory technologist) rated the adequacy and importance of each items via a 5-point Likert scale (1 = not at all, 5 = to a large extent). They were not aware of which items were selected through the Rasch model analysis.

Statistical Analysis

Analyses were conducted using SPSS (version 18.0 for Windows, IL, USA), jMetrik (version 4.1.1 for Mac, created by Patrick Meyer, University of Virginia, www.itemanalysis.com) and the Mplus software (version 6.0 for Windows) [35]. Descriptions for demographics and response frequencies were analyzed using SPSS.

For the development of the IOS-Qs, a Rasch model for polytomous items (i.e., the Rating Scale Model) was applied using the jMetrik [36, 37, 38]. The Rating Scale Model is an extension of the 1-parameter Rasch model and involves an assumption that items differ in their level of difficulty but not their slope. Since the RSM assumed parameter invariance, the person and item parameters are independent of each other. In addition, the Rasch model, including RSM, had the benefit of specific objectivity that estimated latent traits and item difficulties are independent from the sample. These characteristics improve interpretability of the results. Item difficulties, fit statistics, and reliabilities were investigated. Infit mean squares and outfit mean squares were checked to determine item fits. The acceptable range of infit mean squares and outfit mean squares ranged from 0.6 to 1.4 [39]. Since the outfit mean square was sensitive to outliers, we focused on the infit mean square [40]. Items for the IOS-Qs were selected based on the group discussion, the item ratings of experts, and the results of statistical analysis (i.e., description of item response, item difficulties, and fit statistics from Rasch modeling).

For the development of the IOS-Qs, we first examined the psychometric properties and concurrent validity of the IOS-Qs. Cronbach's alpha was calculated. For the Exploratory Factor Analysis, we used principal axis factoring with direct oblimin rotated solution. Appropriate number of factors were determined using the Kaiser criterion (i.e., factors with eigenvalues >1.0) and the screen test. Pearson's correlation for the continuous variables or ­Spearman's correlation for the nominal variables was calculated to investigate the inter-item correlations, item-total correlations, and concurrent validities.

Distinct classes of internet overuse profiles were identified with the Latent Class Analysis (LCA) [41]. Items selected from the development of the IOS-Qs were used, and models for 2–4 classes were investigated. The goodness-of-fit statistics, Akaike information criteria (AIC), Bayesian information criteria (BIC), sample size adjusted BIC (SSABIC), and bootstrapped likelihood ratio test (BLRT) p values were compared across the 3 models. The lower values in the AIC, BIC, and SSABIC reveal a more parsimonious model. A significant p value in the BLRT indicates that the estimated model shows significant improvement in the model fit when the class number is large (k) rather than small (k-1). In order to determine the class number, we mostly utilized the BLRT p values, BIC [42], and SSABIC [43] value.

The socio-demographic variables were compared between the “unproblematic internet users” group and “problematic internet over-users” group using a t test for the continuous variables or chi-square for the categorical variables. To identify any potential associations with DSM-IV psychiatric disorders, we conducted logistic regressions using SPSS. Adjusted ORs with 95% CIs were calculated based on logistic regressions with the prevalence of psychiatric diagnoses as the main outcome variable and group (i.e., problematic internet over-users, unproblematic internet users) as the main predictor while adjusting for age, gender, education, and marital status. Two-tailed p values below 0.05 were considered statistically significant throughout the analysis.

Results

Development of the Short-Form of the IOS-Qs

Expert Ratings about Adequacy and Importance of the Items

Eight experts in the fields of psychiatry, psychology, and clinical pathology rated the adequacy and importance of each items via a 5-point Likert scale (1 = not at all, 5 = to a large extent). Since the original items of the IOS-Q were developed based on the ratings of adequacy and importance [44], all the items of the IOS-Q showed >3 (neutral) in the 5-point Likert scale except for the importance of items 9 and 11. Thus, these 2 items were not included in the IOS-Qs. In addition, the mean of adequacy and importance of items for the IOS-Qs were higher (4.02 and 4.03, respectively) than those of the nonselected items (3.67 and 3.44 respectively). These item selection procedures involved 2 professors in psychiatry and 1 professor in psychology, participating in a series of group discussions about the contents of the items.

Rasch Modeling and Item Selection

Along with the expert ratings, Rasch modeling was conducted. Response frequencies for the development process (n = 188) of the IOS-Qs are presented in online supplementary Table 2. To develop these IOS-Qs, item difficulties and fit statistics were examined using Rasch analysis (Table 1). Since the scale was developed for screening problematic internet over-users, the items that have high difficulty with good fits were selected. Among the original 28 items, 8 were selected to be validated as the IOS-Qs. Item fits ranged from 0.6 to 1.4, acceptable levels [39]. Item number 25 had the highest difficulty (δ = 2.53), however, not too dissimilar from item 24 (δ = 2.11), a close second; the infit was better in item 24 (infit = 0.97) compared to item 25 (infit = 0.89). In addition, no one answered response 2 (i.e., sometimes) in the validation process (online suppl. Table 2), thus the 25th item was omitted in the IOS-Qs.

Table 1.

Item difficulty and fit statistics for the IOS-Qs (development sample, n = 188)

	Difficulty		Infit		Outfit		rt
	δ	SE	M	Z	M	Z
Item number of the original scale (Lee et al., [22] 2017)
7. I become irritated or angry without the internet	0.03	0.26	1.12	0.57	0.90	−0.12	0.57
6. I need to use the internet for a longer period of time to become as satisfied as before	0.59	0.31	0.99	0.05	0.76	−0.31	0.53
22. I like using the internet more than spending time with family	0.69	0.33	1.26	0.91	0.77	−0.25	0.32
21. I like using the internet more than socializing with friends	0.92	0.35	1.09	0.36	0.66	−0.39	0.41
18. I am late for school, work, appointments, and so on, due to my internet use	1.21	0.40	0.86	−0.28	1.00	0.24	0.41
23. I quarrel with family due to my internet use	1.57	0.46	0.69	−0.69	0.18	−1.18	0.55
14. I skip school or work to go somewhere where I can use the internet	1.81	0.50	0.97	0.10	1.52	0.78	0.25
24. I am not interested in anything except using the internet	2.11	0.57	0.97	0.11	0.40	−0.29	0.24
Items not used in short form of the IOS (IOS-Qs)
1. I often think about the internet even while doing other work	−1.25	0.18	1.01	0.09	1.17	0.08	0.58
2. I wait excessively to access the internet again	−0.57	0.22	0.99	0.00	0.77	−0.74	0.60
3. I stay up all night using the internet	−0.27	0.24	0.91	−0.35	0.79	−0.53	0.59
4. I am tired and sleepy during class or work due to my internet use	−0.21	0.24	0.87	−0.52	0.62	−1.09	0.64
5. I check the internet excessively by habit	−1.86	0.16	1.01	0.10	1.19	1.07	0.66
8. I become anxious or nervous without the internet	−0.21	0.24	0.90	−0.41	0.80	−0.49	0.64
9. When I am in a bad mood, using the internet makes me feel better	−1.31	0.18	1.21	1.23	1.22	1.04	0.60
10. I tried to reduce my internet usage, but it is difficult	−1.01	0.19	0.94	−0.26	0.63	−1.64	0.71
11. I underreport the amount of time I spend on the internet	0.03	0.26	1.51	1.94	1.10	0.36	0.30
12. When I use the internet, I lose track of how much time has passed	−2.34	0.15	0.87	−0.90	1.02	0.16	0.71
13. People around me point out that I spend a lot of time on the internet	−0.78	0.21	0.92	−0.39	0.78	−0.79	0.69
15. I don't think I can reduce my internet usage without help from others	1.21	0.40	1.21	0.67	0.41	−0.80	0.47
16. Despite being sick, I continue to use the internet (e.g., lack of sleep, eye fatigue, hand or neck pain, etc.)	−1.05	0.19	1.08	0.47	0.92	−0.24	0.64
17. I spend most of my allowance or salary for internet use	1.06	0.37	2.13	2.63	1.81	1.19	0.20
19. I skip meals or eat while using the internet	−0.43	0.23	1.54	2.29	1.35	1.09	0.44
20. I use the internet instead of doing things I need to do	−0.66	0.21	1.35	1.70	1.11	0.46	0.52
25. I became distant from friends and colleagues since my internet use	2.53	0.69	0.89	0.04	0.24	−0.33	0.28
26. I think that I am addicted to the internet	−1.10	0.25	0.80	−0.88	0.41	−1.88	0.73
27. I want to change my current internet usage habits	−0.66	0.21	0.86	−0.68	0.54	−1.80	0.76
28. I think I use the internet excessively	−1.05	0.19	0.71	−1.74	0.52	−2.36	0.77

δ, internet overuse level; M, mean-square-fit statistic; Z, standardized mean-square-fit statistic; r_t, point-biserial correlation between item and total measured internet overuse level based on the Rasch calibrated item scores and total scores; IOS-Qs, internet overuse screening questionnaire.

Validation of the IOS-Qs

Psychometric Properties of the Short-Form of the IOS-Qs

Using the validation sample (n = 383), psychometric properties of the IOS-Qs were checked. Cronbach's alpha of the selected 8 items was 0.80. To find the structural validity, Exploratory Factor Analysis was performed. The Kaiser-Meyer-Olkin was 0.790, and the significance of Bartlett's test of sphericity (χ² = 1,024.81, p < 0.001) showed that the data were appropriate for the factor analysis. Principle axis factoring with oblimin rotation yielded 2 factor structures, and these latent variables explained 46.20% of the variance. Factor loadings of each item are presented in online supplementary ­Table 3. The items reflecting social isolation (family in item 22 and item 23, friends in item 21) and in general (item 24) were highly loaded in factor 1. The coefficient value of item 14 was equivocal, but the item contained social isolation such as school or work, thus it was included in factor 1. Based on the meaning of the items in factor 1, it was named “social isolation.” Factor 2 was named “misuse and dependence.” Two psychiatrists and a psychologist discussed such naming process. Correlations among each item, sub-total scores for the factors, and total score are presented in online supplementary Table 4. Item-total correlations ranged from 0.43 to 0.83. The inter-item correlations were all significant except for the correlation between item 23 and item 18. These 2 items were loaded to different ­factors.

Concurrent Validity of the IOS-Qs

The mean of the sum score of IOS-Qs was 8.66 (SD 1.64) and ranged from 8 to 20. Correlations with mental distress, sleeping problems, relationships with family, and psychiatric characteristics were assessed (Table 2). The sum score of IOS-Qs showed positive correlations with mental distress (r = 0.151, p = 0.003), sleeping problems (r = 0.178, p < 0.001), all mental disorders (r = 0.262, p < 0.001), mood disorders (r = 0.223, p < 0.001), depressive disorders (r = 0.201, p = 0.003), and anxiety disorders (r = 0.196, p = 0.035), whereas there was negative correlation with perceived importance of family (r = −0.170, p = 0.001).

Table 2.

Correlation among scores of the short-form of the IOS-Qs and psychological and psychiatric characteristics

	Psychological characteristics			Psychiatric characteristics
	distress	sleeping problems	importance of family	all mental disorders	mood disorders	depressive disorders	anxiety disorders
Factor 1 (social isolation)	0.151**	0.130*	−0.144**	0.270***	0.278***	0.251**	0.181**
Factor 2 (misuse and dependence)	0.104*	0.142**	−0.150**	0.138**	0.070	0.046	0.144*
Total	0.151**	0.178***	−0.170**	0.262***	0.223***	0.201**	0.196*

^*p < 0.05,

^**p < 0.01

^***p < 0.001.

IOS-Qs, internet overuse screening questionnaire.

Psychiatric characteristics were differentiated depending on factors. Factor 1, named “social isolation,” showed a positive relationship with psychiatric disorders, but factor 2, named “misuse and dependence,” showed positive relationships only with all mental disorders and anxiety disorders.

LCA and the Comorbidity of Psychiatry Disorders Across the Latent Classes

To validate the IOS-Qs developed using the Rasch approach, latent groups were estimated, and the comorbidity of psychiatric disorders depending on group type were assessed. The frequencies of responses in the validation (n = 383) of the IOS-Qs are presented in online supplementary Table 2.

LCA Model Selection

A latent profile analysis with 1–4 classes was performed on the 8 selected items in the validation sample (n = 383). Table 3 reveals the fit indices and test p values for models with 1 through 4 classes. The AIC, BIC, and SSABIC were the smallest in the 2-class solution. Regarding the BLRT, p values were significant in the 2 and more class solution. Solutions with >3 latent classes contained classes representing very small groups of students (i.e., n = 4) and based on the interpretability, the 2-class solution was selected.

Table 3.

Fit indices for the latent-class analysis of the short-form IOS-Qs

Class number	AIC	BIC	SSABIC	BLRT p value	Smallest class, n (%)
1	1,090.991	1,146.263	1,101.844	−	−
2	5.331	92.188	22.385	<0.001	33 (8.6)
3	883.216	764.775	859.960	<0.001	4 (1.0)
4	1,041.643	891.618	1,012.186	<0.001	4 (1.0)

Adjusted BIC; BLRT p value = p value associated with BLRT.

AIC, Akaike information criteria; BIC, Bayesian information criteria; SSABIC, sample size adjusted BIC; BLRT, bootstrap likelihood ratio test; IOS-Qs, internet overuse screening questionnaire.

Descriptions of the Two Latent Classes: Unproblematic Internet Users and Problematic Internet Over-Users

The largest class, comprising 91.4% (n = 350) of the sample, had low sum scores of the IOS-Qs (sum score = 8.27, SD 0.72), and were thus labeled the unproblematic internet user class. The remaining individuals, comprising 8.6% (n = 33) of the sample, had high sum scores of the IOS-Qs (sum score = 12.76, SD 2.76) and were thus labeled the problematic internet over-user group. An independent t test revealed that the sum score difference between the unproblematic internet user and the problematic internet over-user group was significant (t [32.41] = 9.30, p < 0.001, cohen's d = 2.23).

The prevalence of problematic internet over-user group by sociodemographic variables is presented in Table 4. The mean age of those in the problematic internet over-user group was younger than those in the unproblematic internet user group (30.06 vs. 37.37 years [t (381) = 2.99, p = 0.003, Cohen's d = 0.59]). The percentage of females in the problematic internet over-user group was 78.8% (26 out of 33), while the unproblematic internet user group was 45.4% (159 out of 350; χ² [1] = 13.44, p = 0.003). There was no mean difference in years of education between the 2 groups (t [381] = 0.21, p = 0.835). While the percentage of unmarried individuals in the problematic internet over-user group was higher than expected (63.6% [21 out of 33]), only 50.8% (178 out of 350) were unmarried in the unproblematic internet user group (χ² [2] = 8.16, p = 0.017).

Table 4.

Prevalence of internet over-users according to sociodemographic variables

Variables	Unproblematic internet users (n = 350)	Problematic internet over-users (n = 33)	t or x (df)	p value	Total (n = 383)
Age, years
≤17	1 (0.3)	0 (0.0)			1 (0.3)
18–29	120 (34.3)	20 (60.6)			140 (36.6)
30–39	73 (20.9)	4 (12.1)			77 (20.1)
40–49	82 (23.4)	7 (21.2)			89 (23.2)
50–59	45 (12.9)	2 (6.1)			47 (12.3)
60–69	21 (6.0)	0 (0.0)			21 (5.5)
70–79	6 (1.7)	0 (0.0)			6 (1.6)
≥80	2 (0.6)	0 (0.0)			2 (0.5)
Mean (SD)	37.37 (13.60)	30.06 (11.09)	2.99 (381)	0.003	36.74 (13.55)
Range	17–82	20–55			17–82
Gender
Male	191 (54.6)	7 (21.2)	13.44 (1)	<0.001	198 (51.7)
Female	159 (45.4)	26 (78.8)			185 (48.3)
Education
≤ Elementary school (≤6 years)	8 (2.3)	0 (0.0)			8 (2.1)
Middle school (7–9 years)	8 (2.3)	0 (0.0)			8 (2.1)
High school (10–12 years)	63 (18.0)	7 (21.2)			70 (18.3)
> University (>12 years)	271 (77.4)	26 (78.8)			297 (77.5)
Mean (SD)	14.53 (2.90)	14.64 (2.10)	0.21 (381)	0.835	14.54 (2.84)
Range	0–26	12–20			0–26
Marital status
Married	172 (49.1)	8 (24.2)	8.16 (2)	0.017	180 (47.0)
Divorced/separated/widowed	20 (5.7)	4 (12.1)			24 (6.3)
Unmarried	158 (45.1)	21 (63.6)			179 (46.7)

df, degree of freedom.

Comorbid Psychiatric Disorder

The associations between problematic internet overuse and psychiatric disorders are presented in Table 5. Logistic regressions revealed that problematic internet over-users were significantly associated with 1-year prevalence of any DSM-IV disorders (OR 3.08, p = 0.008), mood disorders (OR 7.11, p = 0.003), and depressive disorders (OR 5.22, p = 0.016) compared to unproblematic internet users after adjusting for age, gender, education, and marital status. The significances were the same in the problematic internet over-user group classified using full items (online suppl. Table 3).

Table 5.

Comorbidities of 1 year of mental disorders in problematic internet over-users (n = 33 out of 383)

Dependent variables	b	SE	OR (95% CI)	p value
All mental disorders	1.13	0.43	3.08 (1.33–7.11)	0.008
Mood disorder	1.96	0.66	7.11 (1.97–25.68)	0.003
Depressive disorder	1.65	0.69	5.22 (1.36–20.05)	0.016
Dysthymic disorder	−	−	−	−
Bipolar disorder	−	−	−	−
Anxiety disorder	1.293	0.97	3.64 (0.55–24.21)	0.181
Post-traumatic stress disorder	−	−	−	−
Obsessive compulsive disorder
Panic disorder	−	−	−	−
Agoraphobia	−	−	−	−
Social anxiety disorder	1.35	1.02	2.86 (0.53–28.41)	0.184
Generalized anxiety disorder	−	−	−	−
Nicotine use disorder	−0.30	1.01	0.74 (0.09–6.34)	0.783
Alcohol use disorder	0.76	0.64	2.15 (0.61–7.51)	0.232

Cut-Off Estimation

The ROC curves identified an optimal cutoff score of 9.5 differentiating problematic internet over-users from unproblematic internet users (area under the curve 0.983 p < 0.001) with a sensitivity of 0.94 and a specificity of 0.94.

Discussion

The current study was designed to develop and validate an IOS-Qs. Results revealed that the IOS-Qs was reliable and valid, thus it would help to enhance efficiency and convenience of screening for problematic internet over-users in research and clinics. Also, patterns of comorbidities in internet over-users were classified using latent-class analysis with the original 28 items and the 8 items, confirming the expected overlap for a valid short-form survey (i.e., the IOS-Qs). Clinicians should pay close attention to adults who answered “sometimes” to >2 items or “frequently” to >1 out of the 8 items on this IOS-Qs.

Eight out of original 28 items were selected based on the item fits and difficulties through Rasch modeling. Since the screening tool was designed to identify problematic internet over-users, items with high difficulties would work more efficiently than the items with low difficulties. It is worth noting that the statistical method (i.e., Rasch modeling) was mainly utilized to select the items, but the content validity of each item should be tested within an expert group. For instance, in the development process of the original IOS-Q, which includes 28 items, 50 members of the addiction society rated the items in terms of adequacy and importance with a 5-point Likert scale [31]. The adequacy and importance of the 8 selected items in the current study were above average in an all item-pool, except the item of “I like using the internet more than spending time with family.” The average for adequacy and importance in all items were 3.94 and 3.90, respectively, while the item (i.e., “I like using the internet more than spending time with family”) showed 3.63 in adequacy and 3.52 in importance in the previous study [31]. Such low adequacy and importance of the item showed the necessity for further examination of content validity.

The content of the items in the IOS-Qs could be a starting point for the early detection of, and intervention for, problematic internet overusers. Results suggest that the neglect of other areas, including social life (e.g., item number 21, 22, 24) and misuse and dependence (e.g., item number 6, 7), are critical symptoms to guide intervention for problematic internet over-users compared to the awareness of self-control (e.g., item number 15, “I don't think I can reduce my internet usage without help from others”) or internet using behavior (e.g., item number 3, “I stay up all night using internet”). Among the items, misuse and tolerance for the internet were common phenomena for addiction (e.g., alcohol addiction, nicotine addiction); however, the social problem (i.e., item number 21, 22, 24) is a relatively unique characteristic for problematic internet overuse. The factor for “social isolation” is significantly associated with psychological and psychiatric characteristics and these results showed the significance of the social factor in problematic internet overuse. Problematic internet use is related with low social support [45, 46] and dysfunctional social behaviors [47]. Family conflict caused by internet overuse is an indicator of internet addiction [48]. Negative consequences in social life due to internet use may be a critical proxy for identifying problematic internet over-users.

Social factors have been included in many previous instruments for measuring problematic internet use. For instance, perceived social benefits online and perceived social control online are included in the Generalized Problematic Internet Use [49], preference for online social interaction in the modified Generalized Problematic Internet Use [50], and social interactions in the Problematic Internet Use Questionnaire [13]. These assessment instruments are to measure the social characteristics in the internet, but the current study showed the importance of problems in social life. Feeling social connection or social control online may not be harmful, but neglecting real life would be a problem. Therefore, items that assess complications in everyday social life due to the overuse of the internet would increase specificity of the questionnaire. Since we did not confirm the potential causal connection for the neglect of social life and internet overuse, additional studies should be conducted to examine the effects of an educational program(s) that focuses on problems in an individual's social life in order to prevent or intervene in problematic internet overuse.

To date, there are no clear definitions or diagnostic criterion for internet over-users [51], thus, identifying problematic internet over-users is the most difficult aspect of developing a valid screening questionnaire. To address this problem, we used latent-class analysis to represent distinctive subgroups based on internet use. As suggested by previous studies, this person-centered analysis (i.e., LCA) can be useful to assess the subtypes of problematic internet users [30, 52].

The current study suggests epidemiologic evidence for the prevalence and comorbidities of problematic internet overusers in a nationwide community adult sample. Most previous studies focused on younger populations [11, 53], thus the results of the current study can be helpful in estimating the current status of internet overuse in Korean adults. In 2017, the Ministry of Science, the Information and Communication Technology, and the Korea Internet and Security Agency found that 8.5% of Koreans (i.e., from children older than 3 to elder adults) used the internet for 35 h or more per week (5 h a day) [3]. The percentage was similar with that of the current study, which noted that 33 out of 383 adults (8.6%) were classified as problematic internet over-users. We can assume that the prevalence of problematic internet users in Korean adults was around 8.6%. However, more nationwide studies using this validated questionnaire are needed in order to better estimate the prevalence.

Among the demographic variables, gender and marital status were different between the unproblematic internet users and problematic internet over-users. Gender has been a critical factor associated with internet overuse for internet addiction. In contrast to the result of the current study, previous studies reported that males are represented more with internet addiction than females [54, 55] or no statistical difference [29, 30]. The differences among the results may be caused by the differences in the age range of the samples, the items, or cultural differences. Specifically, most of the previous studies focused on adolescents or early adults (e.g., college students) [20]. Insufficient research on the specific investigation of problematic internet overuse in mid or late adults hinders the formulation of any definitive explanation. However, the frequency graph of problematic internet over-users across gender (online suppl. Fig. 2), in which mid and late adult females have higher frequency than males, supports the explanation that higher prevalence of problematic internet over-users are mid or late adult women rather than men. While mid or late adult women had much higher frequencies than men, women also showed higher frequencies in early age as well. Further studies should be performed to sophisticate the understanding of the prevalence of problematic internet overuse across genders.

In addition, being unmarried in late life is related to maladaptive psychiatric dysfunction such as depression [56]. Since the current study covers participants in late life, marital status may emerge as a significantly different factor. More studies should be performed to explain the reason of the association between problematic internet overuse and marital status.

Internet addiction deserves clinical attention and needs to be better defined to help clarify comorbid psychiatric symptoms. Our results reveal that problematic internet overuse is closely associated with mental disorders, especially depressive disorders. Previous studies consistently suggested that depression is associated with problematic internet overuse behaviors [6, 10, 55, 57]. Although we cannot guarantee the causal relationship with this correlational study, it is hypothetically possible that the depressive symptom is caused by internet overuse, based on the fact that internet communication with people can alleviate depression and loneliness, at least among the socially isolated and moderately depressed populations, who may tend to rely on social technologies to receive social support [58]. In line with the concept that emphasizes the importance of social life in the content of the items, researchers may wish to pay more attention on the social life of adults with problematic internet use.

This study was the first to develop an IOS-Qs through expert discussions and Rasch modeling. It is also the first to evaluate psychometric properties and psychiatric comorbidity of problematic internet overuse in adults older than 18 years of age in a community-based population. Because of the newly designed questionnaire's brevity, it may be effective for administration in a large-scale survey. Demographic differences and psychiatric comorbidities of problematic internet overuse class indicate internet users with problematic overuse associated with psychosocial and neurobiological factors. Clinicians should consider psychiatric comorbidities when caring for adults overusing the internet.

Statement of Ethics

The review board of Ministry of Health and Welfare, South Korea, approved the study protocol, and all participants gave written informed consent. The research was conducted according to the Helsinki Declaration guidelines.

Disclosure Statement

No competing financial interests exist.

Funding Sources

There is no funding to declare.

Author Contributions

S.P., J.-P.H., H.J.J., and J.-Y.L. designed study; S.P. and B.C.: analyzed data; S.P. drafted the manuscript; S.C., S.L., J.K., and J.E.P. co-wrote and revised the manuscript; all authors provided critical feedback and helped shape the research, analysis, and manuscript.

Supplementary Material

Supplementary data