Think-aloud analysis of commonly used screening instruments for Internet use disorders: The CIUS, the IGDT-10, and the BSMAS

Abstract

Background

Despite the constant publication of new screening instruments for Internet use disorders (IUD), little is known about their content validity. This study aimed to identify potential mismatches between the items' intention and young adults' interpretation of these items when answering three screening instruments that are commonly used in research and clinical settings: The Compulsive Internet Use Scale (CIUS), the 10 Item-Internet Gaming Disorder Test (IGDT-10), and the Bergen Social Media Addiction Scale (BSMAS).

Methods

In total, 30 vocational students (50% female, age = 21.3; SD = 2.1) took part in individual think-aloud interviews. All participants were asked to report their thoughts while completing the CIUS. In addition, participants who reported online games (OG) as their main Internet activity (n = 11) answered the IGDT-10. Participants who reported social networks (SN) as their main Internet activity (n = 18) answered the BSMAS. One participant used OG and SN equally and completed both screening instruments. All interviews were audio-recorded, transcribed, and content-analysed.

Results

Overall, four potential sources for errors were identified: (1) High scorings were often not congruent with the underlying diagnostic criteria. In particular, such discrepancies were found in items aimed to assess dysfunctional emotional regulation strategies and cognitive involvement. (2) Some participants were uncertain which time frame or Internet activity should be considered in their answers. (3) Long and complex items led to the building of mean values or the choice of the middle answer category. (4) Some wordings were perceived to be outdated and difficult to understand.

Discussion

These findings might help to provide recommendations on how to improve screening instruments for IUD. Most important, items should more clearly distinguish between Internet use as a “normal” leisure activity and Internet use that leads to functional impairments in daily life.

Introduction

Internet use disorders (IUD) are widely discussed as a potential new diagnosis in the spectrum of behavioral addictions. To date, the conceptual framework of IUD is still part of a controversial debate. Several researchers have criticized the term “IUD” to be inadequate because it passes over important differences between heterogeneous Internet activities (Starcevic & Billieux, 2017). In the last years, research has mainly focused on the psychological mechanism underlying pathological gaming. This has led to the inclusion of the Internet gaming disorder (IGD) in the 5th edition of the “Diagnostic and Statistical Manual of Mental Disorders” (DSM-5; American Psychiatric Association [APA], 2013). Besides, Gaming Disorder (GD) was included in the 11th edition of the “International Classification of Diseases and Related Health Problems” (ICD-11, World Health Organization [WHO], 2011). Although not yet listed in the DSM-5 or ICD-11 as distinct Internet activity, there is growing evidence that the pathological use of online social networks can also be classified as addictive behavior that leads to significant impairments in daily life (D’Arienzo, Boursier, & Griffiths, 2019).

Screening instruments for IUD

Despite the constant development and publication of new screening instruments, the optimal assessment of IUD is still unclear. Most screening instruments suffer from an insufficient conceptual or methodological background (King, Chamberlain, et al., 2020). Besides, there is a tendency in research to label any excessive Internet use behavior as a potential new behavioral addiction (Billieux, Schimmenti, Khazaal, Maurage, & Heeren, 2015). This may result in overpathologizing “normal” leisure activities of young adults and high rates of false-positive results (Billieux et al., 2015). To date, existing screening instruments either assess Internet use as a general construct or focus on specific Internet applications (e.g., gaming or social networks). The Compulsive Internet Use Scale (CIUS), the 10 Item-Internet Gaming Disorder Test (IGDT-10), and the Bergen Social Media Addiction Scale (BSMAS) are examples of such screening instruments that are widely used for research purposes and in clinical settings.

Compulsive Internet Use Scale (CIUS)

The CIUS (Meerkerk, Van Den Eijnden, Vermulst, & Garretsen, 2009) measures IUD as a general construct. In total, 14 items represent the following criteria for Compulsive Internet Use (CIU): Salience, withdrawal, loss of control, conflict, and mood modification (Meerkerk et al., 2009). These criteria were adopted by the criteria for DSM-IV Dependence (APA, 1994), DSM-IV Pathological Gambling (APA, 1994), and the criteria of behavioral addictions as suggested by Griffiths (1999). All items can be answered on a 5-point Likert scale, ranging from “never” (0) to “very often” (4). Participants scoring at least 28 points in the CIUS are at increased risk for pathological Internet use (Meerkerk et al., 2009). The psychometric quality of the CIUS was tested in previous studies. Good internal reliability criteria (Cronbach's α ranging from 0.88 to 0.90) and a stable one-factor solution were found among different samples (Meerkerk et al., 2009). All items of the CIUS are listed in Table 2.

Table 2.

Compulsive Internet Use Scale (CIUS)

CIUS instruction: “Choose an answer that best applies to you.”
Item	Criterion (Meerkerk et al., 2009)	Difficulties	Suggestions for improvement	Statements of the participants
(1) Do you find it difficult to stop using the Internet when you are online?	Loss of control	The item was often associated with high scorings although participants were able to stop their Internet use in case of important other tasks.	Consider to specify the context to assess the negative impact on daily life (e.g., “[…] to stop using the Internet even if you have other important things to do?”).	“It's not difficult for me to stop when I know that I have to fulfill responsibilities.”
(2) Do you often continue to use the Internet despite your intention to stop?	Loss of control	Strong similarity to the first item.	Check for redundancy; possibly delete one of these items.	“I compare that item with the first question. They definitely look similar to me […].” “That's the same […] that actually results from the first question.”
(3) Do others (e.g., friends and family) say you should use the Internet less?	Conflict	Participants' answers depend on their age. The Internet use of young people is often criticized by parents or grand-parents. This item has the potential to cover a generation conflict rather than characteristics of a behavioral addiction.	Consider to split this item into (1) conflicts with same-aged peers or partners and (2) conflicts with family members.	“Actually, no one except my grandma […]”. “Often, especially my parents.”
(4) Do you prefer to use the Internet instead of spending time with others (e.g., friends and family)?	Preoccupation	To date, many real-life friendships of young adults are maintained online. Some reasons are long distances after moving to another town or sudden life-changing events such as the corona pandemic. In this context, spending time with others online is not necessarily a problematic behavior but (quite the opposite) a functional way to maintain real-life friendships.	Consider to reword this item to assess problematic social withdrawal in real life (e.g., “How often do you prefer to use the Internet rather than spend time offline with others?”)	“[…; My] friends don't live here. The Internet is actually the only option to communicate with them. So, actually very often.”
(5) Are you short of sleep because of the Internet?	Loss of control	No relevant difficulties. The item was easy to understand.
(6) Do you think about the Internet, even when you are not online?	Preoccupation	Participants often relate to boredom or to situations when alternative leisure activities are not available. It remains unclear if high scorings of this item were associated with a negative impact on daily life.	Consider to reword and specify this item (e.g., “How often do you have a strong desire to use the Internet although you have other important things to do?”).	“I guess when it's boring […]. When you're busy, you don't think about it.”
		The term “when you are not online” seems outdated. Nowadays, access to the Internet is possible almost always and everywhere.	See above.	“Sometimes, because I'm actually always on the Internet. No matter whether I'm on the move or at home […].”
		Participants think of online social interactions or important messages that are expected but not about “the Internet” per se.	See above.	“I'm thinking a lot about the Internet […] whether someone important has written to me […] I'm still waiting for a very important answer from a colleague.” “Internet is a very broad topic, that's why you think often about the different dialogues you had on the Internet […].”
(7) Do you look forward to your next Internet session?	Preoccupation	See item 6: The term “look forward to your next Internet session” seems outdated. Internet access is possible almost always and everywhere.	See item 6. Consider to delete one of these items.	“I never really look forward to it, but […] I take it for granted.”
		The anticipation to use the Internet refers to specific activities or social contacts maintained online. This does not reflect a strong desire to use the Internet in a way that leads to negative consequences.	See above.	“That happens […] often because I'm actually looking forward to talk to my colleagues again in the evening.”
		The anticipation to use the Internet refers to end unpleasant work or activities and use the Internet as a “normal” leisure activity to relax.	See above.	“After a long day, I actually always look forward to it […] I sit down on the couch, play a little bit. I say […] often”
		The term “Internet session” seems difficult to understand.	See above.	“What does Internet session mean? […] Is it about WhatsApp? Writing a message? Or just watching a movie or playing games? […]”
(8) Do you think you should use the Internet less often?	Conflict	No relevant comments. The item was easy to understand.
(9) Have you unsuccessfully tried to spend less time on the Internet?	Loss of control	It is unclear if the Internet use leads to negative consequences in daily life.	Consider to reword and specify this item (e.g., “How often have you unsuccessfully tried to spend less time on the Internet because you have noticed negative consequences of your Internet use?”)	“It happened a lot. Considering that I've been on the Internet almost daily since I was 14 years old […] it often happened that I didn't managed to spend less time on the Internet.”
		Overall, participants had significant difficulties to understand the content of the item.	Consider to rephrase the item.	“I don't understand the question.” “Unsuccessful, I would take rarely […] presupposing that I have [… tried to reduce] I have rarely done that. […] I think that's […] strange.”
(10) Do you rush through your homework/schoolwork to use the Internet?	Conflict	Participants relate to the general avoidance of unpleasant tasks but not to the Internet per se.	Consider to rephrase the item (e.g., “How often do you rush through activities you actually enjoy to get on the Internet?”).	“That happens a lot. […] I vacuum faster […] to write again […] with other people […]. It's not necessarily “the Internet” but rather the free time that you have again. So, I would say […] that's often the case.”
(11) Do you neglect your daily obligations (work, school, or family life) because you prefer to use the Internet?	Conflict	Increased risk of overpathologizing: It is not clear if the Internet use leads to significant impairment of daily life.	Consider to assess the temporal context and negative consequences more clearly (e.g., “How often do you experience negative consequences or neglect responsibilities because you prefer to use the Internet?”).	“I think maybe I should watch a movie with my parents […] instead of being online […]. But […] their interests are different. I would say […] sometimes.”
(12) Do you use the Internet when you are feeling sad?	Coping/Mood modification	Participants tend to relate to functional emotional regulation strategies (e.g., social support they receive online).	Consider to rephrase the item and add a temporal reference (e.g., “How often do you use the Internet when you feel sad to forget about your problems?“)	“[…] when something doesn't go the way it should […] you go online for a while […] you have your friends there and chat with them […] they usually calm you down […].”
(13) Do you use the Internet to escape from sorrows or negative feelings?	Coping/Mood modification	Strong similarity to Item 12.	Check for redundancy; possibly delete one of these items.	“That's exactly the same.“ “For me, this is similar to question 12 […] I will answer question 13 the same way I answered number 12.”
(14) How often do you feel anxious, frustrated, or irritated when you are not able to use the Internet?	Withdrawal	Despite high scorings of this item, participants do not refer to “withdrawal” but rather to technical problems.	Consider to specify the instruction of the CIUS, e.g., “In your answers, please do not refer to technical problems.“	“When the Internet router breaks down […] it's annoying […] that's why […] I'm […] very often irritated.”
		Participants had difficulties to respond to this item. Nowadays, access to the Internet is possible almost always and everywhere.	Consider to specify the situation, e.g., “[…] you are not able to use the Internet because of daily obligations (e.g., school, work, or family life).”	“[…] I always have access to the Internet through my mobile phone. For me, such situations don't exist.”
		Increased risk of overpathologizing: Participants refer to situations when important social concerns need to be clarified via digital channels.	Consider to rephrase the item or the instruction of the CIUS to make sure that participants do not refer to (constructive) problem-solving approaches in real life.	“I feel restless when […] I have to clarify something important, and I can't answer my mobile phone or use the Internet.”
Overall findings in the CIUS		Participants' answers strongly depend on the respective Internet activity.	Consider to assess the main activity while using the Internet in the instruction (e.g., “Please name the Internet application you use most frequently at the moment: _______. When answering the items, please refer to the application you named above.”	“It depends […] when I play video games, I stop. WhatsApp, Instagram […] I actually continue.”
		The time frame was unclear.	Consider to specify the time frame in the instruction.	“Is that […] related to the current situation or […] the past?”

10 Item-Internet Gaming Disorder Test (IGDT-10)

The IGDT-10 is a commonly used screening instrument to assess pathological Internet gaming behavior (Király et al., 2019). Its structure is based on the nine IGD criteria as suggested in the DSM-5 (Király et al., 2019). Each criterion is operationalized by a single item, apart from the last criterion “jeopardizing or losing a significant relationship, job, or educational or career opportunity because of participation in Internet games”. This item was operationalized by two items because of its complexity. All items can be answered on a 3-point Likert scale labeled 0 (“never”), 1 (“sometimes”), and 2 (“often”). In previous studies, an unidimensional factor structure provided a good fit to the data in cross-cultural samples (Király et al., 2019). In a Chinese sample, the IGDT-10 was validated with a structured interview (Chiu, Pan, & Lin, 2018). Good internal reliability criteria (Cronbach's α = 0.85) and an adequate diagnostic efficiency (area under the curve = 0.81) were found (Chiu et al., 2018). In a Finish sample, the IGDT-10 exhibited a good internal consistency (Cronbach's α = 0.87) and acceptable standardized item loadings (Männikkö, Routsalainen, Tolvanen, & Kääriäinen, 2019). All items of the IGDT-10 are listed in Table 3.

Table 3.

Internet Gaming Disorder Test - 10 Items (IGDT-10)

IGDT-10 instruction: “Please read the statements below regarding video gaming. The questionnaire refers to video games (both online and offline, played on any platform). Please indicate on a scale from 0 to 2 (0 = never, 1 = sometimes, 2 = often) to what extent and how often these statements applied to you over the past 12 months.”
Item	Criterion (Király et al., 2019)	Difficulties	Suggestions for improvement	Statements of the participants
(1) When you were not playing, how often have you fantasized about gaming, thought of previous gaming sessions, and/or anticipated the next game?	Preoccupation	Participants do not refer to the criterion “preoccupation” but rather to gaming as a leisure activity with friends or to boredom without a negative impact on daily life.	Consider to define the situation more clearly (e.g., “[…] although you have other important things to do?”).	“Of course, because […] you can relax, meet your friends without leaving your house […].” “I often think back to LAN parties with friends […].” “Actually, I always look forward to such things when I have nothing else to do […].”
		The term “imagine playing games“ seems too abstract.	Consider to delete this term.	“Why should I imagine […] oh yeah […] a controller would be really nice in my hand […] (laughs)?”
(2) How often have you felt restless, irritable, anxious, and/or sad when you were unable to play or played less than usual?	Withdrawal	No relevant comments.
(3) Have you ever in the past 12 months felt the need to play more often or played for longer periods to feel that you have played enough?	Tolerance	Participants often refer to a temporary desire to finish a story (particularly when they had to pay for the game).	Rephrase the item and define the situation more clearly (“[…] although you have other important things to do?”)	“I want to finish the game and then put it aside because otherwise I feel like I've wasted the money.” “Sometimes when a game is very captivating […] you want to reach a certain level or the end of a campaign […] just to see an end or the progress of a story […].”
(4) Have you ever in the past 12 months unsuccessfully tried to reduce the time spent on gaming?	Loss of control	Participants' reported Internet use behavior does not lead to negative consequences in daily life.	Rephrase the item to assess more clearly if the gaming behavior leads to negative consequences (e.g., “[…] because you have noticed negative consequences?”)	“When my friends are online […] I want to play with them and ask myself: ‘What else should I do?’ And I have nothing to do anyway. It doesn't have a negative effect on me anyway, I can do it, as long as everything else is going on, I can go online. […] So yes, sometimes.”
(5) Have you ever in the past 12 months played games rather than meet your friends or participate in hobbies and pastimes that you used to enjoy before?	Giving up other activities	Participants refer to positive social gaming experiences. The desire to play online games might be normative and socially determined.	Define the situation more clearly (e.g., “[…] with friends outside of online activities”).	“My friends […] are online as well […] this connects us.”
(6) Have you played a lot despite negative consequences (for instance losing sleep, not being able to do well in school or work, having arguments with your family or friends, and/or neglecting important duties)?	Continuation	The item was perceived to be long and complicated.	Consider to shorten the examples.	“The question is too long. I'm trying to read the question again.” “Boah, this is lengthy.”
		There are various aspects in one question. Participants tend to build mean values across all options instead of considering whether at least one of the mentioned options applies to them.	In the instruction of the IGDT-10, point out that referring to at least one option is sufficient.	“Lack of sleep actually quite often because […] I prefer to game instead of sleeping. I'm not sure about the loss of performance […]. I'm still good in sports. […] I don't really have any duties […]. So I can't say that I've lost much. I would say […] sometimes.”
(7) Have you tried to keep your family, friends, or other important people from knowing how much you were gaming or have you lied to them regarding your gaming?	Deception	Participants refer to financial aspects of gaming.	Consider to extend the item (e.g., “[…] to hide how much time or money you spent with gaming […]?”).	“ […] invested with money but that question is related to time […].”
(8) Have you played to relieve from a negative mood (for instance helplessness, guilt, or anxiety)?	Escape	Participants played games to temporarily distract from negative emotions. However, despite high scorings they usually don't play games to avoid problem solving or functional emotional regulation in a long term.	Rephrase the item and define situation more clearly, e.g., “[…] to escape from negative mood […] in a long term?”.	“[…] to forget about my negative thoughts for a while […] and deal with the problem afterwards.”
(9) Have you ever in the past 12 months risked or lost a significant relationship because of gaming?	Negative consequences	Participants mainly refer to romantic relationships. They did not consider a potential negative impact of problematic gaming on family members or friends.	Consider to define the term “relationship” more clearly.	“I don't know how to interpret the term relationship […] whether it's really a romantic relationship or […] a friendly relationship.” “Not in the last twelve months. I was single.” “No […]. At the moment, I'm single.”
(10) Have you ever in the past 12 months jeopardized your school or work performance because of gaming?	Negative consequences	No difficulties. The item was easy to understand.
Overall findings of the IGDT-10		Offering only three answer options (never, sometimes, always) was perceived to be undifferentiated.	Consider to add more answer options.	“I take sometimes because never is a lie. But sometimes sounds like a lot […] I'll take sometimes anyway because there is nothing in-between […].”

Bergen Social Media Addiction Scale (BSMAS)

The popularity of online social networks changes quickly. Thus, screening instruments addressing specific social networks (e.g., Facebook or Instagram) lose their up-to-dateness easily. The BSMAS is an adoption of the Bergen Facebook Addiction Scale (Andreassen, Torsheim, Brunborg, & Pallesen, 2012) and considers social networks as a general construct. The BSMAS includes six items that refer to the core components of behavioral addiction as proposed by Griffiths (2005): Salience, mood modification, tolerance, withdrawal, conflict, and relapse. All items can be answered on a 5-point Likert scale ranging from “never” (0) to “very often” (4). The BSMAS has a good fit of the model to the data across different gender and age groups, thus confirming the single-factor structure of the instrument (Andreassen et al., 2012). All items of the BSMAS are listed in Table 4.

Table 4.

Bergen Social Media Addiction Scale (BSMAS)

BSMAS instruction: ”Below you find some questions about your relationship to social media and your use of social media (Facebook, Twitter, Instagram, …). Choose for each question the response that describes you best. How often during the last year have you…”
Item	Criterion (Andreassen et al., 2012)	Difficulties	Suggestions for improvement	Statements of the participants
(1) […] spent a lot of time thinking about social media or planned use of social media?	Salience	High scorings rarely correspond to the underlying criterion “salience”. Participants rather thought about specific content or news they saw online or interactions with other people on social media.	Consider to explore if social media use has a negative impact on participants' daily life (e.g., “[…] although you have other important things to do?”).	“When something happens and it's in the news and I see it on Facebook […] I think about it.” “Of course […] when you have a conflict with someone or maybe you're planning a meeting […]. Then you probably think about it a lot.”
		The term “planning” seems too abstract in the context of social media use.	Consider to delete the complete term “planning to use social media”.	“I don't plan it. […] To plan […] that's strange with social media.”
(2) […] felt an urge to use social media more and more?	Tolerance	Participants thought about important social interactions without significant impairments in daily life.	See above (e.g., “[…] although you have other important things to do?”).	“This particular person I'm writing with, online right now […] we are talking about something important. So, I would say […] it happened more often that I really felt the urge to do so.”
(3) […] used social media to forget about personal problems?	Mood modification	High scorings were often associated with functional emotional regulation strategies, e.g., seeking social support via social media.	Reword the item to make sure that participants do not refer to functional emotional regulation strategies.	“[…] only to contact friends and talk about the problems and forget about them.”
		Participants had difficulties to define the term “personal problems”.	Consider to clarify this term.	“It depends on how you define personal problems […] if small things like boredom are part of it […] then sometimes to […] often.”
(4) […] tried to cut down on the use of social media without success?	Relapse	Participants had difficulties to define the term “cut down”.	Consider to clarify the term (e.g., “[…] significantly cut down your social media use to feel confident?”)	“What does cut down mean? […] Is it about a few minutes? Is it about hours?”
(5) […] become restless or troubled if you have been prohibited from using social media?	Withdrawal	Participants tend to refer to long-term technical problems.	Consider to specify the instruction of the BSMAS (e.g., “In your answers, please do not refer to technical problems”).	“If our Wi-Fi […] doesn't work, you get a bit anxious. Because you can't do anything […] if you don't have Internet access for two or three weeks.” “When I was at school and I forgot my phone at home […] I was worried because I lived in the village and I couldn't get away there myself.”
		Participants rarely correspond to the underlying criterion “withdrawal” but rather to important social events that happen in real-life.	Consider to specify the context (e.g., “[…] when you couldn't use social media because of daily obligations?”)	“The other day I confessed my love to my best buddy […]. I was a bit worried that he doesn't answer me […].”
(6) […] used social media so much that it has had a negative impact on your job/studies?	Conflict	No relevant comments. This item was easy to understand.
Overall findings of the BSMAS		The time frame of 12 months seems too long. Participants usually refer to the past weeks.	Consider to shorten the time frame.	“I think a year is such an insanely long period […]. I couldn't summarize something like that.”
		The instruction “How often in the last year have you” is mentioned once above all items. Many participants skipped the instruction and were confused about the time frame when they tried to answer the items.	Consider to include the time frame in the beginning of each item.	“Hard to answer because I don't know what time frame is intended […].” “Oh, up there, I didn't even read this […].”

Think-aloud interviews

Despite the psychometric quality of the CIUS, the IGDT-10, and the BSMAS has been tested frequently in previous studies, little is known about the content validity of these screening instruments. When developing a new screening instrument, the wording of each item should be reviewed for its clarity and relevance to the underlying psychological construct (American Educational Research Association [AERA] et al., 2014). In the field of IUD, such reviews have often been carried out through expert ratings (e.g., King, Billieux, Carragher, & Delfabbro, 2020). However, experts may fail to recognize typical problems of lay individuals, who often interpret items in unexpected ways. Lay individuals may experience difficulties in understanding the item's content, recalling necessary information from the memory, reflecting and judging their own Internet use, and finally choose an answer (AERA et al., 2014; Presser et al., 2004; Ryan, Gannon-Slater, & Culbertson, 2012). Each step is a potential source of errors which directly impairs the screening's validity.

To investigate the validity of screening instruments in a thorough manner, it is of importance to analyse cognitive processes underlying responders' answers. In the broad spectrum of cognitive methods in psychological research, think-aloud interviews are an established approach to make such cognitive processes more explicit (Ryan et al., 2012; Wolcott & Lobczowski, 2021). Think-aloud interviews are conducted under standardized conditions and follow a strict protocol (Ryan et al., 2012). After receiving instructions and comprehensive training on how to “think-aloud”, participants are asked to spontaneously verbalize all thoughts that come to their mind while answering a screening instrument (Ryan et al., 2012). To avoid potential bias in participants' response behavior, the interviewing process must be non-intrusive (Ryan et al., 2012). The interviewer's role is to observe the interview process and remind the participant to constantly verbalize all cognitions (Ryan et al., 2012). As a result, problem-solving processes can be objectively recorded, transcribed, and content-analysed (Presser et al., 2004). Other cognitive methods, such as verbal probes, are typically more directive and intrusive by asking targeted interview questions (Ryan et al., 2012). Summing up, think-aloud interviews are an appropriate method for testing the content validity of screening instruments by lay persons. The feasibility of think-aloud interviews has already been tested in samples of adolescents in the field of substance-related disorders (e.g., Nehlin, Carlsson, & Öster, 2020). In the field of IUD, such in-depth analyses of cognitive processes underlying participants' answers have not yet been carried out.

Aim of this study

This study aimed to investigate the content validity of the CIUS, the IGDT-10, and the BSMAS by analysing young adults' cognitive processes when reporting and valuing their Internet use behavior. Based on clinical impressions, it is hypothesized that several items may have the potential of leading to misconception or overpathologizing of unproblematic Internet use. Since the CIUS, the IGDT-10, and the BSMAS have been developed many years ago and Internet applications are changing fast, it is further hypothesized that the wording is outdated which results in reduced content validity. Our findings might contribute to a better understanding when interpreting quantitative results of other studies. Besides, it may provide recommendations for possible improvements of these screening instruments.

Methods

Participants

All participants were part of the study “Intervening in Problematic Internet use“ (iPIN; Schmidt et al., submitted for publication). The iPIN study aimed to investigate the efficacy of a brief motivational intervention for adolescents and young adults with at-risk Internet use. Participants were proactively screened for problematic Internet use with the CIUS in 12 vocational schools in northern Germany. The tablet-assisted screening took place between March 2018 and March 2019. Inclusion criteria were sufficient knowledge of the German language and a minimum age of 16 years. In total, 8.230 screenings were realized and 1.475 participants with at-risk use of the Internet agreed to be contacted for telephone-based diagnostic baseline interviews. In the baseline interview, IUD was assessed using the structured, clinical interview “Internet-related disorders-Clinical Assessment Tool (I-CAT)”. I-CAT was developed by the iPIN project team and is based on the structure of the Munich Composite International Diagnostic Interview (M-CIDI; Wittchen et al., 1995) which is a fully standardized diagnostic interview to determine mental disorders following the ICD-10 Diagnostic Criteria for Research. Independent from the time spent on the Internet, I-CAT covers all nine criteria for IUD in line with the DSM-5 diagnostic criteria of IGD. Of 937 realized interviews, 497 participants fulfilling at least two criteria for IUD in the last three months were randomly assigned either to the intervention or the control group. The intervention group received up to three telephone consultations. The control group received a brochure with recommendations on how to modify problematic behavior on the Internet. Telephone-based follow-up interviews were conducted after five months (n = 301) and ten months (n = 284). After completing the second follow-up interview, participants were asked to take part in individual think-aloud interviews. In total, 197 participants agreed to take part in the think-aloud interview. Of these, 30 individuals who were stratified by gender, the main activity on the Internet, and the number of IUD criteria in the second follow-up interview were contacted. The sample size was based on reflections on the study aim, sample specificity, and analyses strategy. If eligible individuals could not be contacted or did not appear for the interview, they were invited once again. One potential participant who did not appear twice was excluded. Another participant of those who agreed to take part in the think-aloud study was recruited. The mean time between the second follow-up and the think-aloud interview was 66.6 (SD = 55.8) days with a range from 9 to 175 days. All participants received an incentive of 50 Euros for participation in the think-aloud interview.

Procedure of the think-aloud interview

The interviews were based on the think-aloud approach (Wolcott & Lobczowski, 2021) and a semi-structured interview guide. The BSMAS and IGDT-10 were translated into German. Double back-translation was performed to avoid language bias. Prior to the interview, three exercises were performed to practice “thinking-aloud”. For example, participants were asked to think aloud while they imagined going through their home and counting all windows (“Visualize the place where you live, go slowly through every room and count all windows”). To facilitate this process and to lower potential barriers of thinking aloud, all exercises were demonstrated by the interviewers in advance.

Once these exercises were complete, the interviewer asked the participants to engage in the same think-aloud process as they answer each item of the following screening instruments. First, all items of the CIUS were rated by all participants. While verbalizing all thoughts, participants were asked to fill in the items on the screening sheets they had available in front of them. Depending on their main Internet activity, participants were given paper versions either of the IGDT-10 (those with online games as main Internet activity) or the BSMAS (those with social networks as main Internet activity). Again, participants rated all items either of the BSMAS or of the IGDT-10 while verbalizing all thoughts. Participants were asked to not reflect their thoughts but to report their thoughts concurrently. Besides, they were reminded that this study aimes to understand cognitive processes rather than analysing the item rating itself.

Quality assurance

All interviews (20–30 min) were conducted by trained project members (HS, SS, AB, NM) and took place between January and February 2020. All interviewers received standard operating procedures (SOP's) to ensure the objectivity of the implementation. Prior to the main data collection, three test interviews were conducted with students of the University of Lübeck. Afterwards, minor adjustments were made in terms of the wording and length of the instruction. Weekly project meetings and supervisions with all interviewers, student assistants, and at least one of the senior researchers of the working group (DB, AB, HJR) were conducted to discuss potential difficulties during the interview process.

Analysis

All interviews were audio-recorded, verbatim transcribed, and content-analysed. The transcripts were initially coded by at least two members of the project team, independently from each other. A-priori categories were developed deductively in the project team and named as follows: Quality of statement, comprehension, susceptibility to interference, similarity to other items, and indication of Internet-related problems. The further analysis was an iterative process focusing on general problems while answering the screening instrument. Unclear cases were discussed in detail in the project team with all interviewers and senior researchers until a consensus was formed.

Ethics

The study procedure was carried out in line with the Declaration of Helsinki. It was approved by the ethics committee of the University of Lübeck on 24th September 2019. All participants were informed about the purpose of the study and all provided informed consent.

Clinical trial registration

The interviews were an additional project of the iPIN study that was registered on August 24th, 2018. The clinical trial registration of the iPIN study is available here: https://clinicaltrials.gov/ct2/show/NCT03646448?term=iPIN&draw=2&rank=1.

Results

Sample characteristics

Of 30 participants, 11 participants reported online games as their main Internet activity and filled out the IGDT-10. Besides, 18 participants mainly used social networks and filled out the BSMAS. One participant reported to use online games and social networks equally. Therefore, he filled out both screening instruments (BSMAS and IGDT-10). The sample characteristics are shown in Table 1.

Table 1.

Sample characteristics

	Total1 (N = 30)	BSMAS (n = 19)	IGTD-10 (n = 12)
Sociodemographic variables
Age, M (SD)	21.3 (2.1)	21.1 (2.1)	21.6 (2.1)
Female gender, n (%)	15 (50.0)	15 (78.9)	0 (0.0)
Residental situation
Alone, n (%)	5 (16.7)	2 (10.5)	3 (25.0)
With parents/grand-parents, n (%)	19 (63.3)	12 (63.2)	8 (66.7)
With partner, n (%)	1 (3.3)	1 (5.3)	0 (0.0)
Shared flat, n (%)	5 (16.7)	4 (21.1)	1 (8.3)
Vocational school situation
Still in vocational school, n (%)	18 (60.0)	12 (63.2)	6 (50.0)
Already finished vocational school, n (%)	10 (33.3)	5 (26.3)	6 (50.0)
Dropped out of vocational school, n (%)	2 (6.7)	2 (10.5)	0 (0.0)
Partnership, n (%)	12 (40.0)	9 (47.4)	3 (25.0)
Internet-related variables
Number of fulfilled IUD criteria
0–2 (unproblematic Internet use), n (%)	16 (53.3)	10 (52.6)	7 (58.3)
3–4 (problematic Internet use), n (%)	8 (26.7)	4 (21.1)	4 (33.3)
5–9 (pathological Internet use), n (%)	6 (20.0)	5 (26.3)	1 (8.3)

Notes: ¹ In total, 30 participants were interviewed. One participant reported to use online games and social media equally and completed both screening instruments (BSMAS and IGDT-10). All sociodemographic data and the number of fulfilled IUD criteria were collected in the second follow-up interview of the iPIN study.

Results of the think-aloud interviews

General difficulties when answering the screening instrument are presented separately for the CIUS, the IGDT-10, and the BSMAS. Specific difficulties of each item and suggestions for improvement are presented in Tables 2–4.

CIUS

The CIUS instruction (“Chose the answer that best applies to you”) was perceived to be unspecific and caused several comprehension problems. Participants were not sure which time frame (“Is this about the last few months or the last year?”) or Internet activity should be considered (“I'm wondering. I associate being online with WhatsApp or Facebook […] I don't know if that was meant here?”). Items aimed to assess excessive Internet use as a dysfunctional emotional regulation strategy were often rated with high scorings. However, when analysing participants' cognitions underlying their scoring, we rarely found evidence for dysfunctional emotional regulation strategies. Despite the Internet was temporarily used to distract from daily problems, participants did not generally avoid specific emotions (e.g., sadness, anger) as suggested in the underlying DSM-5 criterion (“If something went wrong, you go online for a while, meet your friends there […] and calm down”). High scorings on items aimed to assess cognitive involvement (e.g., “Do you think about the Internet, even when you are not online?” or “Do you already look forward to your next Internet session?”) were often related to a lack of leisure alternatives or boredom. Besides, participants report that they were waiting for important news and social interactions. This does not necessarily reflect problematic Internet use. Besides, several items were perceived as redundant (“I compare that to the first question, they definitely look similar to me”). In contrast, items aimed to assess impairments caused by dysfunctional Internet use (e.g., insufficient sleep) were merely well understood. A detailed overview of participants' difficulties when answering the CIUS and suggestions for improvement are presented in Table 2.

IGDT-10

The 3-point Likert scale of the IGDT-10 was perceived to be undifferentiated. In cases of uncertainty, participants often chose the middle response category (“I take sometimes because never would be a lie. But sometimes sounds like a lot […] I'll take sometimes anyway because there is nothing in-between […].”). Furthermore, several items were perceived to be long and complicated. Offering multiple options (“Did you continue playing despite negative consequences, e.g., negative impact on sleep or performance at school or work, arguments with family or friends, and/or neglect of important responsibilities”) has led to the building of mean values across all options instead of reflecting if at least one option is true. High scorings of items underlying the criterion craving were often related to unproblematic use, e.g., the anticipation of new game releases or LAN parties with friends. Besides, participants reported that the IGDT-10 tends to overpathologize (“[…the IGDT-10] is pretty focused on negative things […] not things like finding friends on the Internet, maintaining contact with friends online, such things are not included, that's […] what I do on the Internet. I find new people with whom I can talk, I […] keep in touch simply via the Internet. I've met really cool friends just through the Internet. I've never met them, but we even know what we look like, we know each other REALLY well. And that […] is really missing in this questionnaire […] positive aspects of the Internet […].”). Table 3 shows all occurring difficulties while answering the IGDT-10 and suggestions for modifications.

BSMAS

Following the original version of the BSMAS, the introduction (“How often in the last year have you…”) was mentioned only once in the introduction. Participants repeatedly skipped the information about the time frame (“So, first question […] oh […] How often in the last year […] I almost missed it.”). Furthermore, reflecting the Internet use for one year was perceived to be difficult (“One year is such a long period. It's insanely difficult to summarize.”). Participants were not sure what social media applications should be considered (“I'm wondering: Social media […] what does that term include? […] Snapchat, Instagram, WhatsApp. I'm not sure if that's part of it?”). Participants did not refer to dysfunctional emotional regulation strategies but to long-term technical problems (“When I was at school, and I forgot my phone at home, or there was not enough battery, I was worried because I lived in the village, and I couldn't get away there myself.”). This does not reflect addictive behavior. In some cases, even functional emotional regulation strategies (e.g., seeking social support online) were associated with high scorings (“[…] only to contact friends and talk about the problems […]”).

Despite most items were well understood, some specific wordings seem rather unusual or too abstract. Participants reported that their social network use was not planned but simply took place (“If I want to, I'll use social media, but I don't plan it. […] The word planning […] that's strange with social media.”). Additionally, participants thought about the content they saw online rather than the use of the social networks, e.g., real-life social events that were planned via social media. Table 4 presents details and possible modifications of the BSMAS for each item separately.

Discussion

Despite the constant development and publication of new screening instruments for IUD, to date, little is known about their content validity. In this study, we analysed cognitive processes of young adults when valuing their Internet use behavior with the three established screening instruments (CIUS, IGDT-10, and BSMAS). The aim was to identify potential mismatches between items' intention and participants' comprehension of these items. Based on clinical impressions, it was hypothesized that some items tend to overpathologize young adults' Internet use. Besides, it was hypothesized that the wording of some items is outdated which might lead to misperceptions.

Across all screening instruments, we found that high scorings were often not congruent with the underlying diagnostic criteria for problematic Internet use. These mismatches were particularly noted for items assessing Internet use as a dysfunctional emotional regulation strategy and cognitive involvement. This finding is in line with an expert rating analysing the face validity of screening instruments for problematic online gaming (King, Billieux, et al., 2020). Despite high scorings, participants relate to Internet use as a “normal” leisure activity without negative impairments in daily life. Besides, participants thought about specific content they saw online or about real-life relationships maintained online but not about “the Internet” per se. All in all, such discrepancies between items' intention and participants' comprehension of these items may lead to an increased risk of overpathologizing leisure activites of young adults and may generate false-positive results. In the light of the global Covid-19 pandemic, in which many leisure activities necessarily took place online, careful attention should be paid to ensure that items do not misjudge normal or even functional behavior of young adults (e.g., the fulfillment of social needs via online platforms). In upcoming revisions of these screenings instruments, the context and potential negative impairments caused by dysfunctional Internet use should be clarified, e.g., by adding “even if you have other important things to do” to the original item “How often do you find it difficult to stop using the Internet when you are online?”. Alternatively, the instruction text of the screening instrument could highlight functional impairments caused by excessive Internet use: “In your answers, please consider if your Internet use behavior leads to neglecting important obligations (e.g., friendships, family, work, or school)”. To further avoid false-positive results, the instruction of the screening instrument should ensure that participants do not refer to technical problems.

Besides these mismatches between items' intention and participants' comprehension of items, participants often experience difficulties with the time frame. Both, the fact that no time frame was given (CIUS) and long time frames of 12 months (IGDT-10, BSMAS) caused difficulties. In line with recommendations to improve screening instruments in other psychological fields (Meerwijk & Weiss, 2017), it should be discussed to shorten the time frame of the IGDT-10 and BSMAS. Despite both screening instruments have defined a time frame of 12 months, participants still tend to refer to situations they had previously encountered or chose a rough estimate. It is further recommended either to integrate the time frame at the beginning of each item or to highlight the time frame more clearly in the instruction with capatial letters. Apart from the unclear time frame, participants were not sure to which application they should refer to when answering the CIUS or the BSMAS. Thus, it might be helpful to ask for the main Internet application in the instruction.

Another finding was that several items were perceived outdated or incomprehensible. Rereading or stumbling are important indicators of an invalid responding in think-aloud interviews (Darker & French, 2009). Such phenomena are highly associated with choosing the middle response option in case of uncertainty (Darker & French, 2009). In this study, these phenomena occurred with very long items or those with many examples in parentheses. It is recommended to shorten items that were considered too long and complicated. Furthermore, screening instruments for IUD might need to account for cultural and social differences. In line with an other think-aloud approach in a sample of adolescents answering screening instruments for alcohol use (Nehlin et al., 2020), we found that outdated wordings that are not commonly used in this age group might lead to misconceptions (e.g., “planning to use social media” in the BSMAS). In upcoming revisions, adolescents and young adults should be included in the item development process. Redundancies should be avoided. Particularly in the CIUS, participants experienced difficulties because of strong similarities (e.g., Item 1 and Item 2).

Summing up, these findings may contribute to a better understanding about how young adults perceive and interpret self-report screening instruments for IUD that are commonly used in both research and in clinical work. These findings may further help to explain heterogeneous prevalence rates for IUD that may strongly depend on the utilized screening instrument. Self-report screening instruments are not developed for diagnosing a specific disorder. It is important to be aware that individuals may fulfill symptoms above a specific cut-off but still not meet criteria for clinically relevant distress or functional impairments caused by dysfunctional Internet use. The rationale of IUD screening instruments based on self-ratings is to identify individuals at risk for problematic or even pathological Internet use behavior to refer them to professionals for in-depth clinical interviews. To further improve the content validity, it seems necessary to evaluate the CIUS, the IGDT-10, and the BSMAS in both clinical and general populations with different ages and in addition with in-depth clinical diagnostic interviews.

Limitations and strengths

Several limitations should be addressed when interpreting the results. In general, the quality of think-aloud interviews depends on the participant's ability to articulate own thoughts while answering a screening instrument. Specific personality traits (e.g., introversion) and social desirability might impair this process. Besides, only male participants reported online games as main activity on the Internet. Therefore, the content validity of the IGDT-10 was analysed in men but not in women. Conversely, our sample allows limited conclusions about the comprehension of the BSMAS from the perspective of men. Another limitation is the small sample size (n = 30) that limits the generalizability of the results. However, this study had an exploratory intention and does not claim to provide a representative sample. The sample size was carefully considered in line with other qualitative approaches and think-aloud studies (e.g., Blair & Conrad, 2011). Results of such qualitative studies help to reword and improve screening instruments. In the next step, the revised version could be validated in the context of large samples. Despite these limitations and to the best of our knowledge, this is the first study aimed to investigate the content validity of commonly used screening instruments for IUD. The think-aloud approach provides less distraction or bias in terms of social desirability compared to “classic” question-answer interviews. The sample consists of individuals with a heterogeneous amount of fulfilled criteria for IUD which provides as much representativity as possible. Nevertheless, replications in unselected participants would be helpful to add further findings of individuals without prior screening for problematic Internet use.

Conclusion

This study provides insights into young adults' cognitive processes when responding to the three commonly used screening instruments for problematic Internet use behavior. We identified four main sources for errors: (1) High scorings were often not congruent with the underlying diagnostic criteria for problematic Internet use. This increases the risk of overpathologizing “normal” Internet use of young adults. In particular, such discrepancies were found in items assessing Internet use as a dysfunctional emotion regulation strategy and cognitive involvement. (2) The instructions of the screening instruments were unclear. Participants were not sure to what time frame or Internet application they should relate. (3) Long periods of 12 months and complex items with many examples lead to difficulties. As a result, participants tend to build mean values or chose the middle answer category. (4) Several wordings were perceived to be outdated and incomprehensible. All in all, these findings might help to generate recommendations on how to revise and clarify several items. As a main indication for upcoming revisions, items should separate more clearly between “normal” Internet use without negative consequences and Internet use that leads to functional impairments in daily life. To further improve the content validity, it might be helpful to include a heterogeneous sample of adolescents and young adults with and without IUD in the process of item development.

Funding sources

The think-aloud interviews were part of the iPIN-study that was funded by the German Federal Ministry of Health (Grant number ZMVI1-2517DSM210).

Authors’ contribution

HS: Study concept and design, gathering of screening data in vocational schools as part of the iPIN study, conduction of baseline telephone interviews and conduction of the think-aloud interviews, interpretation of the findings, preparation of the manuscript draft. DB: Study concept and design, major analysis and interpretation of the findings, revision of the manuscript. AB: Study concept and design, conduction of think-aloud interviews, analysis and interpretation of the findings, revision of the manuscript. GB, SH: Study concept and design, analysis and interpretation of the findings, revision of the manuscript. SB: Conceptual support and revision of the manuscript. HJR: PI, study concept and design, obtained the conduction of the study, analysis, and interpretation of findings, revision of the manuscript.

Conflict of interest

All authors declare that they do not have a conflict of interest.

Acknowledgment

We would like to thank all vocational students who have agreed to participate in the study. Furthermore, we thank all students and project team members who supported us with the conduction and the analysis of all think-aloud interviews: Clara Becker, Fanny Lindner, Nina Meyer, Lisa Feldhoff, Svenja Sürig, and Charlotte Miriam Tiemann.