Development and testing of a 3-item screening tool for Problematic Internet Use

Abstract

Introduction

To develop and validate the PRIUSS-3 screening scale, a short scale to screen for Problematic Internet Use.

Methods

This scale development study applied standard processes using separate samples for training and testing dtatasets. We recruited participants from schools and colleges in 6 states and 2 countries. We selected 3 initial versions of a PRIUSS-3 using correlation to the PRIUSS-18 score. We evaluated these 3 potential screening scales for conceptual coherence, factor loading, sensitivity and specificity. We selected a 3-item screening tool and evaluated it in two separate testing sets using receiver operating curves (ROCs).

Results

Our study sample included 1079 adolescents and young adults. The PRIUSS-3 included 3 items addressing: 1) anxiety when away from the internet, 2) loss of motivation when on the internet, and 3) feelings of withdrawal when away from the internet. This screening scale had a sensitivity of 100% and specificity of 69%. A score of 3 or greater on the PRIUSS-3 was the threshold to follow-up with the PRIUSS-18.

Discussion

Similar to other clinical screens, the PRIUSS-3 can be administered quickly in a clinical or research setting. Positive screens should be followed by administering the full PRIUSS-18. Given the pervasive presence of the internet in youth's lives, screening and counseling for PIU can be facilitated by use of this validated screening tool.

Introduction

Problematic Internet Use (PIU) is an emerging health concern among US adolescents and young adults (AYAs). A 2012 study developed a conceptual framework for PIU using empirical data and defined it as “Internet use that is risky, excessive or impulsive in nature leading to adverse life consequences, specifically physical, emotional, social or functional impairment.¹” A study of high school adolescents estimated prevalence of PIU to be 4%.² Moreover, three recent studies sampling university students estimated the prevalence to be between 4-6%,^3-5 a rate comparable to other mental health conditions such as depression that are often addressed clinically among this population.⁶ PIU has been associated with both social and health consequences including poor academic performance, stress, and fewer positive health behaviors.⁴ Longitudinal studies have also suggested bidirectional relationships between PIU and other mental health conditions such as depression.^7-9

Given that internet use begins increasingly earlier in adolescence and even into childhood, primary care physicians are uniquely positioned to conduct early screening for PIU.¹⁰ The Problematic and Risky Internet Use Screening Scale (PRIUSS)¹¹ was developed based on the PIU conceptual framework.¹ The PRIUSS was validated for use among AYAs in English and Dutch. The PRIUSS has 18 items and 3 subscales: social impairment, emotional impairment, and risky/impulsive internet use. A short screening tool could fit into existing adolescent screening tools, and be followed by a PRIUSS-18 for those that screen positive. A similar screening approach has been applied for depression, the Patient Health Questionnaire 2-item (PHQ-2)¹² screen is used for initial screening, followed by the 9-item PHQ-9 for those that screen positive.¹³ The purpose of this study was to develop and test a shorter version of the PRIUSS-18 to create the PRIUSS-3.

Methods

Participants

Consistent with standard scale development procedures,¹⁴ we recruited participants for separate data collections to serve as the training and testing datasets for our analyses. Participants for both the training and testing datasets were AYAs aged 18 through 23 years. In order to evaluate the performance of the selected items on a younger adolescent population, we recruited an additional population for a second supplementary testing data set of adolescents aged 13-16 years.

Training set

The goal of the training set recruitment was to achieve a multisite sample to represent possible regional or school variations in technology use habits. College students were recruited from 4 college campuses in the Midwest and the West for a single cross-sectional survey. Students were recruited from lectures across a variety of departments including Education, Nursing, Computer Science and Psychology via in-class announcements and emails. IRB approval was granted by each of the 4 institutions involved.

Primary Testing set

Data for the testing set were derived from a sample of college students who were part of an ongoing longitudinal study involving yearly interviews. Participants were recruited using registrar lists to identify a random sample of students from two large universities, one in the Midwest and one in the West. Students completed a yearly assessment of a variety of health issues¹⁵; the data collection for the present study was included in the 2014 assessment. IRB approval was granted by the 2 universities involved.

Supplementary testing set

The goal of the supplementary testing set was to confirm selection of the 3 items and include a younger adolescent population in our scale validation. Participants were recruited from a school-based sample in the Netherlands, a country with similar internet use rates for youth compared to the US.^{16, 17} Students were recruited from schools using flyers distributed in classrooms. IRB approval was granted by Utrecht University.

Measures

The Problematic and Risky Internet Use Screening Scale (PRIUSS) was developed in 2011 and is grounded in a published conceptual framework for PIU among AYAs.¹⁸ The conceptual framework identifies seven core constructs of PIU: three describing the nature of PIU (‘risky internet use,’ ‘impulsive internet use,’ and ‘internet use dependency’); three describing PIU's impact on adolescent health and well-being (‘physical impairment,’ ‘emotional impairment,’ and ‘social/functional impairment’), and a final construct describing factors predisposing adolescents to PIU (‘psychosocial risk factors’). Each of these 7 constructs was described by a comprehensive list of associated characteristics, behaviors, and symptoms, which in turn formed the basis for development of a screening scale.

Descriptive terms from the conceptual framework were adapted to scale questions and then reduced in number using standard psychometric methods for scale development. The end product, the Problematic and Risky Internet Use Screening Scale (PRIUSS), referred to as the PRIUSS-18 in this study, is an 18-item risk-based screening scale for PIU with questions organized into three subscales: social impairment, emotional impairment, and risky/impulsive Internet use. This scale has been validated in English and in Dutch.¹⁹

The PRIUSS-18 response options utilize a Likert scale with scores of 0 through 4. Answers include: Never=0, Rarely=1, Sometimes=2, Often=3, Very often=4. We maintained these response options for the PRIUSS-3. In the PRIUSS-18, a cutoff of 25 is used to identify those at risk of PIU.¹¹ Previous studies using this cutoff for the PRIUSS suggest that approximately 7-11% of adolescents may be at risk for PIU.^{11, 20, 21}

Analysis

Our goal was to identify three items from the PRIUSS-18, one from each subscale within the PRIUSS-18, that most accurately distinguished between low and high risk for Problematic Internet Use. Consistent with the goals of a screening scale, our objective was to maximize sensitivity so that individuals with a positive screen would be evaluated with the PRIUSS-18. To develop this screen we followed an approach similar to Fisher et al including using a training set to evaluate initial 3-item combinations both conceptually and analytically, and a testing set to confirm validity of our proposed 3-item scale.²² We adapted this method by adding an additional testing set to include younger adolescents and the Dutch version of the PRIUSS.

Development of preliminary 3-item PRIUSS scales using training set

Our first goal was to develop 3 potential 3-item scale options from the larger PRIUSS-18 to evaluate using the training dataset. To identify potential items we first used Spearman correlation coefficients to evaluate the correlation between participants' response on each PRIUSS-18 item and their overall PRIUSS-18 score. We used Spearman because the PRIUSS-18 scores appeared to have a negative exponential distribution and the Spearman approach does not rely on having a particular data distribution.

Given that the PRIUSS-18 has three subscales we selected one item from each of these three subscales that had the highest Spearman correlation coefficient. These three items represented the PRIUSS-3a. Two investigators then evaluated whether there was any conceptual overlap in the 3 items. We also evaluated items within each subscale that had similar Spearman's correlation coefficients. Using these parameters we developed two alternate scales, PRIUSS-3b and PRIUSS-3c.

Initial assessments of preliminary 3-item PRIUSS scales using training set

After developing 3 potential 3-item scales we assessed each of these potential scales and their factor loading based on a previous validation study.¹¹ We confirmed that the selected items were above average on factor loading among items in each subscale. We assessed the composite score of the 3 items on the PRIUSS-3a, PRIUSS-3b and PRIUSS-3c to ensure that the composite score was highly (>0.7 Spearman coefficient) correlated with the PRIUSS-18 score. We then examined box plots to determine the positive predictive value and negative predictive values of the PRIUSS-3a, PRIUSS-3b and PRIUSS-3c.

Selection of final 3-item PRIUSS scale based on training set findings

As a final step, we plotted Receiver Operating Curves (ROC) for each of the 3 candidate scale versions. We evaluated several threshold values, representing summarized scores on the 3 items that would trigger a positive screen and indicate further testing was needed. We evaluated the specificity and sensitivity for all possible thresholds of PIU risk for each of the three scale candidates. Based on these findings we then selected the threshold with both clinical and analytic significance and the 3-item scale to use in the testing set.

Assessment of 3-item PRIUSS scale in testing set

Using the selected 3-item scale and threshold, we evaluated the sensitivity and specificity of the PRIUSS-3 screening tool on a testing sample to identify PIU (as measured by the PRIUSS-18). We then conducted a supplementary test of the PRIUSS-3 on a second testing sample to confirm performance in this separate yonger population.

Missing data

We removed participants with more than 2 missing items on the PRIUSS-18. For participants who had missing responses for one or two PRIUSS-18 questions, these responses were imputed as the average across remaining items in the respective subscale from which it was missing, a recommended approach for scale data.²³

Results

Participants

We had a total of 1079 participants in this study; participant demographics by dataset are detailed below and in Table 1.

Table 1. Demographics of samples.

	Training set n=332 participants	Testing set 1 n=283 participants	Supplementary testing set n=464 participants
Age Mean(SD)	19.2 years (1.6)	21.4 years (0.7)	14.3 years (1.8)
Gender N(%)	Female: 289 (87.1%)Male: 43 (12.9%)	Female: 161 (57.8%)Male: 122 (43.1%)	Female: 219 (47.2%)Male: 244 (52.6%)No response: 1
Race/Ethnicity N(%)	Caucasian: 272 (82%) Asian: 25 (7.6%) Hispanic: 11 (3.2%) African American/Black: 15 (4.5%) More than One: 9 (2.7%)	Caucasian: 211 (74.6%) Asian: 40 (14%) Hispanic: 6 (2.1%) African American/Black: 4 (1.4%) More than one: 20 (7.1%) Other: 2 (0.7%)	Born in Netherlands: 453 (97.6%) Born outside Netherlands: 8 (1.7%) No Response: 3 (0.6%)
PRIUSS score	Mean(SD): 15.7 (10.8) Median: 13 Range: 0-60	Mean(SD): 12.1 (9.7) Median: 10 Range: 0-49	Mean (SD): 13.5 (9.3) Median: 12 Range: 0-55
Prevalence of PIU measured by PRIUSS	59 (17.7%)	26 (9.1%)	54 (11.6%)

PIU: Problematic Internet Use, PRIUSS: Problematic and Risky Internet Use Screening Scale

Training set

A total of 334 participants were recruited to the training set, the response rate was 53.3% and 2 participants (0.6%) were excluded due to missing data. Thus, the training set was made up of 332 college students who had average age of 19.2 (SD=1.6) years, were 87.1% female and 82% Caucasian.

Testing set

The testing set included 283 college students with response rate 98.5%. These participants were 57.8% female, 74.6% Caucasian and had an average age of 21.4 (SD=0.7) years. No participants were excluded for missing data.

Supplementary testing set

The supplementary testing set was comprised of 474 adolescents with response rate of 98%, a total of 10 participants (2.1%) were excluded due to missing data. Thus, the supplementary testing set sample included 464 adolescents who were 47.2% female, 97.6% born in Netherlands, and had an average age of 14.3 (SD=1.8) years.

Development of preliminary 3-item PRIUSS scales using training set

To select items from the PRIUSS-18 we selected one item from each of the PRIUSS-18 subscales. For Subscale 1, the selected item was social anxiety: “how often do you experience increased social anxiety due to your internet use” with correlation of rho =0.61, p<0.001 to the PRIUSS-18 score. For Subscale 2 the selected item referenced anxiety: “how often do you feel anxious because you are away from the internet,” with correlation rho=0.66, p- < 0.001 to the PRIUSS-18 score. For Subscale 3 the selected item was losing motivation: “how often do you lose motivation to do other things that need to get done because of the internet” with correlation rho=0.71, p<0.001. These 3 items comprised the PRIUSS-3a.

We noted in the PRIUSS-3a that there were two items that referenced anxiety. We had concerns that in a three-item scale, participants might respond similarly to the two anxiety-related questions based on the similar concepts, even though these two questions are rooted in different subscales within the PRIUSS-18. We also noted that in the PRIUSS-3a, while the first and third items had distinctly higher correlations to the overall PRIUSS-18 score compared to other items in their subscale, the second item, “feel anxious when away from the Internet” had a correlation nearly identical to other items in its subscale. Thus, we developed two alternate scales, hereafter called PRIUSS-3b and PRIUSS-3c, by selecting an alternate second item by identifying an item with the next highest Spearman correlation coefficient. The PRIUSS-3b includes the second item “how often do you feel irritated when away from the internet” and PRIUSS-3c includes the second item “how often do you feel withdrawal when away from the internet.” These three potential scales and correlation results are included in Table 2.

Table 2. Correlation between PRIUSS-18 items, proposed PRIUSS-3 scales and PRIUSS-18 Sum.

PRIUSS- 3 version	Subscale	Item	Spearman Correlation with PRIUSS-18 Sum	Factor Loading (SE)a
3a/b/c	1	Social anxiety: experience increased social anxiety due to your internet use	S=2377082 rho=0.61***	0.79 (0.04)
3b	2	Irritated: feel irritated when away from the internet	S = 2080401 rho = 0.66***	0.86 (0.04)
3a	2	Anxious: feel anxious because you are away from the internet	S = 2050761 rho = 0.66***	0.88 (0.04)
3c	2	Withdrawal: Feel withdrawal when away from the internet	S = 208017 rho = 0.66***	0.82 (0.04)
3a/b/c	3	Lose motivation: lose motivation to do other things that need to get done because of the internet	S = 1743216 rho=0.71***	0.76 (0.04)

^aFactor loadings from paper Jelenchick et al 2014¹¹

^***All p-values <0.001

Initial assessments of preliminary 3-item PRIUSS scales using training set

We examined box plots of the PRIUSS-3a, 3b and 3c, included in Figure 1 (online). We confirmed that chosen items were above average for factor loadings among items in each respective subscale. We also determined that the composite score of the PRIUSS-3a, 3b, and 3c was correlated at 0.7 or greater with the PRIUSS-18 score.

Figure 1.

After examining the distribution of sensitivity and specificity at various thresholds, we applied the criteria that a total score 3 or greater on the PRIUSS-3 would be used as a cutoff to prompt screening with the PRIUSS-18. Thus, examining the positive predictive and negative predictive value of PRIUSS-3a, 3b, and 3c, we determined that all PRIUSS-3 candidate scales would capture nearly all PIU cases. The PRIUSS-3a had a specificity of 62%, PRIUSS-3b specificity was 50%, and PRIUSS 3c specificity was 59%. These findings are included in Table 3.

Table 3. Performance of three candidate PRIUSS-3 scales on Training Set.

	PRIUSS 3a		PRIUSS 3b		PRIUSS 3c
Area under ROC Curve	.97 (95% CI: .95-.98)		.95 (95% CI: .93-.98)		.96 (95% CI: .93-.98)
Threshold*	Specificity	Sensitivity	Specificity	Sensitivity	Specificity	Sensitivity
3	.62	1.0	.50	1.0	.59	1.0
4	.81	.98	.70	.98	.80	.98
5	.95	.90	.86	.93	.90	.86
6	.97	.51	.93	.81	.97	.66

^*Only the thresholds with highest combinations of sensitivity and specificity are shown. There were 13 possible thresholds for each scale.

Selection of final 3-item PRIUSS scale based on training set findings

In consideration of these tests we selected the PRIUSS-3c for evaluation with the testing sets. Key factors in this decision included wanting to avoid having 2 of the 3 items focused on anxiety, and a prioritization for a scale with high sensitivity. Given that the PRIUSS-3 was intended as a screening tool to prompt further evaluation if positive, we determined that 100% sensitivity was worth the tradeoff of a lower (59%) specificity. The ROC curve for the PRIUSS-3c is included as Figure 2.

Figure 2.

Assessment of 3-item PRIUSS scale in testing sets

In testing set 1 the Spearman correlation between PRIUSS-3c sum and PRIUSS-18 sum was rho=.86, p<0.001. The area under curve for the PRIUSS-3c on testing set 1 was: .96 (95% CI: .93-.99). The specificity was 69% (95% CI: 63%-74%) and the sensitivity was 100% (95% CI: 100%-100%).

In the supplementary testing set, the Spearman correlation between PRIUSS-3c sum and PRIUSS-18 sum was rho=.73, p<0.001. The area under curve for the PRIUSS-3c on the supplementary testing set was: .91 (95% CI: .86-.96). The specificity was 75% (95% CI: 70%-79%) and the sensitivity was 90% (95% CI: 80%-96%).

Discussion

To develop a validated screening instrument to identify AYA patients at risk for PIU, we selected items representing the three constructs included in the PRIUSS-18. We conducted rigorous testing of these items to develop a shorter screening tool. Our finalized screening scale, the PRIUSS-3 was found to have high correlation with the sum scores for the PRIUSS-18 and excellent sensitivity to identify PIU. Thus, the PRIUSS-3 can be used as a screening tool; patients with a total score of 3 or greater on the PRIUSS-3 can then be administered the PRIUSS-18 as a more thorough evaluation.

This screening tool may be useful in clinical practice for providers who see AYAs. This 3-item screening tool could fit within existing adolescent health screens such as those based on the Guidelines for Adolescent Preventive Services.²⁴ Adolescents support health risk assessments as a valuable tool to report confidential information and receive face-to-face feedback from providers.²⁵ However, studies suggest that less than a quarter of adolescents are offered these screens.^26-28 It is possible that including issues that are timely and important to parents and teens may enhance the value of these screens. Studies suggest that time spent using the internet is a highly salient concern for parents; a recent report found that approximately a third of parents reported concerns about their child's internet use in the past year.²⁹ As these behavioral health screens cover a wide variety of health risks and are typically administered at the start of a clinical visit, positive screening results can be followed by a more definitive diagnostic test in the same visit. The PRIUSS-3 had a specificity of 69% in our testing set; thus, almost 70% patients without PIU will be effectively screened out, saving time for both patients and providers. Those who screen in with the PRIUSS-3 can then complete the PRIUSS-18. This process would be very similar to those outlined in screening for depression, where a positive PHQ-2³⁰ is followed by a PHQ-9.¹³ The PRIUSS-3 may also be well positioned to be placed within other larger national surveys, such as the Youth Risk Behavior Survey (YRBS)³¹ to evaluate a national sample of youth for risk of PIU. In this way, epidemiologic data approximating PIU prevalence could inform future prevention and intervention efforts.

Limitations of this study include that we did not evaluate all possible combinations of items and thresholds to create a PRIUSS-3 from the PRIUSS-18, instead, we methodically and purposefully selected a set of three promising scales and then evaluated all thresholds for these three scales. This follows previous scale development designs such as those outlined by Fisher et al.²² We noted that the sensitivity of the PRIUSS-3 was 100% in the first testing set and 90% in the second testing set which consisted of younger teens who were Dutch. Thus, sensitivity may be lower for this screening test in younger teens, or it may be related to testing the PRIUSS-3 in a different country. Another limitation is that our initial training set was disproportionately female, and racial diversity in our sample was limited. As gender and race have been shown to affect internet use rates³², further study in this area is needed. For our study sample recruited from the Netherlands, we limited to following typical procedures of recording demographics as whether or not the teen was born in the country and were unable to collect information about race/ethnicity. Our testing and training data covered many ages, genders, and geographic locations, and encompassed a total of 1079 young adults recruited across three separate data collections. Thus, our three samples represented diversity in geography and the demographics of AYAs living in those areas. It is possible that including participants who were enrolled in a longitudinal study in our testing set could have introduced selection bias. Despite these limitations, our study presents a validated short screen for PIU. Future work can use this validated screening instrument in clinical settings and future research studies, including studies to evaluate feasibility or implementation of the PRIUSS-3 in clinics.

List of Abbreviations (in order of use in the paper)

PIU

Problematic Internet Use

PRIUSS

Problematic and Risky Internet Use Screening Scale

AYA

Adolescents and Young Adults

PHQ

Patient Health Questionnaire

ROC

Receiver Operating Curves

YRBS

Youth Risk Behavior Survey