Abstract
Background and Aims
The past-year prevalence of problematic pornography use (PPU) was 1–6% in adult populations. As a result of treatment obstacles and barriers, such as unaffordable treatments, only a minority of problematic pornography users may seek treatment. Having a free, online, self-help program may overcome treatment barriers and may help those individuals who cannot receive traditional or offline treatment for PPU. Although the effectiveness of such online programs reducing substance use and problematic gambling have been reported, no prior study has examined the efficacy of an online self-help intervention aiming to reduce PPU.
Methods
This two-armed randomized controlled trial (RCT) will examine the effectiveness of an online self-help program (Hands-off) to reduce PPU, while also considering psychopathological comorbidities. The six-week intervention condition includes six core modules developed to reduce PPU based on motivational interviewing, cognitive behavioral therapy, mindfulness, and wise social-psychological intervention techniques. The target sample size is 242 participants. Self-report questionnaires will be administered at baseline, right after the end of the intervention, at one-month, and three-month follow-ups after the end of the intervention. The primary outcome will be the level of PPU. Secondary outcomes will include pornography use frequency, pornography craving, pornography use-avoidance self-efficacy, sex mindset, sexual satisfaction, negative and positive emotions, and life satisfaction. Data will be analyzed on an intention-to-treat basis using linear mixed models.
Results
Results will be reported at conferences and published in a scientific peer-reviewed journal. The participants will be sent a lay-person-friendly summary of the results via e-mail.
Keywords: cognitive-behavioral therapy, motivational interview, online intervention, problematic pornography use, wise social-psychological intervention
Introduction
In the past decades, pornography use has started to increase among young adults—even among pre-adolescent children—presumably due to the widespread access to the Internet (Lewczuk, Wojcik, & Gola, 2019; Price, Patterson, Regnerus, & Walley, 2016). Based on findings of recent nationally-representative studies from Australia (Rissel et al., 2017) and the US (Grubbs, Kraus, & Perry, 2019), 69–76% of men and 33–41% of women used pornography in the past year. However, only 4–6% of men and 1–2% of women reported having felt addicted to pornography (Grubbs et al., 2019; Rissel et al., 2017). Although pornography use may have little or no adverse effect on most people's lives, it may become problematic for some and may result in severe adverse consequences (e.g., job loss, problems in romantic relationships (Bőthe, Tóth-Király, et al., 2018b; Bőthe, Tóth-Király, Demetrovics, & Orosz, 2017; Bőthe, Tóth-Király, Potenza, Orosz, & Demetrovics, 2020; Ford, Durtschi, & Franklin, 2012; Perry, 2019)) leading to treatment-seeking behavior among men and women as well (Bőthe, Tóth‐Király, Demetrovics, & Orosz, 2020; Gola, Lewczuk, & Skorko, 2016; Lewczuk, Szmyd, Skorko, & Gola, 2017).
Despite the inclusion of Compulsive Sexual Behavior Disorder (CSBD) in the 11th version of International Statistical Classification of Diseases and Related Health Problems (ICD-11) (World Health Organization, 2019), there is no official diagnosis for problematic pornography use (PPU) (Fernandez & Griffiths, 2019). However, as PPU is often considered as a manifestation of CSBD, the same diagnostic guidelines may be applied (Fernandez & Griffiths, 2019). For problematic users, pornography is the central focus of their life; they experience failures when trying to control their use with recurring unsuccessful efforts to regulate or reduce it, and they engage in pornography use despite the adverse consequences (e.g., job loss) (Bőthe, Tóth-Király et al., 2018b; World Health Organization, 2019). Besides the aforementioned potential adverse consequences, high rates of other psychological and psychiatric problems were reported concerning PPU. Mood disorders (31–72%), anxiety disorders (33–47%), substance use disorders (14–41%), and attention deficit hyperactivity disorder (ADHD) (3–67%) are reported to be comorbid and prevalent among problematic pornography users or individuals with CSBD (Blankenship & Laaser, 2004; Bőthe, Koós, Tóth-Király, Orosz, & Demetrovics, 2019; Grubbs, Volk, Exline, & Pargament, 2015; Kafka & Hennen, 2002; Kraus, Potenza, Martino, & Grant, 2015; Nelson, Padilla-Walker, & Carroll, 2010; Raymond, Coleman, & Miner, 2003; Reid, 2007; Reid, Carpenter, Gilliland, & Karim, 2011; Reid, Davtian, Lenartowicz, Torrevillas, & Fong, 2013; Scanavino et al., 2013; Wéry et al., 2016; Willoughby, Busby, & Young-Petersen, 2019; Willoughby, Carroll, Nelson, & Padilla-Walker, 2014). Nevertheless, evidence-based treatments offered for problematic pornography have remained scarce worldwide.
Based on the findings of recent literature reviews discussing the treatment approaches for PPU, problematic online sexual activities, and CSBD (Dhuffar & Griffiths, 2015; Garcia et al., 2016; Hook, Reid, Penberthy, Davis, & Jennings, 2014; Kaplan & Krueger, 2010; Naficy, Samenow, & Fong, 2013; Sniewski, Farvid, & Carter, 2018; von Franqué, Klein, & Briken, 2015; Wéry & Billieux, 2017), cognitive behavior therapy (CBT), acceptance and commitment therapy (ACT), motivational interviewing techniques, and mindfulness-based approaches may be efficient in the treatment of CSBD and PPU. However, the evidence is mostly based on case reports and uncontrolled studies (Dhuffar & Griffiths, 2015; Garcia et al., 2016; Hook et al., 2014; Kaplan & Krueger, 2010; Naficy et al., 2013; Sniewski et al., 2018; von Franqué et al., 2015; Wéry & Billieux, 2017). To the best of the authors' knowledge, only three studies examined the effectiveness of short (i.e., 12 sessions), ACT- and CBT-based interventions aiming to reduce specifically PPU (i.e., not hypersexuality, sexual addiction, or compulsive sexual behaviors), including a control group (Crosby, 2011; Crosby & Twohig, 2016; Minarcik, 2016). Each of these studies reported a significant improvement in the participants' pornography use-related symptoms and problems compared to the control group not only during the post-treatment assessments but in the follow-up assessments as well (i.e., 12–20 weeks after the end of the interventions) (Crosby, 2011; Crosby & Twohig, 2016; Minarcik, 2016). Although these studies demonstrated the efficiency of ACT- and CBT-based methods in the reduction of PPU, they only reached 12 to 28 participants presumably due to the offline settings of the interventions (i.e., individual sessions with a therapist).
Even though effective interventions exist for PPU (Crosby, 2011; Crosby & Twohig, 2016; Minarcik, 2016), there is still a significant unmet need for scalable treatment. Web-based (online) interventions may contribute to the reduction of traditional treatment barriers and may facilitate treatment utilization. Treatment barriers and obstacles include social and individual barriers as well, such as high mental health stigma, feelings of shame to talk about pornography use, or unaffordable treatment costs (Dhuffar & Griffiths, 2016). Online self-help tools may reduce both social and individual treatment barriers by being free and easy-to-use tools, providing anonymity, privacy, flexibility, short or no waiting time, and supporting participants' autonomy and self-efficacy (Baumgartner et al., 2019; Haug, Castro, Wenger, & Schaub, 2018; Herrero et al., 2019; Weisel et al., 2018). Despite the advantages mentioned above and the cost-effectiveness of online interventions,—compared to face-to-face interventions—online interventions should not be considered as replacements for face-to-face treatments but as additional tools that can provide treatment for those individuals who are not yet in treatment or hesitant to seek traditional forms of treatment. Also, it has to be noted that online interventions have some disadvantages as well, such as low treatment commitment and adherence, limited individualization options, limited treatment tailoring, and, thus, higher dropout rates (Haug et al., 2018; Schaub et al., 2016).
Currently, there is a gap in the literature with respect to evidence-based, scalable, online interventions that can effectively reduce PPU. Therefore, the aim of the present study is to test the effectiveness of a web-based self-help program (Hands-off) to reduce PPU in a two-armed randomized controlled trial (RCT), while also considering psychopathological comorbidities. The primary outcome will be the change in the level of PPU between baseline and follow-up assessments (after the end of the six-week-long intervention (t1), at a one-month follow-up (t2), and three-month follow-up (t3)) compared to a control condition. The secondary outcome measures will include beneficial changes in the intervention groups' pornography use frequency, pornography craving, pornography use-avoidance self-efficacy, beliefs about the malleability of sexual life, sexual satisfaction, negative and positive emotions, and life satisfaction over time compared to the control group.
Methods
Study design
The web-based (online) self-help program, Hands-off, will be evaluated with a two-armed RCT comparing the efficacy of the (1) intervention condition with the (2) wait-list control condition. Participants assigned to the control condition will access the intervention materials three months after completing the baseline questionnaire set. Participants will be unaware of the hypotheses of the study but will be aware of the assigned condition. Any blinding of study personnel is unwarranted, as they will not be directly involved in the intervention or the assessment. After completing the baseline self-report questionnaire (t0), participants will be randomly assigned to one of the two study arms. Further assessments will take place right after the end of the six-week-long intervention (t1), at a one-month follow-up (t2), and three-month follow-up (t3) after the end of the intervention (Fig. 1). The study was preregistered on the Open Science Framework (OSF) Website (https://osf.io/5tqkb/).
Sample size calculation
Two groups (treatment vs. control group) will be compared over time (t0, t1, t2, and t3). Based on the results of previous PPU reduction interventions (Crosby, 2011; Crosby & Twohig, 2016; Minarcik, 2016), small effect sizes (Cohen d = 0.2) may be anticipated when comparing the treatment and control conditions. After calculations conducted with the G*Power software, a sample size of 121 in each study group would have 80% power to detect the difference with an alpha error of 5% and two-tailed testing. Therefore, the recruitment of a minimum of 242 participants would be sufficient to detect differences between the treatment and control groups over time.
Recruitment of study participants
The recruitment takes place since February 2019 and is currently ongoing until the target number of minimum participants is reached (N = 242). Participants will be recruited on pornography sites, psychology news websites, newsletters, and social media sites. The study will mainly recruit people from Switzerland and Hungary; however, participants from other countries will not be excluded. Participants will not be compensated for participation.
Registration and consent procedure
Participants will register online by providing minimal personal data, including their e-mail address and necessary sociodemographic information (e.g., age, gender). Informed consent will be obtained before enrollment in the study. Participants will be assigned to be randomized if they are eligible according to the inclusion and exclusion criteria (see Table 1).
Table 1.
Inclusion criteria | Rationale |
1. Minimum age of 18 | To ensure the minimum age of participation |
2. Proficiency in English (at least intermediate level) | To ensure that participants will be able to understand the information provided |
3. Internet access at least once every week and a valid email address | To ensure access to the intervention |
4. Completion of the informed consent | To ensure knowledge of procedures and the declaration of consent |
Exclusion criteria | Rationale |
1. Self-reported engagement in other treatment for problematic pornography use | To avoid confounding treatment effects |
Randomization and trial flow
After the baseline assessment, randomization will take place at an individual level by an automated, computer-based algorithm on the intervention website. A randomization list will be created in a 1:1 ratio. After the randomization, participants in the intervention condition will receive immediate access to the online intervention, while participants in the control condition will receive access to the intervention three months after the completion of the baseline questionnaire. Participants will know to which group they have been assigned. The intervention condition will last six weeks. Follow-up assessments will be completed online and will take place right after the end of the six-week-long intervention (t1), at a one-month follow-up (t2), and three-month follow-up (t3) (Fig. 1). If the final assessment is not completed, up to two reminders will be sent out 2 and 5 days later.
Hypotheses
Concerning the primary outcome, we hypothesize that participating in the intervention condition (study arm 1)—compared to the control condition (study arm 2)—will result in lower levels of PPU comparing the baseline and the follow-up assessments. Regarding the secondary outcomes, we hypothesize that participating in the intervention condition—compared to the control condition—will result in lower levels of pornography use frequency, lower levels of pornography craving, lower levels of negative emotions in general, higher levels of sexual satisfaction, higher levels of pornography avoidance self-efficacy, higher levels of beliefs about the malleability of sexual life, higher levels of life satisfaction, and higher levels of positive emotions in general over time.
Intervention
Hands-off (www.hands-off.net) is an automated online self-help program developed by the authors of this paper to reduce PPU. The program includes a dashboard, a pornography use diary, six treatment modules, and a booster module that were developed to reduce PPU. The program was developed based on the principles of motivational interviewing (Rollnick & Miller, 1995), cognitive-behavioral therapy (Meichenbaum, 1977), mindfulness techniques (Altman, 2014), and “wise” social-psychological interventions (Walton, 2014; Walton & Wilson, 2018; Yeager & Walton, 2011). The modules were created based on previous online interventions developed by the Swiss Research Institute for Public Health and Addiction that effectively reduced substance use, alcohol use, and problematic gambling (Baumgartner et al., 2019; Schaub et al., 2016, 2013). The core modules (1–6) must be completed in their intended sequence (i.e., finishing each module unlocks access to the next one). The booster module can be completed one-month after finishing the intervention condition. It was designed to reflect on the difficulties after finishing the program and to motivate the participants further to implement the acquired knowledge in their everyday life. Participants are encouraged to repeat any modules if they feel they need to.
Intervention condition - Study arm 1
Dashboard
The main page is the dashboard. It was created to provide useful information quickly and to display the dates of the follow-up assessments. The participants can also enter their pornography use information directly from the past week, and they can plan their target frequency of pornography use for the next week as well. An activity planner is also included on the dashboard that reminds the participants to plan activities for the current week and rate their level of anticipated and actual enjoyment of the given activity (Fig. 2).
Self-help intervention modules
Six self-help intervention modules and one booster module are included in the intervention condition presented in the Modules section of the website (Fig. 3). A summary of each module's content is presented in Table 2. Participants are encouraged to complete one module each week. As mentioned above, modules 1–6 can only be completed in their intended sequence, and the booster module can only be completed one-month after finishing the intervention condition. Participants are encouraged to repeat any modules if they would find it helpful. Red bars indicate any progress the participants have already made within each module; green bars indicate if modules are completed.
Table 2.
Modules | Content |
Module 1: Introduction and the possibility of change |
• General overview • Introduction to fictional companions • Reflections on personal pornography use (e.g., advantages and disadvantages, reasons for change, reviewing useful resources for a change) |
Module 2: Why do I watch porn, and how can I change it? Identifying internal and external risk situations |
• Identification of the internal and external risk situations that can lead to pornography use • Learning how to deal with these risk situations |
Module 3: How to feel better without porn? |
• Learning how to change personal pornography using habits • Learning how to integrate joyful activities into everyday life |
Module 4: What can I do about my cravings? |
• Identification of personal triggers for cravings • Learning strategies to reduce craving |
Module 5: Behaviors, Emotions, and Thoughts (BET): I BET you can do it |
• Getting to know automatic negative thoughts and the most frequent common thinking errors • Learning about the relations between one's thoughts, emotions, and pornography use • Learning strategies to challenge automatic negative thoughts and develop balanced thoughts |
Module 6: How to preserve your success? |
• Reviewing the main contents of the previous modules • Identification of one's toughest moments in the program and how he/she overcame them • Planning strategies to prevent relapses to previous pornography use habits |
Booster module: Success in the long run |
• Reviewing one's past month and the strategies he/she used to reduce his/her pornography use and to improve his/her mood • Making plans for the future to preserve success in the long run |
Pornography use diary
After the completion of the first module, there will be daily assessments of pornography use (frequency of use per day) and mood (Positive and Negative Affect Scale – PANAS (Gyollai, Simor, Köteles, & Demetrovics, 2011)) (Fig. 4). Participants will be asked to record their targeted (i.e., how much pornography they want to use over the upcoming week) and their actual pornography use (i.e., how much pornography they used during the past week). Additionally, they can register their mood each day. A personal graph will be generated with their inputs for visual feedback. Setting daily pornography use goals may strengthen the participants' self-efficacy, while mood tracking may contribute to the identification and understanding of the relationships between their mood and pornography use.
Fictional companions
Four fictional companions were created representing typical problematic pornography users to encourage reflection on specific questions in each module (Fig. 5). These companions share their thoughts, fears, and achievements in written form. Participants can choose one character that they best identify with based on their situation but may also view the input provided by the other companions by clicking on their icons.
Safety
During the participation in the program, participants are provided with an emergency button (“Help Me” tab) for immediate responses to frequently asked questions and access to emergency contacts (Fig. 6). Participants will be informed about how to use this information.
Other elements
In some exercises (e.g., pros and cons of pornography use), participants will be asked to enter their answers by clicking on checkboxes or entering text freely. This information will be accessible for them anytime in their Profile (“My inputs” section). Entering information during the program would not only result in higher levels of engagement but may also be considered as an effective self-persuasion technique (Aronson, 1999; Sniewski & Farvid, 2019). Moreover, adherence to the treatment will be monitored, and reminders will be sent via e-mails after seven (14, 21, 28) days of non-compliance (i.e., not logging in to the intervention website or not completing the daily diary of pornography use and mood).
Control condition – Study arm 2
Participants randomized to the wait-list control group will be provided the opportunity to participate in the intervention condition (study arm 1) three months after registration. Participants will also complete the baseline (t0) and the follow-up (t1, t2, and t3) assessments.
Measures
Sociodemographic data will include gender, age, level of education, socio-economic status, religious views, place of residence, relationship status, and sexual orientation. Sexuality-related questions will include the number of lifetime sexual partners and the number of lifetime casual sexual partners (Bőthe, Bartók et al., 2018a). Table 3 provides an overview of and a schedule for the measures that will be administered.
Table 3.
Assessment instruments | Baseline (t0) | After finishing the program (t1) | One-month follow-up (t2) | Three-month follow-up (t3) |
Sociodemographic questions | X | |||
Pornography use-related measures | X | X | X | X |
Pornography use-related questions (frequency of use, duration of use per occasion) | X | X | X | X |
Previous treatment for problematic pornography use | X | |||
Problematic pornography use (PPCS) | X | X | X | X |
Self-reported perceived addiction to pornography use | X | X | X | X |
Moral incongruence regarding pornography use | X | X | X | X |
Pornography craving (PCQ) | X | X | X | X |
Pornography avoidance self-efficacy (PASS) | X | X | X | X |
Sexuality-related measures | X | X | X | X |
Sexuality-related questions | X | X | X | X |
Sexual satisfaction | X | X | X | X |
Beliefs about the malleability of sexual life (SMS) | X | X | X | X |
Life quality measures | X | X | X | X |
Life satisfaction (SWLS) | X | X | X | X |
General positive and negative emotions (PANAS) | X | X | ||
Psychopathological symptoms measures | X | X | X | X |
Depression, anxiety, and somatization (BSI-18) | X | X | X | X |
ADHD (ASRS-5) | X | X | X | X |
Substance use (NIDA) | X | X | X | X |
Alcohol use (SIP) | X | X | X | X |
Suicide risk (P4) | X | X | X | X |
Side effects (wanted and unwanted effects) | X |
Primary outcome
The primary outcome of interest will be the Problematic Pornography Consumption Scale (PPCS) (Bőthe, Tóth-Király et al., 2018b). The PPCS assesses past-six-month PPU covering six factors: salience, tolerance, mood modification, withdrawal, relapse, and conflict (Griffiths, 2005). The scale includes 18 items with three items per each factor. Respondents indicate answers on seven-point scales (1 = “never”; 7 = “all the time”). Higher scores on the scale indicate higher levels of PPU. A score of 76 (out of 126) or higher indicates a high-risk of PPU.
Secondary outcomes
Besides the PPCS, the frequency of pornography use (one question) (Bőthe, Bartók et al., 2018a; Bőthe, Tóth-Király et al., 2019b), duration of pornography use per each occasion (one question) (Bőthe, Bartók, et al., 2018a; Bőthe, Tóth-Király et al., 2020), previous treatment-seeking (two questions) (Bőthe, Tóth-Király et al., 2020), perceived addiction to pornography use (one question) (Grubbs et al., 2019), and moral incongruence regarding pornography use (one question) (Grubbs et al., 2019) will be assessed with pre-established questions.
Pornography craving will be measured with the one-factor Pornography Craving Questionnaire (PCQ) (Kraus & Rosenberg, 2014) including 12 items rated on a seven-point scale (1 = “disagree completely”; 7 = “agree completely”). Items assess current pornography craving. Higher scores on the scale indicate higher levels of craving for pornography use.
Self-efficacy regarding pornography use will be measured with the one-factor Pornography-Use Avoidance Self-Efficacy Scale (PASS) (Kraus, Rosenberg, Martino, Nich, & Potenza, 2017), including 18 items. Participants indicate their confidence in relation to avoiding pornography use (0% = “not confident at all”; 100% = “completely confident”). Higher scores on the scale indicate higher levels of craving for pornography use.
Participants' current level of sexual satisfaction (one item) (Mark, Herbenick, Fortenberry, Sanders, & Reece, 2014) will be assessed with one reliable and valid pre-established question.
Beliefs about the changeability of sexual life will be assessed with the one-factor Sex Mindset Scale (SMS) (Bőthe et al., 2017), including five items (three reverse coded items) rated on a six-point scale (1 = “strongly disagree”; 6 = “strongly agree”). Higher scores on the scale indicate higher levels of beliefs in the malleability of one's sexual life.
Participants' satisfaction with their life will be assessed with the one-factor Satisfaction with Life Scale (SWLS) (Diener, Emmons, Larsen, & Griffin, 1985), including five items rated on a seven-point scale (1 = “strongly disagree”; 7 = “strongly agree”). Higher scores on the scale indicate higher levels of satisfaction with one's life.
Participants' current levels of general positive and negative feelings will be assessed with the ten-item Positive and Negative Affect Scale (PANAS) (Gyollai et al., 2011). The scale includes two factors (positive affect and negative affect) with five items per each factor rated on a five-point scale (1 = “very slightly or not at all”; 5 = “extremely”). Higher scores on the factors indicate higher levels of positive and negative affect, respectively.
Control variables
The levels of psychiatric symptoms in the past seven days will be assessed with the short version of the Brief Symptom Inventory (BSI-18) (Asner-Self, Schreiber, & Marotta, 2006). The scale includes three factors (depressive symptoms, anxiety symptoms, and somatization symptoms) with six items per each factor rated on a five-point scale (1 = “not at all”; 5 = “extremely”). Higher scores on the factors indicate higher levels of depressive, anxiety, and somatization symptoms, respectively.
ADHD symptoms in the past six months will be assessed with the one-factor Adult ADHD Self-report Screening Scale for DSM-5 (ASRS-5) (Ustun et al., 2017), including six items rated on a five-point scale (1 = “never”; 7 = “very often”). Higher scores on the scale indicate higher levels of self-report adult ADHD symptoms.
The past 90-day substance use will be assessed with the ten-item NIDA Assist (NIDA) (Group, 2002). Each item is rated on a five-point scale (1 = “never”; 5 = “daily or almost daily”). Higher scores indicate a higher frequency of substance use.
Past 90-day alcohol use will be assessed with the one-factor, 15-item Alcohol-Related Problems: Short Inventory of Problems (SIP) (Miller, Tonigan, & Longabaugh, 1995). Each item is rated on a four-point scale (1 = “never”; 5 = “daily or almost daily”). Higher scores indicate higher levels of alcohol use-related problems.
Participants' levels of suicidal thoughts and suicidal risk will be assessed with the one-factor P4 Suicidality Screener (P4) (Dube, Kroenke, Bair, Theobald, & Williams, 2010), including five items rated on a two-point scale (0 = “no”; 1 = “yes”). If an elevated risk of suicide (i.e., scoring “yes” for any of the questions) is identified at any of the five assessments, the participant will be advised to call an emergency number and/or visit any local facility presented in a prepared list.
Potential wanted and unwanted side effects of the program will be assessed with four pre-established questions (Rozental, Boettcher, Andersson, Schmidt, & Carlbring, 2015).
Data analyses
Data will be analyzed on an intention-to-treat basis. Missing data will be handled by applying multiple imputations with 100 iterative estimations per value (Schafer & Graham, 2002). Demographic variables and other baseline variables will be inserted into the prediction model to estimate missing values. Differences in the outcomes between the treatment and control conditions will be examined with linear mixed models (LMM). LMMs will be specified appropriately to model clusters and repeated measures by defining random effects for study arms and time (repeated measures). For non-normal continuous outcomes, appropriate distributions (e.g., zero-inflated) will be specified. Additionally, per-protocol analyses will be conducted.
Data security
The program was developed and programmed in PHP 7.3 and JavaScript embedded in the Content Management System Drupal 7, which uses and MySQL-Database. The intervention will be maintained and kept updated by the ISGF. All connections are encrypted and password-protected through the SSL protocol. Participants will only access their information. The final data will be exported from the database and will be stored in a password-protected file at the PI's institution. Email addresses will be deleted after the study is completed.
Ethics
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration. The present research was approved by the Institutional Ethical Review Board of the Eötvös Loránd University (2018/249-2).
Patient, dissemination, and public involvement
Former problematic pornography users and experts in PPU research, wise social-psychological interventions, and CBT evaluated the content and the presentation of the intervention. There was no public or patient involvement in the study design, hypotheses, or outcome measures. The results of the present study will be presented at scientific conferences and will be published in a topic-relevant peer-reviewed journal. The participants will be sent a lay-person-friendly summary of the results of the study via e-mail.
Discussion
Although several studies reported effective treatment approaches for PPU, most of them used less rigorous research designs (i.e., case study design or studies without control groups) (Dhuffar & Griffiths, 2015; Garcia et al., 2016; Hook et al., 2014; Kaplan & Krueger, 2010; Naficy et al., 2013; Sniewski et al., 2018; von Franqué et al., 2015; Wéry & Billieux, 2017). Three studies using ACT- and CBT-based methods were conducted in an RCT framework and demonstrated positive, long-term effects in relation to PPU in the intervention group (Crosby, 2011; Crosby & Twohig, 2016; Minarcik, 2016). Despite these positive preliminary findings, no prior study has examined the effectiveness of evidence-based, scalable, online interventions that can effectively reduce PPU. Thus, this is the first RCT to test the efficacy of an online intervention reducing PPU, assessing clinically relevant outcomes (e.g., changes in pornography use behaviors, improvements in mental health, and quality of life). The expected findings will extend our insights in designing effective online interventions in general and, more specifically, for PPU. Given that this intervention will be effective, it will be freely available for users and may be translated into several languages enabling worldwide dissemination. This online self-help program may reduce PPU-related treatment barriers by being a free and easy-to-use tool; providing anonymity, privacy, and flexibility; and supporting participants' autonomy and self-efficacy (Baumgartner et al., 2019; Haug et al., 2018; Herrero et al., 2019; Weisel et al., 2018).
However, some limitations of the present study should be mentioned. First, dropout rates are expected to be high in line with reports from previous online intervention studies (Rooke, Copeland, Norberg, Hine, & McCambridge, 2013; Schaub et al., 2015). Second, online interventions are often characterized by a low adherence rate because of the distant nature of the interventions (e.g., lack of personal relationships or non-personalized exercises) (Amann et al., 2018). Third, individuals with PPU who are currently receiving psychosocial or pharmacological treatments to reduce their PPU will be excluded from the study. Fourth, individuals who are more familiar with the use of the Internet may be more likely to participate in the study that may result in self-selection bias (Wantland, Portillo, Holzemer, Slaughter, & McGhee, 2004). Fifth, all measures will be self-reported, which may result in report bias. However, anonymous, online, self-reported data collection may be beneficial in sexuality-related studies when participants are asked to report about sensitive topics and problematic behaviors by alleviating stress resulting in more honest responses (Griffiths, 2012).
Funding sources
The research was supported by the Hungarian National Research, Development, and Innovation Office (Grant numbers: KKP126835, NKFIH-1157-8/2019-DT). BB was supported by the ÚNKP-18-3 New National Excellence Program of the Ministry of Human Capacities to develop the intervention protocol. BB was funded by a postdoctoral fellowship award by Team SCOUP – Sexuality and Couples – Fonds de recherche du Québec, Société et Culture during the finalization of the paper.
Authors' contribution
BB, CB, MPS, ZD, and GO set up the initial idea and plan for this study. BB did the first draft of the paper and prepared the final manuscript. BB and GO developed the intervention of study arm 1. CB programmed and implemented the intervention website. CB, MPS, and ZD helped throughout the development of the intervention and gave valuable feedback to the present study protocol. All authors approved the final version of the manuscript submitted for publication. BB is the guarantor.
Conflict of interest
The authors declare no conflict of interest.
Acknowledgments
We would like to thank László Naske and Ivett Lénárt for creating the design and the graphics of the program.
References
- Altman, D. (2014). The mindfulness toolbox: 50 practical tips, tools & handouts for anxiety, depression, stress & pain. PESI Publishing & Media. [Google Scholar]
- Amann, M., Haug, S., Wenger, A., Baumgartner, C., Ebert, D. D., Berger, T., et al. (2018). The effects of social presence on adherence-focused guidance in problematic cannabis users: Protocol for the CANreduce 2.0 randomized controlled trial. JMIR Research Protocols, 7(1), e30. 10.2196/resprot.9484. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Aronson, E. (1999). The power of self-persuasion. American Psychologist, 54(11), 875–884. 10.1037/h0088188. [DOI] [Google Scholar]
- Asner-Self, K. K., Schreiber, J. B., & Marotta, S. A. (2006). A cross-cultural analysis of the brief symptom inventory-18. Cultural Diversity and Ethnic Minority Psychology, 12(2), 367–375. 10.1037/1099-9809.12.2.367. [DOI] [PubMed] [Google Scholar]
- Baumgartner, C., Bilevicius, E., Khazaal, Y., Achab, S., Schaaf, S., Wenger, A., et al. (2019). Efficacy of a web-based self-help tool to reduce problem gambling in Switzerland: Study protocol of a two-armed randomised controlled trial. BMJ Open, 9(12), e032110. 10.1136/bmjopen-2019-032110. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Blankenship, R., & Laaser, M. (2004). Sexual addiction and ADHD: Is there a connection? Sexual Addiction & Compulsivity, 11(1–2), 7–20. 10.1080/10720160490458184. [DOI] [Google Scholar]
- Bőthe, B., Bartók, R., Tóth-Király, I., Reid, R. C., Griffiths, M. D., Demetrovics, Z., et al. (2018a). Hypersexuality, gender, and sexual orientation: A large-scale psychometric survey study. Archives of Sexual Behavior, 47(8), 2265–2276. 10.1007/s10508-018-1201-z. [DOI] [PubMed] [Google Scholar]
- Bőthe, B., Koós, M., Tóth-Király, I., Orosz, G., & Demetrovics, Z. (2019a). Investigating the associations of adult ADHD symptoms, hypersexuality, and problematic pornography use among men and women on a largescale, non-clinical sample. The Journal of Sexual Medicine, 16(4), 489–499. 10.1016/J.JSXM.2019.01.312. [DOI] [PubMed] [Google Scholar]
- Bőthe, B., Tóth-Király, I., Demetrovics, Z., & Orosz, G. (2017). The pervasive role of sex mindset: Beliefs about the malleability of sexual life is linked to higher levels of relationship satisfaction and sexual satisfaction and lower levels of problematic pornography use. Personality and Individual Differences, 117, 15–22. [Google Scholar]
- Bőthe, B., Tóth‐Király, I., Potenza, M. N., Orosz, G., & Demetrovics, Z. (2020). High-frequency pornography use may not always be problematic. The Journal of Sexual Medicine, 17(4), 793–811. [DOI] [PubMed] [Google Scholar]
- Bőthe, B., Tóth‐Király, I., Demetrovics, Z., & Orosz, G., (2020). The short version of the Problematic Pornography Consumption Scale (PPCS-6): A reliable and valid measure in general and treatment-seeking populations. The Journal of Sex Research, 1–11. [DOI] [PubMed] [Google Scholar]
- Bőthe, B., Tóth-Király, I., Zsila, Á., Griffiths, M. D., Demetrovics, Z., & Orosz, G. (2018b). The development of the problematic pornography consumption scale (PPCS). The Journal of Sex Research, 55(3), 395–406. 10.1080/00224499.2017.1291798. [DOI] [PubMed] [Google Scholar]
- Crosby, J. M. (2011). Acceptance and commitment therapy for the treatment of compulsive pornography use: A randomized clinical trial. Utah State University. [Google Scholar]
- Crosby, J. M., & Twohig, M. P. (2016). Acceptance and commitment therapy for problematic internet pornography use: A randomized trial. Behavior Therapy, 47(3), 355–366. 10.1016/j.beth.2016.02.001. [DOI] [PubMed] [Google Scholar]
- Dhuffar, M. K., & Griffiths, M. D. (2015). A systematic review of online sex addiction and clinical treatments using CONSORT evaluation. Current Addiction Reports, 2(2), 163–174. 10.1007/s40429-015-0055-x. [DOI] [Google Scholar]
- Dhuffar, M. K., & Griffiths, M. D. (2016). Barriers to female sex addiction treatment in the UK. Journal of Behavioral Addictions, 5(4), 562–567. 10.1556/2006.5.2016.072. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Diener, E., Emmons, R. A., Larsen, R. J., & Griffin, S. (1985). The satisfaction with life scale. Journal of Personality Assessment, 49(1), 71–75. 10.1207/s15327752jpa4901_13. [DOI] [PubMed] [Google Scholar]
- Dube, P., Kroenke, K., Bair, M. J., Theobald, D., & Williams, L. S. (2010). P4 suicidality screener. Primary Care Companion to the Journal of Clinical Psychiatry, 12(6), 6–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fernandez, D. P., & Griffiths, M. D. (2019). Psychometric instruments for problematic pornography use: A systematic review. Evaluation & The Health Professions, 1–71. 10.1177/0163278719861688. [DOI] [PubMed] [Google Scholar]
- Ford, J. J., Durtschi, J. A., & Franklin, D. L. (2012). Structural therapy with a couple battling pornography addiction. American Journal of Family Therapy, 40(4), 336–348. 10.1080/01926187.2012.685003. [DOI] [Google Scholar]
- Garcia, F. D., Assumpção, A. A., Malloy-diniz, L., De Freitas, A. A. C., Delavenne, H., & Thibaut, F. (2016). A comprehensive review of psychotherapeutic treatment of sexual addiction. Journal of Groups in Addiction & Recovery, 11(1), 59–71. 10.1080/1556035X.2015.1066726. [DOI] [Google Scholar]
- Gola, M., Lewczuk, K., & Skorko, M. (2016). What matters: Quantity or quality of pornography use? Psychological and behavioral factors of seeking treatment for problematic pornography use. The Journal of Sexual Medicine, 13(5), 815–824. 10.1016/j.jsxm.2016.02.169. [DOI] [PubMed] [Google Scholar]
- Griffiths, M. D. (2005). A “components” model of addiction within a biopsychosocial framework. Journal of Substance Use, 10(4), 191–197. 10.1080/14659890500114359. [DOI] [Google Scholar]
- Griffiths, M. D. (2012). The use of online methodologies in studying paraphilias – A review. Journal of Behavioral Addictions, 1(4), 143–150. 10.1556/JBA.1.2012.4.1. [DOI] [PubMed] [Google Scholar]
- Group, W. A. W. (2002). The alcohol, smoking and substance involvement screening test (ASSIST): Development, reliability and feasibility. Addiction, 97(9), 1183–1194. 10.1046/j.1360-0443.2002.00185.x. [DOI] [PubMed] [Google Scholar]
- Grubbs, J. B., Kraus, S. W., & Perry, S. L. (2019). Self-reported addiction to pornography in a nationally representative sample: The roles of use habits, religiousness, and moral incongruence. Journal of Behavioral Addictions, 8(1), 88–93. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Grubbs, J. B., Volk, F., Exline, J. J., & Pargament, K. I. (2015). Internet pornography use: Perceived addiction, psychological distress, and the validation of a brief measure. Journal of Sex & Marital Therapy, 41(1), 83–106. 10.1080/0092623X.2013.842192. [DOI] [PubMed] [Google Scholar]
- Gyollai, Á., Simor, P., Köteles, F., & Demetrovics, Z. (2011). Psychometric properties of the Hungarian version of the original and the short form of the Positive and Negative Affect Schedule (PANAS). Neuropsychopharmacologia Hungarica, 13(2), 73–79. [PubMed] [Google Scholar]
- Haug, S., Castro, R. P., Wenger, A., & Schaub, M. P. (2018). Efficacy of a mobile phone-based life-skills training program for substance use prevention among adolescents: Study protocol of a cluster-randomised controlled trial. BMC Public Health, 18(1102), 1–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Herrero, R., Mira, A., Cormo, G., Etchemendy, E., Baños, R., García-Palacios, A., et al. (2019). An Internet based intervention for improving resilience and coping strategies in university students: Study protocol for a randomized controlled trial. Internet Interventions, 16, 43–51. 10.1016/j.invent.2018.03.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hook, J. N., Reid, R. C., Penberthy, J. K., Davis, D. E., & Jennings, D. J. (2014). Methodological review of treatments for nonparaphilic hypersexual behavior. Journal of Sex & Marital Therapy, 40(4), 294–308. 10.1080/0092623X.2012.751075. [DOI] [PubMed] [Google Scholar]
- Kafka, M. P., & Hennen, J. (2002). A DSM-IV axis I comorbidity study of males (n = 120) with paraphilias and paraphilia-related disorders. Sexual Abuse: A Journal of Research and Treatment, 14(4), 349–366. 10.1177/107906320201400405. [DOI] [PubMed] [Google Scholar]
- Kaplan, M. S., & Krueger, R. B. (2010). Diagnosis, assessment, and treatment of hypersexuality. The Journal of Sex Research, 47(2–3), 181–198. 10.1080/00224491003592863. [DOI] [PubMed] [Google Scholar]
- Kraus, S. W., Potenza, M. N., Martino, S., & Grant, J. E. (2015). Examining the psychometric properties of the Yale-Brown Obsessive-Compulsive Scale in a sample of compulsive pornography users. Comprehensive Psychiatry, 59, 117–122. 10.1016/j.comppsych.2015.02.007. [DOI] [PubMed] [Google Scholar]
- Kraus, S. W., & Rosenberg, H. (2014). The pornography craving questionnaire: Psychometric properties. Archives of Sexual Behavior, 43(3), 451–462. 10.1007/s10508-013-0229-3. [DOI] [PubMed] [Google Scholar]
- Kraus, S. W., Rosenberg, H., Martino, S., Nich, C., & Potenza, M. N. (2017). The development and initial evaluation of the pornography-use avoidance self-efficacy scale. Journal of Behavioral Addictions, 6(3), 354–363. 10.1556/2006.6.2017.057. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lewczuk, K., Szmyd, J., Skorko, M., & Gola, M. (2017). Treatment seeking for problematic pornography use among women. Journal of Behavioral Addictions, 6(4), 445–456. 10.1556/2006.6.2017.063. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lewczuk, K., Wojcik, A., & Gola, M. (2019). Increase in the prevalence of onliine pornography use – Objective data analysis from the period between 2004 and 2016 in Poland. 1–45. 10.31234/osf.io/tmn4r. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Mark, K. P., Herbenick, D., Fortenberry, J. D., Sanders, S., & Reece, M. (2014). A psychometric comparison of three scales and a single-item measure to assess sexual satisfaction. The Journal of Sex Research, 51(2), 159–169. 10.1080/00224499.2013.816261. [DOI] [PubMed] [Google Scholar]
- Meichenbaum, D. (1977). Cognitive behaviour modification. Scandinavian Journal of Behaviour Therapy, 6(4), 185–192. 10.1080/16506073.1977.9626708. [DOI] [Google Scholar]
- Miller, W. R., Tonigan, J. S., & Longabaugh, R. (1995). The drinker inventory of consequences (Project MATCH monograph No. 4). Rockville: US Department of Health and Human Services. [Google Scholar]
- Minarcik, J. (2016). Proposed treatment of problematic pornography use: A cognitive-behavioral approach. University of Arkansas. [Google Scholar]
- Naficy, H., Samenow, C. P., & Fong, T. W. (2013). A review of pharmacological treatments for hypersexual disorder. Sexual Addiction & Compulsivity, 20(1–2), 139–153. 10.1080/10720162.2013.769843. [DOI] [Google Scholar]
- Nelson, L. J., Padilla-Walker, L. M., & Carroll, J. S. (2010). “I believe it is wrong but I still do it”: A comparison of religious young men who do versus do not use pornography. Psychology of Religion and Spirituality, 2(3), 136–147. 10.1037/a0019127. [DOI] [Google Scholar]
- Perry, S. L. (2019). Is the link between pornography use and relational happiness really more about masturbation? Results from two national surveys. The Journal of Sex Research, 1–13. 10.1080/00224499.2018.1556772. [DOI] [PubMed] [Google Scholar]
- Price, J., Patterson, R., Regnerus, M., & Walley, J. (2016). How much more XXX is generation X consuming? Evidence of changing attitudes and behaviors related to pornography since 1973. The Journal of Sex Research, 53(1), 12–20. 10.1080/00224499.2014.1003773. [DOI] [PubMed] [Google Scholar]
- Raymond, N. C., Coleman, E., & Miner, M. H. (2003). Psychiatric comorbidity and compulsive/impulsive traits in compulsive sexual behavior. Comprehensive Psychiatry, 44(5), 370–380. 10.1016/S0010-440X(03)00110-X. [DOI] [PubMed] [Google Scholar]
- Reid, R. C. (2007). Assessing readiness to change among clients seeking help for hypersexual behavior. Sexual Addiction & Compulsivity, 14(3), 167–186. 10.1080/10720160701480204. [DOI] [Google Scholar]
- Reid, R. C., Carpenter, B. N., Gilliland, R., & Karim, R. (2011). Problems of self-concept in a patient sample of hypersexual men with attention-deficit disorder. Journal of Addiction Medicine, 5(2), 134–140. 10.1097/ADM.0b013e3181e6ad32. [DOI] [PubMed] [Google Scholar]
- Reid, R. C., Davtian, M., Lenartowicz, A., Torrevillas, R. M., & Fong, T. W. (2013). Perspectives on the assessment and treatment of adult ADHD in hypersexual men. Neuropsychiatry, 3(3), 295–308. 10.2217/npy.13.31. [DOI] [Google Scholar]
- Rissel, C., Richters, J., de Visser, R. O., McKee, A., Yeung, A., & Caruana, T. (2017). A profile of pornography users in Australia: Findings from the second Australian study of health and relationships. The Journal of Sex Research, 54(2), 227–240. [DOI] [PubMed] [Google Scholar]
- Rollnick, S., & Miller, W. R. (1995). What is motivational interviewing? Behavioural and Cognitive Psychotherapy, 23(4), 325–334. 10.1017/S135246580001643X. [DOI] [PubMed] [Google Scholar]
- Rooke, S., Copeland, J., Norberg, M., Hine, D., & McCambridge, J. (2013). Effectiveness of a self-guided web-based cannabis treatment program: Randomized controlled trial. Journal of Medical Internet Research, 15(2), e26. 10.2196/jmir.2256. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rozental, A., Boettcher, J., Andersson, G., Schmidt, B., & Carlbring, P. (2015). Negative effects of internet interventions: A qualitative content analysis of patients’ experiences with treatments delivered online. Cognitive Behaviour Therapy, 44(3), 223–236. 10.1080/16506073.2015.1008033. [DOI] [PubMed] [Google Scholar]
- Scanavino, M. D. T., Ventuneac, A., Helena, C., Abdo, N., Tavares, H., Amaral, M. L. S. D., et al. (2013). Compulsive sexual behavior and psychopathology among treatment-seeking men in Sao Paulo, Brazil. The Journal of Sexual Medicine, 10, 310. 10.1016/j.psychres.2013.01.021. [DOI] [PubMed] [Google Scholar]
- Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. 10.1037/1082-989X.7.2.147. [DOI] [PubMed] [Google Scholar]
- Schaub, M. P., Blankers, M., Lehr, D., Boss, L., Riper, H., Dekker, J., et al. (2016). Efficacy of an internet-based self-help intervention to reduce co-occurring alcohol misuse and depression symptoms in adults: Study protocol of a three-arm randomised controlled trial. BMJ Open, 6(5), e011457. 10.1136/bmjopen-2016-011457. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schaub, M. P., Haug, S., Wenger, A., Berg, O., Sullivan, R., Beck, T., et al. (2013). Can reduce - the effects of chat-counseling and web-based self-help, web-based self-help alone and a waiting list control program on cannabis use in problematic cannabis users: A randomized controlled trial. BMC Psychiatry, 13(1), 305. 10.1186/1471-244X-13-305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schaub, M. P., Wenger, A., Berg, O., Beck, T., Stark, L., Buehler, E., et al. (2015). A web-based self-help intervention with and without chat counseling to reduce cannabis use in problematic cannabis users: Three-arm randomized controlled trial. Journal of Medical Internet Research, 17(10), e232. 10.2196/jmir.4860. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sniewski, L., & Farvid, P. (2019). Abstinence or acceptance? A case series of men’s experiences with an intervention addressing self-perceived problematic pornography use. Sexual Addiction & Compulsivity, 26(3–4), 191–210. 10.1080/10720162.2019.1645058. [DOI] [Google Scholar]
- Sniewski, L., Farvid, P., & Carter, P. (2018). The assessment and treatment of adult heterosexual men with self-perceived problematic pornography use: A review. Addictive Behaviors, 77, 217–224. 10.1016/j.addbeh.2017.10.010. [DOI] [PubMed] [Google Scholar]
- Ustun, B., Adler, L. A., Rudin, C., Faraone, S. V., Spencer, T. J., Berglund, P., et al. (2017). The world health organization adult attention-deficit/hyperactivity disorder self-report screening scale for DSM-5. JAMA Psychiatry, 74(5), 520–526. 10.1001/jamapsychiatry.2017.0298. [DOI] [PMC free article] [PubMed] [Google Scholar]
- von Franqué, F., Klein, V., & Briken, P. (2015). Which techniques are used in psychotherapeutic interventions for nonparaphilic hypersexual behavior? Sexual Medicine Reviews, 3(1), 3–10. 10.1002/smrj.34. [DOI] [PubMed] [Google Scholar]
- Walton, G. M. (2014). The new science of wise psychological interventions. Current Directions in Psychological Science, 23(1), 73–82. 10.1177/0963721413512856. [DOI] [Google Scholar]
- Walton, G. M., & Wilson, T. D. (2018). Wise interventions: Psychological remedies for social and personal problems. Psychological Review, 125(5), 617–655. 10.1037/rev0000115. [DOI] [PubMed] [Google Scholar]
- Wantland, D. J., Portillo, C. J., Holzemer, W. L., Slaughter, R., & McGhee, E. M. (2004). The effectiveness of web-based vs. non-web-based interventions: A meta-analysis of behavioral change outcomes. Journal of Medical Internet Research, 6(4), e40. 10.2196/jmir.6.4.e40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Weisel, K. K., Lehr, D., Heber, E., Zarski, A. C., Berking, M., Riper, H., et al. (2018). Severely burdened individuals do not need to be excluded from internet-based and mobile-based stress management: Effect modifiers of treatment outcomes from three randomized controlled trials. Journal of Medical Internet Research, 20(6), e211. 10.2196/jmir.9387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wéry, A., & Billieux, J. (2017). Problematic cybersex: Conceptualization, assessment, and treatment. Addictive Behaviors, 64, 238–246. 10.1016/j.addbeh.2015.11.007. [DOI] [PubMed] [Google Scholar]
- Wéry, A., Vogelaere, K., Challet-Bouju, G., Poudat, F.X., Caillon, J., Lever, D., et al. (2016). Characteristics of self-identified sexual addicts in a behavioral addiction outpatient clinic. Journal of Behavioral Addictions, 5(4), 623–630. 10.1556/2006.5.2016.071. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Willoughby, B. J., Busby, D. M., & Young-Petersen, B. (2019). Understanding associations between personal definitions of pornography, using pornography, and depression. Sexuality Research and Social Policy, 16(3), 342–356. 10.1007/s13178-018-0345-x. [DOI] [Google Scholar]
- Willoughby, B. J., Carroll, J. S., Nelson, L. J., & Padilla-Walker, L. M. (2014). Associations between relational sexual behaviour, pornography use, and pornography acceptance among US college students. Culture, Health and Sexuality, 16(9), 1052–1069. 10.1080/13691058.2014.927075. [DOI] [PubMed] [Google Scholar]
- World Health Organization . (2019). International statistical classification of diseases and related health problems (11th ed.). [Google Scholar]
- Yeager, D. S., & Walton, G. M. (2011). Social-psychological interventions in education: They’re not magic. Review of Educational Research, 81(2), 267–301. 10.3102/0034654311405999. [DOI] [Google Scholar]