Skip to main content
NCBI home page
Internet Interventions logoLink to Internet Interventions
. 2024 May 15;37:100750. doi: 10.1016/j.invent.2024.100750

Effectiveness and uptake of a transdiagnostic emotion regulation mobile intervention among university students: Protocol for a randomized controlled trial

Tajda Laure 1,⁎, Marilisa Boffo 1, Rutger CME Engels 1, Danielle Remmerswaal 1
PMCID: PMC11141155  PMID: 38827123

Abstract

Background

Going to university is a major life event, which can be stressful and negatively affect mental health. However, it also presents an opportunity to establish a foundation for positive life trajectories. To support university students, a mobile transdiagnostic emotion regulation (ER) intervention has been developed, offering both broad-based (universal) and targeted (indicated) preventative support. ER, a transdiagnostic factor underlying various mental health problems, is a critical intervention target in students, a demographic particularly susceptible to mental health issues. Cultivating ER can help manage immediate stressors and foster long-term wellbeing. This paper describes the study protocol for a Randomized Controlled Trial (RCT) evaluating the effectiveness and uptake of such mobile transdiagnostic ER intervention.

Method

The superiority parallel-group RCT involves 250 participants randomized to either the intervention condition (i.e., full access to the mobile intervention, (n = 125) or to a waitlist control condition (n = 125). Primary outcomes include ER skills and stress symptoms. Secondary outcomes include mental health parameters (anxiety, depression, resilience) and intervention uptake (i.e., objective engagement, subjective engagement, ER skills application in real life). Outcomes are assessed at baseline, week 3, 8 and 12, with continuous log-data collection for user engagement.

Discussion

This study evaluates the effectiveness and uptake of a transdiagnostic ER mobile intervention for the student population addressing their ER developmental needs. If successful, the results will validate our approach to intervention development and whether focusing on learning transfer (i.e., application of the learnt skills in real-life) and personalization using a recommendation system, can boost the real-world application of skills and intervention impact.

Keywords: Mobile intervention, Transdiagnostic, Mental health, Emotion regulation, University students, Randomized controlled trial

Highlights

  • •

    Emerging adulthood is a period of life where young adults are vulnerable to the development of mental health problems.

  • •

    The ROOM app was designed to facilitate adaptive emotion regulation skills among university students.

  • •

    Learning adaptive emotion regulation skills can prevent various mental health problems due to its transdiagnostic nature.

  • •

    ROOM recommends relevant content and helps users apply emotion regulation skills in real-life settings.

  • •

    This study uses an RCTdesign to evaluate ROOM’s impact on mental health, emotion regulation skills, and user experience.

1. Introduction

The time at university goes beyond academic development; it is a transformative phase in individuals' lives marked by exploration and personal growth (Arnett, 2007). Students face many challenges, such as academic pressures, navigating interpersonal relationships, financial stressors, and career planning (Acharya et al., 2018). For many, this is the first time they need to manage life's challenges with increased self-reliance in an environment less structured than what they were used to before. This transitional phase termed emerging adulthood (Arnett, 2007), introduces an uncharted territory with a unique set of stressors for which students are often not fully prepared. Unsurprisingly, research indicates that this demographic is susceptible to mental health problems, with nearly 30 % of students worldwide experiencing at least one such problem (Auerbach et al., 2016, Auerbach et al., 2018) with depression, anxiety, and substance use disorder being the most common (Pedrelli et al., 2015). While this period is marked by increased rates of mental health problems, it can also act as a springboard for shaping positive adult life trajectories (Schulenberg et al., 2004). In this developmental stage, preventative interventions and promotion of resilience and wellbeing can have a lasting impact (Stengård and Appelqvist-Schmidlechner, 2010). Fostering development and utilization of adaptive emotional regulation skills has emerged as a prime target for interventions within this population (Mouatsou and Koutra, 2023).

1.1. Emotional regulation skills

Emotion regulation skills, considered vital for healthy development, involve regulating emotional responses to perceived stressors through strategies like distraction, reappraisal, affect modulation or suppression (Gross, 2014). They evolve considerably from late adolescence through early adulthood, with the rate of this progression differing based on individual characteristics and contextual influences (Liew et al., 2023). Research shows that during the first two years of university, students increasingly rely on maladaptive emotion regulation strategies like substance use and avoidant behaviors while the use of adaptive emotion regulation strategies, like cognitive reappraisal, diminishes (Conley et al., 2020). This tendency towards avoidant behaviors may stem from the continuation of brain maturation in emerging adulthood. The maturation of the prefrontal cortex leads to increases in future-oriented behaviors and decreases in risk-taking ones through improved processing of incentives and rewards alongside the regulatory control and behavior selection. This improvement in the prefrontal cortex functioning fosters a shift in the attention from positive feedback to negative feedback, which drives the reduction of approach and increases of avoidance behavior. Although this process is crucial in emerging adulthood, it may also explain why students engage in avoidance behaviors and why this period poses a risk for developing mental health issues (Taber-Thomas and Pérez-Edgar, 2014). Maladaptive emotion regulation strategies such as avoidance can provide short-term relief. However, when they are used consistently across various situations, they can have negative effects on individuals' mental health, contributing to depression, anxiety, substance abuse, and eating disorders (Berking and Wupperman, 2012; Yoon and Rottenberg, 2020). Because of its role in the development and maintenance of various mental health problems, emotion (dys)regulation is recognized as a transdiagnostic factor (Fernandez et al., 2016). Consequentially, interventions targeting emotion regulation address the root cause of multiple mental health problems such as mood and anxiety disorders. Equipping students with adaptive emotion regulation skills may help them navigate present challenges, foster resilience to adversity (Finkelstein-Fox et al., 2018; Wu et al., 2013), and lay the foundation for long-term mental well-being (Aldao et al., 2016; Gatto et al., 2022).

1.2. Wellbeing interventions for university students

The trend in intervention strategies is moving towards quick and digital solutions to cut costs, generate access to all students, and address the growing need for therapeutic options (Gentili et al., 2022). Digital tools align well with the characteristics of the current generation of university students, predominantly comprised of Generation Z, who are digital natives (Cohen et al., 2021; Seemiller and Grace, 2017). Technology and social media are fully integrated into their lives for communication, learning, and for acquisition of new skills (Hathaway and O'Shields, 2022). They engage with mobile apps more than any generation before (Dimock, 2019). Unsurprisingly, mental health apps are well received by them, and they view them as efficient and helpful either as an alternative to or supplement for traditional in-person treatment (Cohen et al., 2021; Holtz et al., 2023).

Although Generation-Z is more open to discuss mental health than generations before (APA, 2018), many are not willing to seek support due to mental health stigma (Cohen et al., 2021), time and financial constraints, and a preference for self-management (Harrer et al., 2019; Theurel and Witt, 2022). Therefore, e-health interventions, offering low-threshold, anonymous, and flexible access, may present a cost-effective way to overcome some of these barriers and promote mental health among the university students (Anderson et al., 2016; Harrer et al., 2019;Davies et al., 2014; Ferrari et al., 2022; Harrer et al., 2019; Lattie et al., 2019). However, the evidence supporting digital solutions for this group is inconsistent. Scientists emphasize the need for more rigorous and transparent studies (Davies et al., 2014; Harrer et al., 2019; Lattie et al., 2019), with an added focus on user experience and engagement to ensure impactful and sustainable implementation on campuses, an aspect often overlooked (Lattie et al., 2019).

1.2.1. Why another mental health app amid thousands of existing ones?

Approximately 15.000 mental health apps are on the market, yet only about 3–4 % of them are founded on scientific evidence (Eis et al., 2022). Few commercially available apps are specifically designed to address the unique challenges and experiences of university students. General mental health apps have been found to not meet students' needs which include privacy and robust security measures, intuitive, user-friendly interfaces; credible, informative content; customizability (e.g., being able to set notifications) (Melcher et al., 2022). Financial considerations are another significant factor; most students are only willing to use apps that are free or offer free trial periods (Melcher et al., 2022). Similarly, Becker and Torous (2019) added that this population values efficiency, system responsiveness, and content relevance, which seems to lack in existing mental health apps. Creating an evidence-based mental health app that is tailored to student needs and wants may improve its acceptability and, in turn, its impact on the targeted outcomes.

1.2.2. The Erasmus University Rotterdam Student wellbeing app

In 2019, Erasmus University Rotterdam (EUR) initiated the development of a mobile app with both a universal (e.g., increasing mental health literacy) and indicated (e.g., supporting those displaying signs of distress) prevention strategy, to be made freely available to all EUR students. The development followed the Center for e-Health Research (CeHRes) roadmap, a holistic framework for creating e-health tools (van Gemert-Pijnen et al., 2011). This framework combines human-centered design principles and scientific research, incorporating ongoing, iterative evaluations across all stages of development. It actively involves target users (i.e., university students) and key stakeholders in the evaluation cycles to ensure the tool aligns with end-users' needs and its context of use (Kip et al., 2022; van Gemert-Pijnen et al., 2011). The app's fundamental feature is its transdiagnostic approach to mental health. Instead of targeting specific mental health problems like anxiety or depression, the intervention focuses on emotion regulation – a key transdiagnostic factor - to increase resilience and prevent mental health issues among this population (Barlow et al., 2020). The app offers a library of digital interventions promoting self-awareness, mental health literacy, and the development of adaptive emotion regulation skills (e.g., reappraisal, self-soothing, acceptance, modulation of negative affect), supporting students in managing external (e.g., failing a test) and internal stressors (e.g., automatic negative thoughts). To ensure that every student finds content corresponding to their needs and preferences, the app features techniques from various therapeutic approaches, including Positive Psychology (PP) (Seligman, 2002), Acceptance and Commitment Therapy (Hayes et al., 2006), Self-Compassion (Neff et al., 2005), Mindfulness (Creswell, 2017), and Cognitive Behavioral Therapy (Beck, 2011). The app content is evidence-based and has undergone extensive testing throughout the development stages. Specifically, user experience with and impact of active intervention ingredients (i.e., mental health exercises) were evaluated via experimental (Laure et al., 2024, Manuscript submitted for publication) and formative evaluations (Villegas Mejia et al., 2024).

To address students' growing privacy concerns, the app development followed privacy by design principles complying with the highest standards of privacy regulations to protect students' confidential data. The app intentionally does not incorporate GPS tracking or mobile sensors, as students are reluctant to being tracked (Melcher et al., 2020, Melcher et al., 2022). While the app includes a recommendation system to enhance content personalization and responsiveness of the tool, this complies with the privacy-by-design approach and maintains the user's freedom to choose the content they engage with.

Lastly, unlike many commercially available apps that aim to maximize screen time for revenue, this app is offered at no cost for students and prioritizes real-world application of learned strategies (i.e., learning transfer) (Villegas Mejía et al., 2024; Villegas Mejia et al., 2024) over prolonged engagement with the app itself. This also addresses users' concerns about screen time stress (Nakshine et al., 2022). Thus, the app features a minimalist, calming design and incorporates tangible tools, such as implementation intentions and associative cues, to help users integrate new skills into their daily lives (i.e., generalization and habit formation; Villegas Mejia et al., 2024).

1.3. Study objectives

This study focuses on the summative evaluation phase of the CeHRes roadmap (van Gemert-Pijnen et al., 2011) and examines the app's impact and uptake among distressed university students using a superiority parallel-group Randomized Controlled Trial (RCT) in a naturalistic setting (i.e., daily life of students). The focus is on distressed students for two main reasons. First, prior research (Melcher et al., 2022; Villegas Mejia et al., 2024; Laure et al., 2024) indicates that students are more inclined to use mental health apps when they already experience some degree of stress. Second, focusing on distressed students allows assessment of the app's impact on a population for which the app's content is highly relevant and likely stands to gain the most from this intervention. Participants receiving access to the mobile intervention for 12 weeks are compared to a wait-list control group. Those with access to the intervention are encouraged to engage with it for a minimum of three weeks daily. However, they are free to use it for as long and as much as they want. Outcome assessments are conducted at weeks 3, 8, and 12 to observe short to medium-term changes as well as fluctuations in the primary outcomes over 3 months.

  • (i)

    Primary outcomes. These focus on the intervention primary targets, i.e., stress symptoms and emotion regulation skills. In line with our previous study (Laure et al., 2023, Laure et al., 2024), it is hypothesized that students with access to the app will report improvements in their perceived stress symptoms and emotion regulation skills compared to the waitlist group.

  • (ii)

    Secondary outcomes. These include depressive and anxiety symptoms and resilience levels. These outcomes are considered as secondary, distal targets as they are not the direct focus of the intervention but are expected to increase because of enhanced adaptive emotion regulation skills. We expect decreases in levels of depressive and anxiety symptoms and increases in resilience levels after 12 weeks in the intervention group compared to the waitlist group.

  • (iii)

    Secondary outcomes. Another secondary goal is to analyze the app uptake and users' learning experience by examining engagement metrics and participants' subjective experience (i.e., helpfulness, likability) with the app components, and understanding the extent to which users apply the learned skills in real-life settings. Based on our previous studies, user tests, and the amount of content available, we expect users will be the most engaged with the app for the first two to three weeks, where a peak in learning and practicing with the help of the app is expected. Subsequently, we expect users to increasingly apply the learnt skills in their daily lives, with app usage reducing as they start accessing the app only for additional support or memory refreshment to practice the techniques.

Finally, to gain further insight into how distinct groups (e.g., bachelor/master, gender identity) differ in app usage, exploratory analyses will be carried out. This information will inform decisions on how to improve, promote, and adapt the app to best serve its diverse user base.

2. Materials and methods

The study was approved by the Institutional Review Board of Erasmus University Rotterdam (Reference: ETH2324–0193) and has been registered at ClinicalTrials.gov (Identifier: NCT06224647).

2.1. Study sample

A sample of 250 EUR students is recruited to participate in the study. The sample size was calculated using G*Power 3.1.9.7 for a mixed ANOVA design with two groups and four measurement points. Based on our previous study (Laure et al., 2023, Laure et al., 2024) and existing meta-analyses (Ang et al., 2022; Harrer et al., 2019; Lattie et al., 2019), the estimated effect size for our primary outcomes (stress symptoms and emotion regulation skills) is assumed to be small (F = 0.1), with an α of 0.05, power of 0.80, and a moderate correlation among repeated measures (0.5). Based on these parameters, the total required sample size is 138. To account for dropout, which is generally high in digital intervention studies [i.e., > 40 %] (Torous et al., 2020) we are increasing the sample size to 250.

2.2. Eligibility criteria

Individuals are eligible to participate in the study when they are currently enrolled at EUR, are between 18 and 30 years old, have access to a smartphone and internet, are comfortable using the English language for verbal and written communication. The study focuses on distressed university students; therefore, only those with a Perceived Stress Scale score above 13 (moderate to high stress) are eligible (Cohen et al., 1983). Individuals with scores higher than 20 on the Patient Health Questionnaire [PHQ-9] scale (Kroenke et al., 2001), an active official medical diagnosis of psychosis, bipolar disorder, clinical depression, or anxiety disorder, and/or are undergoing pharmacological treatment, and/or treatment with experimental drugs are ineligible to participate in this study. The exclusion criteria were set to minimize the risk of worsening severe mental health issues and safeguard participants, while accurately assessing the intervention's safety and effectiveness.

2.3. Randomization

To achieve a balanced representation in terms of gender, education level, and student origin between control and intervention groups a stratified randomization strategy is implemented. Stratification takes place once the target group of eligible participants is reached. Participants are classified in strata based on three criteria: gender identity (categorized as male, female, others), education level (categorized as bachelor and master), and student origin (categorized as national or international).

Following the stratification, randomization occurs within each stratum to allocate participants into either the control or intervention group, with 1:1 allocation ratio. The stratified randomization approach ensures that both groups are comparable in terms of these key characteristics thereby reducing impact of confounders and enhancing the validity and reliability of results (Kang et al., 2008).

The randomization and allocation process are automatically done by Qualtrics, reducing the possibility of selection bias. Participants receive an automatic email generated by Qualtrics informing them about their group allocation. Those allocated to the intervention group receive a link to the mobile application with further instructions related to the app installation and the assessments. Those allocated to the waitlist group are informed they have been placed on a waitlist and receive instructions regarding the assessments. The randomization process, condition allocation, and any deviations from the initial assignments will be documented in the study results for transparency and reproducibility.

2.3.1. Intervention and control condition

Participants randomized to the intervention condition receive full access to the ROOM app intervention while participants randomized to the waitlist group receive access to the mobile intervention upon completion of the trial (i.e., after completing the last follow-up survey). Both groups are asked to complete assessments at four time points, with intervention condition receiving additional questions related to the engagement aspect of the intervention at the second and final follow-up assessment.

2.4. Mobile transdiagnostic emotion regulation intervention

This intervention was designed to offer participants a personalized and interactive approach to managing and understanding their emotional well-being. It includes four main elements: (a) daily monitoring of emotional states, (b) a library of exercises placed in six intervention categories, (c) a self-assessment module, (d) a space for creating a personal collection of tools, and (e) a recommendation system linking users to content relevant to their needs and wishes. Each of these app features is described in detail below.

  • (a)

    Daily monitoring of emotional states. Upon the first daily access to the app, participants are prompted to complete an ecological momentary assessment (EMA) consisting of seven emotional states (i.e., happiness, energy, relaxation, fatigue, frustration, sadness, and stress) rated on a Likert scale from 0 to 5. The results are immediately transformed into visual feedback represented by circles for each emotional state. This visual feedback serves as an intuitive and accessible gateway into the user's emotional well-being (see Fig. 1).

  • (b)
    Exercises. The exercises within the ROOM app are placed in six intervention categories: (1) Upregulation of positive affect, (2) Mindfulness, (3) Self-Compassion, (4) Breathing and relaxation, (5) Cognitive defusion, and (6) Cognitive restructuring (see Appendix A). Each exercise unfolds through four integral parts:
    • 1.
      Introduction of the Technique: Offering overview and context of the chosen method.
    • 2.
      Practicing the Technique: A guided practice session within the app, accommodating users at all levels of familiarity by offering additional information in the info icons.
    • 3.
      Debriefing Information: After completing the practice, users can learn about the typical experiences of others and receive reassurance regarding the uniqueness of everyone's experience, especially when their outcome differs from the average user's. In such instances, users are encouraged to either revisit the exercise, refine their approach to enhance its effectiveness, or explore other exercises that may align more closely with their needs.
    • 4.
      Exercise evaluation: Upon completion, participants are asked to assess exercise likability and helpfulness.
    • 5.
      Transfer element: If users find the technique beneficial, they can proceed to a section offering tips on how to integrate the technique into their daily life, promoting engagement with the techniques in real life outside of the app (i.e., learning transfer).
  • (c)

    Self-assessment module. The goal of this section is to help users deepen their self-understanding, by assessing both stable trait-like (e.g., personality traits) and state-like characteristics (e.g., distress symptoms). This module includes 14 validated questionnaires evaluating: perfectionism (Burgess et al., 2016), general self-efficacy (Chen et al., 2004), life purpose (Crumbaugh and Hentrion, 1988), cognitive fusion (Gillanders et al., 2014), tolerance of uncertainty (Huntley et al., 2020), values (Self-developed), emotion regulation (Gross and John, 2003), mindfulness (Carlson and Brown, 2005), loneliness (Gierveld and Van Tilburg, 2006), wellbeing (Stewart-Brown et al., 2009), stress (Cohen et al., 1983), depressive (Kroenke et al., 2001) and anxiety symptoms (Spitzer et al., 2006), and burnout levels (Schaufeli et al., 2002). After completing a questionnaire, users receive normative feedback on their scores and are recommended exercises to enhance relevant skills.

  • (d)

    Personal collection of tools. Each exercise is symbolized by an item commonly found in a student's room, such as lamps, plants, or lightbulbs. If users find an exercise beneficial, they can add its corresponding item to the app virtual “Room,” customizing the space to their liking (See Fig. 2).

  • (e)

    Recommendation system. Exercise recommendations are generated based on the selected and not-selected exercises so far and past liking and helpfulness ratings of completed exercises, moderated by current emotional states (positive and negative emotional scores when the daily EMAs are completed) and temporal context (day/evening and weekday/weekend). A recommender engine has been built using a federated reinforcement learning approach (i.e., single bandit models), whereby an agent model running locally on the user's phone progressively tailors exercise recommendations to the users' feedback (chosen exercises + ratings). The use of a reinforcement learning approach is aimed at optimizing the relevance of exercise choices to individual users (exploitation) while at the same time promoting some degree of exploration of techniques that may be new or challenging to them (exploration). The underlying oracle for the reinforcement learning algorithm is based on online logistic regression with ‘exercise ID’ and ‘exercise Features’ (describing elements of an exercise, such as being short, medium, or long in duration) as the main predictors plus their interactions with the EMA and temporal context. We have three separate oracles for exercise click probability upon a recommendation, liking rating and helpfulness rating. When completing their daily EMA (i.e., contextual predictor), participants are presented with three exercise suggestions. The suggested exercises are sampled without replacement with sample probabilities based on a weighted combination of predicted click probability, predicted liking rating and predicted helpfulness rating. Weighted scores are turned into probabilities using a softmax function. When participants do not complete the daily EMA, these scores do not influence the oracle predictions.

Fig. 1.

Fig. 1

Open in a new tab

Emotional State Tracking Feature in ROOM app.

Fig. 2.

Fig. 2

Open in a new tab

App virtual “Room” displaying objects linked to completed exercises.

The recommendation system additionally integrates several hardcoded safety measures. Specifically, if a user's daily EMA indicates significant stress and frustration (i.e., scores ≥4), they will be recommended relaxation and breathing exercises. If they report significant feelings of sadness (scores ≥4), mindfulness exercises will be excluded from the recommendations (see Appendix B).

2.5. Outcome measures

2.5.1. Primary outcomes

Changes in stress symptoms are measured with the Perceived Stress Scale (PSS-10; Cohen et al., 1983), a validated 10-item measure of perceived stress. Participants are instructed to indicate how often they felt or thought a certain way over the last month on a scale from 0 (never) to 4 (very often). The summed score ranges from 0 to 40, with higher scores indicating higher levels of perceived stress symptoms. The PSS-10 is used both to screen for eligibility and assess the level of stress in the study over time.

Changes in emotion regulation skills are evaluated with the Emotion Regulation Skills Questionnaire (ERSQ) (Grant et al., 2018), which measures seven distinct emotion regulation skills: awareness, sensation, clarity, understanding, acceptance, tolerance, compassionate self-support, readiness to face distressing situations, and the modification of adverse emotions. Every skill is evaluated with three items, with participants rating their frequency on a scale ranging from 0 (never) to 4 (almost always). Beyond individual subscale scores, the ERSQ provides a cumulative score, calculated as the mean of all item responses. Higher scores are indicative of higher ER skills.

2.5.2. Secondary outcomes

Changes in depressive and anxiety symptoms will be evaluated with the 9-item Patient Health Questionnaire (PHQ-9) (Kroenke et al., 2001) and the 7-items Generalized Anxiety Disorder Questionnaire (GAD-7) (Spitzer et al., 2006), respectively. Both instruments prompt participants to specify the frequency of certain symptoms experienced in the past two weeks, using a scale from 0 (never) to 3 (almost daily). The cumulative scores for PHQ-9 can span from 0 to 27, and for GAD-7 from 0 to 21. A higher score on either scale indicates higher levels of depression or anxiety symptoms, respectively.

Changes in resilience levels will be evaluated with the 10-item Connor-Davidson Resilience Scale (CD-RISC-10) (Campbell-Sills and Stein, 2007). Each item is scored on a 5-point scale ranging from 0 (not true at all) to 4 (true nearly all the time), with total scores ranging from 0 to 40. Higher scores indicate greater resilience levels.

2.5.3. Intervention uptake

The app uptake will be assessed by analyzing users' objective engagement parameters alongside their subjective experience with the app.

2.5.3.1. Objective engagement patterns

Objective engagement patterns are retrieved from app usage analytics (i.e., log data) and consist of exercise completion rate, average time spent per exercise, how many times participants started but did not finish an exercise, engagement in the self-assessment module (questionnaire completion), how many “objects” participants collected and placed in their personal collection (i.e., virtual room), and general log data regarding app access.

2.5.3.2. Subjective engagement parameters
2.5.3.2.1. Perceived likeability and helpfulness of app's components

Likeability and helpfulness of the app's features are evaluated on a 1 (not at all) to 100 (very much) scale, with exercises evaluated every time a participant engages and completes an exercise. At week 8, other components are assessed outside the app. Participants rate the Self-assessment module (MapMyself) and learning transfer elements for helpfulness and likability, the enjoyment of collecting objects in the Virtual Collectibles Room, and the relevance and usefulness of the in-app Recommendations (see Appendix C).

2.5.3.2.2. User experience with the app

At weeks 8 and 12, participants in the intervention condition will complete the 26-item User Experience Questionnaire (UEQ) (Hinderks et al., 2019) measuring Attractiveness (overall impression of the product), Perspicuity (easiness of use of the product), Efficiency (can the user solve the task without unnecessary efforts), Dependability (does the user feel in control of the interaction), and Novelty (does the product catch the interest of users) of the app. The items have the form of a semantic differential (i.e., each item is represented by two terms with opposite meanings, e.g., attractive - unattractive). The items are scaled from −3 to +3. The −3 represents the most negative answer, 0 is a neutral answer, and + 3 is the most positive answer. The scale displays sufficient reliability - the Cronbach alpha's per scale varies between α = 0.65 for Dependability scale to α = 0.89 for the Attractiveness scale.

2.5.3.2.3. Engagement with the techniques outside of the app

ROOM app includes a feature where users can report the number of times they have completed an exercise without using the app. This counter is used to evaluate the engagement with the exercises outside of the app. Additionally, a self-developed questionnaire is used to inquire how often participants independently applied the learnt technique (e.g., self-compassion, breathing technique, savoring, being mindful) in their daily lives over the past month, rated on a scale from 0 (not at all) to 4 (daily). This questionnaire complements the in-app collected data as participants might use a certain technique but not a full exercise to regulate their emotions. As such it provides a specific insight into the participants transfer of skills learnt through the app in real life environment (see Appendix C).

2.5.3.2.4. Future engagement with and likelihood of recommending the app to peers

Upon conclusion of the study, intervention condition participants are asked about the likelihood of continued use of the app by indicating how likely they are to continue using the app post-study, rated on a scale from 1 (definitely not) to 5 (definitely). In addition to that, they are asked to indicate the likelihood of recommending the app to fellow students on a scale from 1 (definitely not) to 5 (definitely).

2.6. Study procedure

Students are recruited through multiple channels, including social media platforms, faculties' screens, and intranet communications (i.e., e-mails), via study advisors, student associations, and events like the student wellbeing week at EUR. Interested participants can sign up via a Qualtrics page. As soon as the study commences, they are emailed a link to the Qualtrics page with comprehensive information about the study objectives, procedures, anticipated benefits and potential risks, remuneration details, and consideration regarding privacy and data confidentiality. After they review the study information, those deciding to participate can complete the informed consent form electronically and are automatically linked to the eligibility screening questionnaire. Eligible participants immediately receive an email with a link to the baseline survey. On day 3 and 5 they receive reminder e-mails if they fail to complete it after the initial invitation. Upon baseline survey completion, they are automatically randomized by Qualtrics software to the waitlist control or intervention condition. Participants are notified via email to which condition they are assigned to, and they receive a unique randomly generated user ID (i.e., research code). The research codes for participants in the intervention condition are activated through the Web Service feature in Qualtrics linked to the ROOM app data server gateway so participants can proceed to access the app by entering the code upon first login. The process of consenting to the study and receiving access can be completed within an hour.

As part of the app's onboarding process, users are informed about the different app features and instructed on ways to use the application. They are encouraged to explore various exercises offered in app and save their favorites in their virtual Room, curating a personalized wellbeing toolkit. Furthermore, participants can choose to: (i) receive a reminder on a third consecutive day of inactivity, (ii) participate in a 21-day challenge involving daily assessments of emotional states (i.e., EMA) and completion of one exercise, or (iii) decide not to receive reminders. The 21-day challenge is based on the previous study (Laure et al., 2023) where the analyses indicated that completing EMA and engaging with exercises daily improved participants ER skills and decreased stress symptoms (Laure et al., 2024). No additional efforts are made to further promote app use.

At week 3 (day 21), week 8 (day 56), week 12 (day 84) participants in both conditions receive an email with the link to the follow-up survey. The participants who do not complete the survey receive a reminder email on day 3 and are called by the researcher 7 days after the initial invitation to the survey is sent. The call's purpose is to identify any technical issues, and to verify whether a participant has decided to withdraw from the study. Participants can also report bugs, ask questions, give feedback, or withdraw from the study using an online form that can be accessed in the ROOM General.

Throughout the trial, participants in both conditions can maintain their usual daily activities and routines without any special directions or restrictions related to the usage of other apps or external mental health support. At the 12-week follow-up they are requested to disclose use of any type of mental health resources for the purpose of evaluation of external factors influencing the study outcomes.

Upon completion of the follow up survey at week 12, the waitlisted participants' research codes are activated, and they receive an e-mail with the instructions and links to the app.

Lastly, participants are rewarded with digital gift vouchers for survey completion. Completing the first three surveys is compensated with 5€ each, while the completion of the final survey is worth 15€. Once the study is concluded, the researcher reviews the assessment adherence at the four time points to calculate the compensation and send the gift vouchers to the participants, including the debriefing information. The compensation is designed to incentivize consistent participation throughout the four assessment points.

See Fig. 3 for an overview of the study phases and Table 1 for the overview of the assessment schedule.

Fig. 3.

Fig. 3

Open in a new tab

CONSORT participant flow chart.

Table 1.

Timeline of assessment.a

Outcomes Measures In/outside app t0 t1 t2 t3
Primary outcomes
Stress symptomsa PSS-10 Outside app x x x
Emotion regulation skills ERSQ Outside app x x x



Secondary outcomes
Depressive symptomsa PHQ-9 Outside app x
Anxiety symptoms GAD-7 Outside app x x
Resilience CD-RISC-10 Outside app x x



Engagement patterns
App usage data Log-data In-app Continuously collected throughout the study
Exercise likability and helpfulness Self-developed In-app Upon exercise completion
App features Likability/Helpfulness Self-developed Outside app x
Subjective engagement with the app UEQ Outside app x x
Use of techniques outside of the app Self-developed Outside app x x
Use of techniques outside of the app Exercise counter In-app Logged by participants in the app



Future engagement with the appb
Likelihood of continued use Self-developed Outside app x
Likelihood of recommending the app to others Self-developed Outside app x
Open in a new tab
a

PSS-14 and PHQ-9 are applied during eligibility screening and is therefore not administered at baseline assessment (t0).

b

The assessment of engagement patterns and prospective app usage is exclusive to the intervention group.

3. Data analysis plan

The data collected will be reviewed for inconsistencies or discrepancies arising from data recording or measurement. Descriptive statistics and exploratory graphs will be used to identify outliers and verify assumptions of linearity, homoscedasticity, and normality. Normality will be verified using kurtosis, skewness, alongside the Kolmogorov - Smirnov test. Results from the preliminary analyses will be detailed for each time point (T0, T1, T2, T3) and descriptive statistics will be provided for both study arms. Missing data patterns will be examined to determine the most appropriate method for handling the missing values. Sensitivity analyses will be used in case of significant levels of missing data (10 % >). Lastly, baseline variables, such as gender, study level, and student origin (national vs. international), will be assessed as potential covariates in the planned analyses using correlational and logistic regression analyses to determine significant relationships with the dependent variables.

3.1. Analyses of primary outcomes

To evaluate the intervention impact on primary outcomes (ERSQ and PSS-14 scores) across the four assessment points, a mixed ANOVA will be used, with time point as within-subjects factor and condition as the main predictor.

3.2. Analysis of secondary outcomes

3.2.1. Distress symptoms and resilience

As the primary outcomes, the scores on PHQ-9, GAD-7 and CD-RISC-10, will be compared among the two study arms using a mixed ANOVA.

3.2.2. Intervention uptake

3.2.2.1. Objective engagement

App engagement metrics will be assessed using descriptive statistics (i.e., means [M] with standard deviations [SD] and/or medians [Mdn] with interquartile ranges [IQR]). For each exercise, the completion rates and time spent will be determined. The same will be done with the questionnaires in the self-assessment module, EMA and interactions with the virtual Room. The raw usage data (e.g., time spent in the app) will be visualized against time (i.e., daily and/or weekly patterns) to observe patterns of usage and irregularities (i.e., unusually high peaks, and unexpected drops in app usage) and get a better understanding of the app engagement cycle and content lifespan.

3.2.2.2. Subjective engagement patterns

Perceived likability and helpfulness of app features (i.e., exercises, self-assessment module, virtual room, in-app recommendations), and future engagement and recommendation of the app to others will be evaluated using descriptive statistics (M, SD, Mdn, IQR) to capture central tendency and variability of responses.

Similarly, UEQ scores will be analyzed by calculating and reporting the M, Mdn, and SD, IRQ for each dimension (attractiveness, perspicuity, efficiency, dependability, novelty) separately. The scores will be compared against the benchmarks determined for the UEQ (Schrepp et al., 2017).

4. Discussion

The paper details a protocol for a two-armed RCT study evaluating the impact of a transdiagnostic mobile app intervention on students' ER skills, distress symptoms, and resilience levels. Next to that, intervention uptake will be examined by evaluating app usage patterns in tandem with participants' subjective experience with the intervention components, and the extent they generalize the learnt skills to their daily context. The latter information may explain the effects of the intervention on their ER skills, distress symptoms and resilience, or the lack thereof. Additionally, evaluating the uptake can identify relevant barriers preventing implementation of the app within the university ecosystem, like technical issues or poor user experience with certain features, guiding future refinements of the app. In a broader context, this study will inform whether our approach to the development of the tool was successful and whether focusing on learning transfer and content tailoring through a recommendation system can increase the real-life skill application, and ultimately the app impact on student wellbeing.

4.1. Strengths and limitations

This study has several strengths. First, the RCT design includes a substantial sample size, enhancing the reliability and generalizability of study results. Second, this study goes beyond traditional app usage metrics (e.g., app usage duration, task completion rates) by incorporating a broader spectrum of engagement measures, such as the application of the learnt skills outside the app, to provide a more holistic view of participant engagement with the intervention. This approach may provide deeper insights into user behaviors and help explain the intervention's impact. Third, to our knowledge this intervention is unique in its theoretical underpinning focusing on transdiagnostic factors and the use of human-centered design approach to meet students' needs and preferences. Therefore, the intervention employs a variety of therapeutic approaches under the principle that one size does not fit all. Furthermore, the intervention features content tailored to this demographic and an intelligent recommender system that links users to content relevant to their needs and preferences while protecting their privacy, in line with an ethical use of artificial intelligence systems. This research has the potential to generate insights that pave the way for further exploration and development in the field of digital interventions for student wellbeing.

The study also comes with some limitations. Specifically, the target group consists of distressed university students randomized into an intervention and a waitlist control group, the latter experiencing a delayed access to the intervention. While the waitlist control group ensures that all participants receive access to the intervention and enables comparison of the intervention against no intervention condition, it also presents some challenges. For example, it prevents blinding of participants, which may lead to expectation effects, e.g., participants in the group may report better outcomes because they expect the intervention to work, or conversely, those waitlisted could reported worse outcomes resulting from the disappointment of not receiving the intervention immediately. Next to that, knowing that they will eventually receive the intervention, waitlisted participants might postpone behavioral changes or seeking alternative help, which can affect comparative results (Patterson et al., 2016). Altogether, use of the waitlist control group may inflate the intervention's effects and make it difficult to generalize the findings to the broader student population. Lastly, the RCT spans three months, primarily examining the intervention short-term effects. While immediate effects may be evident, without a longer follow-up we cannot determine if these effects are durable.

5. Conclusion

This protocol paper presents an examination of a preventative, transdiagnostic mobile intervention for university students, aiming to gain insights into its uptake and effectiveness in improving their ER skills, mental health, and resilience. Despite limitations such as lack of blinding and the short-term nature of the RCT which necessitate careful interpretation of the findings, the insights gained will offer a foundational evaluation that can inform the refinement of the app and contribute to the development of the e-mental health and university students' wellbeing fields.

Trial status

Recruitment commenced in November 2023 and the study has begun on the 5th of February. As of 20th of April, 200 participants are participating in the trial.

CRediT authorship contribution statement

TL, MB, RCMEE, and DR designed the study. TL wrote the first draft of the manuscript. MB, RCMEE, and DR revised the manuscript. All authors contributed feedback, read, and approved the final manuscript.

Declaration of competing interest

TL, RCMEE, DR, and MB, all employees of Erasmus University Rotterdam, are directly or indirectly involved in the ROOM app intervention development.

Acknowledgments

We are grateful to all who contributed to this project. Special thanks to Oege Dijk and the rest of the Thoughtworks team for their pivotal role in the app recommender system, from development to integration, and for assisting with its description in this article. We are thankful to the IJsfontein team, especially Kelley van Evert and Ronald Benschop for preparing the app and data collection systems for this study. Finally, thanks to Djameela Dulloo, our project manager, for expertly coordinating the development of the app for the RCT. This study was funded by Erasmus Trustfonds (Grant ID: GWA.100939) and Nationaal Programma Onderwijs (NPO).

Footnotes

Appendix A

Supplementary data to this article can be found online at https://doi.org/10.1016/j.invent.2024.100750.

Appendix A. Supplementary data

Supplementary material 1

Overview of the exercises classified into six intervention categories.

mmc1.docx (27.4KB, docx)
Supplementary material 2

Recommendation system.

mmc2.docx (17.2KB, docx)
Supplementary material 3

Self-developed questionnaire for evaluation of distinct intervention features a.

mmc3.docx (17.8KB, docx)

References

  1. Acharya L., Jin L., Collins W. College life is stressful today – emerging stressors and depressive symptoms in college students. J. Am. Coll. Heal. 2018;66(7):655–664. doi: 10.1080/07448481.2018.1451869. [DOI] [PubMed] [Google Scholar]
  2. Aldao A., Gee D.G., De Los Reyes A., Seager I. Emotion regulation as a transdiagnostic factor in the development of internalizing and externalizing psychopathology: current and future directions. Dev. Psychopathol. 2016;28(4pt1):927–946. doi: 10.1017/S0954579416000638. [DOI] [PubMed] [Google Scholar]
  3. American Psychological Association [APA] Stress in America: Generation Z. Stress in America™ Survey. 2018. https://www.apa.org/news/press/releases/stress/2018/stress-gen-z.pdf
  4. Anderson K., Burford O., Emmerton L. Mobile health apps to facilitate self-care: a qualitative study of user experiences. PLoS One. 2016;11(5) doi: 10.1371/journal.pone.0156164. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Ang W.H.D., Lau S.T., Cheng L.J., Chew H.S.J., Tan J.H., Shorey S., Lau Y. Effectiveness of resilience interventions for higher education students: a meta-analysis and metaregression. J. Educ. Psychol. 2022;114(7):1670–1694. doi: 10.1037/edu0000719. [DOI] [Google Scholar]
  6. Arnett J.J. Emerging adulthood: what is it, and what is it good for? Child Dev. Perspect. 2007;1(2):68–73. doi: 10.1111/j.1750-8606.2007.00016.x. [DOI] [Google Scholar]
  7. Auerbach R.P., Alonso J., Axinn W.G., Cuijpers P., Ebert D.D., Green J.G., Hwang I., Kessler R.C., Liu H., Mortier P., Nock M.K., Pinder-Amaker S., Sampson N.A., Aguilar-Gaxiola S., Al-Hamzawi A., Andrade L.H., Benjet C., Caldas-de-Almeida J.M., Demyttenaere K., Bruffaerts R. Mental disorders among college students in the World Health Organization World Mental Health Surveys. Psychol. Med. 2016;46(14):2955–2970. doi: 10.1017/S0033291716001665. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Auerbach R.P., Mortier P., Bruffaerts R., Alonso J., Benjet C., Cuijpers P., Demyttenaere K., Ebert D.D., Green J.G., Hasking P., Murray E., Nock M.K., Pinder-Amaker S., Sampson N.A., Stein D.J., Vilagut G., Zaslavsky A.M., Kessler R.C. WHO World Mental Health Surveys International College Student Project: prevalence and distribution of mental disorders. J. Abnorm. Psychol. 2018;127(7):623–638. doi: 10.1037/abn0000362. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Barlow D.H., Harris B.A., Eustis E.H., Farchione T.J. The unified protocol for transdiagnostic treatment of emotional disorders. World Psychiatry. 2020;19(2):245–246. doi: 10.1002/wps.20748. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Beck J.S. 2nd ed. The Guildford Press; 2011. Cognitive Behavior Therapy: Basic and Beyond. [Google Scholar]
  11. Becker T.D., Torous J.B. Recent developments in digital mental health interventions for college and university students. Curr. Treat. Options Psychiatry. 2019;6(3):210–220. doi: 10.1007/s40501-019-00178-8. [DOI] [Google Scholar]
  12. Berking M., Wupperman P. Emotion regulation and mental health. Curr. Opin. Psychiatry. 2012;25(2):128–134. doi: 10.1097/YCO.0b013e3283503669. [DOI] [PubMed] [Google Scholar]
  13. Burgess, A. M., Frost, R. O., & DiBartolo, P. M. (2016). Development and validation of the Frost Multidimensional Perfectionism Scale–Brief. J. Psychoeduc. Assess., 34(7), 620–633. doi: 10.1177/0734282916651359. [DOI]
  14. Campbell-Sills L., Stein M.B. Psychometric analysis and refinement of the Connor–Davidson Resilience Scale (CD-RISC): validation of a 10-item measure of resilience. J. Trauma. Stress. 2007;20(6):1019–1028. doi: 10.1002/jts.20271. [DOI] [PubMed] [Google Scholar]
  15. Carlson L.E., Brown K.W. Validation of the Mindful Attention Awareness Scale in a cancer population. J. Psychosom. Res. 2005;58(1):29–33. doi: 10.1016/j.jpsychores.2004.04.366. [DOI] [PubMed] [Google Scholar]
  16. Chen G., Gully S.M., Eden D. General self-efficacy and self-esteem: toward theoretical and empirical distinction between correlated self-evaluations. J. Organ. Behav. 2004;25(3):375–395. doi: 10.1002/job.251. [DOI] [Google Scholar]
  17. Cohen S., Kamarck T., Mermelstein R. A global measure of perceived stress. J. Health Soc. Behav. 1983;24(4):385. doi: 10.2307/2136404. [DOI] [PubMed] [Google Scholar]
  18. Cohen K.A., Stiles-Shields C., Winquist N., Lattie E.G. Traditional and nontraditional mental healthcare services: usage and preferences among adolescents and younger adults. J. Behav. Health Serv. Res. 2021;48(4):537–553. doi: 10.1007/s11414-020-09746-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Conley C.S., Shapiro J.B., Huguenel B.M., Kirsch A.C. Navigating the college years: developmental trajectories and gender differences in psychological functioning, cognitive-affective strategies, and social well-being. Emerg. Adulthood. 2020;8(2):103–117. doi: 10.1177/2167696818791603. [DOI] [Google Scholar]
  20. Creswell J.D. Mindfulness interventions. Annu. Rev. Psychol. 2017;68(1):491–516. doi: 10.1146/annurev-psych-042716-051139. [DOI] [PubMed] [Google Scholar]
  21. Crumbaugh J.C., Hentrion R. International Forum for Logotherapy. Viktor Frankl Inst of Logotherapy; 1988. The PIL Test: administration, interpretation, uses theory and critique. [Google Scholar]
  22. Davies E.B., Morriss R., Glazebrook C. Computer-delivered and web-based interventions to improve depression, anxiety, and psychological well-being of university students: a systematic review and meta-analysis. J. Med. Internet Res. 2014;16(5) doi: 10.2196/jmir.3142. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Dimock M. Defining generations: Where Millennials end and Generation Z begins. 2019. https://www.pewresearch.org/short-reads/2019/01/17/where-millennials-end-and-generation-z-begins/ (January 17)
  24. Eis S., Solà-Morales O., Duarte-Díaz A., Vidal-Alaball J., Perestelo-Pérez L., Robles N., Carrion C. Mobile applications in mood disorders and mental health: systematic search in Apple App Store and Google Play Store and review of the literature. Int. J. Environ. Res. Public Health. 2022;19(4):2186. doi: 10.3390/ijerph19042186. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Fernandez K.C., Jazaieri H., Gross J.J. Emotion regulation: a transdiagnostic perspective on a new RDoC domain. Cogn. Ther. Res. 2016;40(3):426–440. doi: 10.1007/s10608-016-9772-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Ferrari M., Allan S., Arnold C., Eleftheriadis D., Alvarez-Jimenez M., Gumley A., Gleeson J.F. Digital interventions for psychological well-being in university students: systematic review and meta-analysis. J. Med. Internet Res. 2022;24(9) doi: 10.2196/39686. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Finkelstein-Fox L., Park C.L., Riley K.E. Mindfulness and emotion regulation: promoting well-being during the transition to college. Anxiety Stress Coping. 2018;31(6):639–653. doi: 10.1080/10615806.2018.1518635. [DOI] [PubMed] [Google Scholar]
  28. Gatto A.J., Elliott T.J., Briganti J.S., Stamper M.J., Porter N.D., Brown A.M., Harden S.M., Cooper L.D., Dunsmore J.C. Development and feasibility of an online Brief Emotion Regulation Training (BERT) program for emerging adults. Front. Public Health. 2022;10 doi: 10.3389/fpubh.2022.858370. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. van Gemert-Pijnen J.E., Nijland N., van Limburg M., Ossebaard H.C., Kelders S.M., Eysenbach G., Seydel E.R. A holistic framework to improve the uptake and impact of ehealth technologies. J. Med. Internet Res. 2011;13(4) doi: 10.2196/jmir.1672. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Gentili, A., Failla, G., Melnyk, A., Puleo, V., Tanna, G. L. Di, Ricciardi, W., & Cascini, F. (2022). The cost-effectiveness of digital health interventions: a systematic review of the literature. Front. Public Health, 10. doi: 10.3389/fpubh.2022.787135. [DOI] [PMC free article] [PubMed]
  31. Gierveld J.D.J., Van Tilburg T. A 6-item scale for overall, emotional, and social loneliness. Res. Aging. 2006;28(5):582–598. doi: 10.1177/0164027506289723. [DOI] [Google Scholar]
  32. Gillanders D.T., Bolderston H., Bond F.W., Dempster M., Flaxman P.E., Campbell L., Kerr S., Tansey L., Noel P., Ferenbach C., Masley S., Roach L., Lloyd J., May L., Clarke S., Remington B. The development and initial validation of the Cognitive Fusion Questionnaire. Behav. Ther. 2014;45(1):83–101. doi: 10.1016/j.beth.2013.09.001. [DOI] [PubMed] [Google Scholar]
  33. Grant M., Salsman N.L., Berking M. The assessment of successful emotion regulation skills use: development and validation of an English version of the Emotion Regulation Skills Questionnaire. PLoS One. 2018;13(10) doi: 10.1371/journal.pone.0205095. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Gross J. In: Handbook of Emotion Regulation. 2nd ed. Gross J., editor. The Guildford Press; 2014. Emotion regulation: conceptual foundations and empirical foundations; pp. 3–24. [Google Scholar]
  35. Gross J.J., John O.P. Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. J. Pers. Soc. Psychol. 2003;85(2):348–362. doi: 10.1037/0022-3514.85.2.348. [DOI] [PubMed] [Google Scholar]
  36. Harrer M., Adam S.H., Baumeister H., Cuijpers P., Karyotaki E., Auerbach R.P., Kessler R.C., Bruffaerts R., Berking M., Ebert D.D. Internet interventions for mental health in university students: a systematic review and meta-analysis. Int. J. Methods Psychiatr. Res. 2019;28(2) doi: 10.1002/mpr.1759. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Hathaway L., O’Shields B. Understanding and engaging Generation Z. ACSMs Health Fit J. 2022;26(4):39–42. doi: 10.1249/FIT.0000000000000774. [DOI] [Google Scholar]
  38. Hayes S.C., Luoma J.B., Bond F.W., Masuda A., Lillis J. Acceptance and commitment therapy: model, processes and outcomes. Behav. Res. Ther. 2006;44(1):1–25. doi: 10.1016/j.brat.2005.06.006. [DOI] [PubMed] [Google Scholar]
  39. Hinderks A., Schrepp M., Domínguez Mayo F.J., Escalona M.J., Thomaschewski J. Developing a UX KPI based on the user experience questionnaire. Computer Standards & Interfaces. 2019;65:38–44. doi: 10.1016/j.csi.2019.01.007. [DOI] [Google Scholar]
  40. Holtz B.E., Kanthawala S., Martin K., Nelson V., Parrott S. Young adults’ adoption and use of mental health apps: efficient, effective, but no replacement for in-person care. J. Am. Coll. Heal. 2023:1–9. doi: 10.1080/07448481.2023.2227727. [DOI] [PubMed] [Google Scholar]
  41. Huntley C.D., Young B., Tudur Smith C., Fisher P.L. Uncertainty and test anxiety: psychometric properties of the Intolerance of Uncertainty Scale – 12 (IUS-12) among university students. Int. J. Educ. Res. 2020;104 doi: 10.1016/j.ijer.2020.101672. [DOI] [Google Scholar]
  42. Kang M., Ragan B.G., Park J.-H. Issues in outcomes research: an overview of randomization techniques for clinical trials. J. Athl. Train. 2008;43(2):215–221. doi: 10.4085/1062-6050-43.2.215. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Kip H., Keizer J., da Silva M.C., Beerlage-de Jong N., Köhle N., Kelders S.M. Methods for human-centered ehealth development: narrative scoping review. J. Med. Internet Res. 2022;24(1) doi: 10.2196/31858. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Kroenke K., Spitzer R.L., Williams J.B.W. The PHQ-9. J. Gen. Intern. Med. 2001;16(9):606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Lattie E.G., Adkins E.C., Winquist N., Stiles-Shields C., Wafford Q.E., Graham A.K. Digital mental health interventions for depression, anxiety, and enhancement of psychological well-being among college students: systematic review. J. Med. Internet Res. 2019;21(7) doi: 10.2196/12869. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Laure T., Engels C.M.E., Remmerswaal D., Spruijt-Metz D., Konigorski S., Boffo M. Optimization of a transdiagnostic mobile emotion regulation intervention for university students: protocol for a microrandomized trial. JMIR Research Protocols. 2023;12 doi: 10.2196/46603. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Laure T., Remmerswaal D., Engels C.M.E., Konigorski S., Boffo M. 2024. Evaluation of a Mobile Transdiagnostic Emotion Regulation Intervention for University Students: Micro-randomized Trial. (manuscript submitted for publication) [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Liew J., Morris A.S., Kerr K.L. APA Handbook of Adolescent and Young Adult Development. American Psychological Association; 2023. Emotion regulation processes as transdiagnostic in adolescence and early adulthood: a neurobioecological systems framework; pp. 91–106. [DOI] [Google Scholar]
  49. Melcher J., Hays R., Torous J. Digital phenotyping for mental health of college students: a clinical review. Evidence Based Mental Health. 2020;23(4):161–166. doi: 10.1136/ebmental-2020-300180. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Melcher J., Camacho E., Lagan S., Torous J. College student engagement with mental health apps: analysis of barriers to sustained use. J. Am. Coll. Heal. 2022;70(6):1819–1825. doi: 10.1080/07448481.2020.1825225. [DOI] [PubMed] [Google Scholar]
  51. Mouatsou C., Koutra K. Emotion regulation in relation with resilience in emerging adults: the mediating role of self-esteem. Curr. Psychol. 2023;42(1):734–747. doi: 10.1007/s12144-021-01427-x. [DOI] [Google Scholar]
  52. Nakshine V.S., Thute P., Khatib M.N., Sarkar B. Increased screen time as a cause of declining physical, psychological health, and sleep patterns: a literary review. Cureus. 2022 doi: 10.7759/cureus.30051. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Neff K.D., Hsieh Y.-P., Dejitterat K. Self-compassion, achievement goals, and coping with academic failure. Self Identity. 2005;4(3):263–287. doi: 10.1080/13576500444000317. [DOI] [Google Scholar]
  54. Patterson B., Boyle M.H., Kivlenieks M., Van Ameringen M. The use of waitlists as control conditions in anxiety disorders research. J. Psychiatr. Res. 2016;83:112–120. doi: 10.1016/j.jpsychires.2016.08.015. [DOI] [PubMed] [Google Scholar]
  55. Pedrelli P., Nyer M., Yeung A., Zulauf C., Wilens T. College students: mental health problems and treatment considerations. Acad. Psychiatry. 2015;39(5):503–511. doi: 10.1007/s40596-014-0205-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Schaufeli W.B., Martínez I.M., Pinto A.M., Salanova M., Bakker A.B. Burnout and engagement in university students. J. Cross-Cult. Psychol. 2002;33(5):464–481. doi: 10.1177/0022022102033005003. [DOI] [Google Scholar]
  57. Schrepp M., Hinderks A., Thomaschewski J. Construction of a benchmark for the User Experience Questionnaire (UEQ) International Journal of Interactive Multimedia and Artificial Intelligence. 2017;4(4):40. doi: 10.9781/ijimai.2017.445. [DOI] [Google Scholar]
  58. Schulenberg J.E., Sameroff A.J., Cicchetti D. The transition to adulthood as a critical juncture in the course of psychopathology and mental health. Dev. Psychopathol. 2004;16(04) doi: 10.1017/S0954579404040015. [DOI] [PubMed] [Google Scholar]
  59. Seemiller C., Grace M. Generation Z: educating and engaging the next generation of students. About Campus: Enriching the Student Learning Experience. 2017;22(3):21–26. doi: 10.1002/abc.21293. [DOI] [Google Scholar]
  60. Seligman M. In: Handbook of Positive Psychology. Snyder C.R., Lopez S.J., editors. Oxford University Press; 2002. Positive psychology, positive prevention, and positive therapy; pp. 3–12. [Google Scholar]
  61. Spitzer R.L., Kroenke K., Williams J.B.W., Löwe B. A brief measure for assessing generalized anxiety disorder. Arch. Intern. Med. 2006;166(10):1092. doi: 10.1001/archinte.166.10.1092. [DOI] [PubMed] [Google Scholar]
  62. Stengård E., Appelqvist-Schmidlechner K. 2010. Mental Health Promotion in Young People–An Investment for the Future. [Google Scholar]
  63. Stewart-Brown S., Tennant A., Tennant R., Platt S., Parkinson J., Weich S. Internal construct validity of the Warwick-Edinburgh Mental Well-being Scale (WEMWBS): a Rasch analysis using data from the Scottish Health Education Population Survey. Health Qual. Life Outcomes. 2009;7(1):15. doi: 10.1186/1477-7525-7-15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Taber-Thomas B., Pérez-Edgar K. In: Emerging Adulthood Brain Development. Arnett J.J., editor. vol. 1. Oxford University Press; 2014. [DOI] [Google Scholar]
  65. Theurel A., Witt A. Identifying barriers to mental health help-seeking in French university students during the Covid-19 pandemic. Creat. Educ. 2022;13(02):437–449. doi: 10.4236/ce.2022.132025. [DOI] [Google Scholar]
  66. Torous J., Lipschitz J., Ng M., Firth J. Dropout rates in clinical trials of smartphone apps for depressive symptoms: a systematic review and meta-analysis. J. Affect. Disord. 2020;263:413–419. doi: 10.1016/j.jad.2019.11.167. [DOI] [PubMed] [Google Scholar]
  67. Villegas Mejia C., Laure T., Remmerswaal D., Dulloo D., Boffo M. 2024. The ROOM App: Transdiagnostic Mobile Intervention for Emotion Regulation. Conceptualization, Design and Evaluation. (osf.io/e9utb, January) [Google Scholar]
  68. Villegas Mejía C., Remmerswaal D., Engels C.M.E., Ludden G.D.S., Boffo M. Macro-engagement in mHealth: exploring user engagement beyond the screen. Digital Health. 2024;10 doi: 10.1177/20552076231225591. [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. Wu G., Feder A., Cohen H., Kim J.J., Calderon S., Charney D.S., Mathé A.A. Understanding resilience. Front. Behav. Neurosci. 2013;7 doi: 10.3389/fnbeh.2013.00010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Yoon S., Rottenberg J. Why do people with depression use faulty emotion regulation strategies? Emot. Rev. 2020;12(2):118–128. doi: 10.1177/1754073919890670. [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary material 1

Overview of the exercises classified into six intervention categories.

mmc1.docx (27.4KB, docx)
Supplementary material 2

Recommendation system.

mmc2.docx (17.2KB, docx)
Supplementary material 3

Self-developed questionnaire for evaluation of distinct intervention features a.

mmc3.docx (17.8KB, docx)

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES