Skip to main content
NCBI home page
Internet Interventions logoLink to Internet Interventions
. 2024 Jan 24;35:100710. doi: 10.1016/j.invent.2024.100710

What to do when the unwanted happens? Negative event management in studies on internet- and mobile-based interventions for youths and adults with two case reports

Christina Schulte a,1, Theresa Sextl-Plötz a,1, Harald Baumeister b, Ingrid Titzler c, Lasse B Sander d, Cedric Sachser e, Lena Steubl b, Anna-Carlotta Zarski a,c,f,⁎
PMCID: PMC10818076  PMID: 38283258

Abstract

Background

Despite severely burdened individuals, often being excluded from research studies on internet- and mobile-based interventions (IMIs), negative events (NEs) including suicidal thoughts and behaviors (STBs) can still occur during a trial. NEs require monitoring and adequate safety measures. However, study protocols frequently lack comprehensive descriptions of procedures for managing NEs.

Aims

This study aimed to illustrate the assessment, monitoring, and procedures for addressing NEs in two studies on IMIs in adults and youth using case reports, to identify strengths and weaknesses of the NE management approaches, and to derive key learnings and recommendations.

Methods

Two case reports were drawn from two distinct IMI studies. The first study, PSYCHOnlineTHERAPY, evaluates the combination of an IMI with on-site psychotherapy for anxiety and depressive disorders in adults (adult blended study). The second study evaluates a standalone, therapist-guided IMI for post-traumatic stress disorder (PTSD) in youth (youth standalone study). Potential NEs were predefined depending on the study sample. The case studies thoroughly document the systematic recording and ongoing monitoring of NEs through self-report and observer-based assessments during the interventions. The cases illustrate a variety of NE management strategies, including automated and personalized approaches, adapted to the specific nature and severity of the NEs. The NE management approaches are visualized using decision trees.

Results

In the adult blended case study, online questionnaires detected STBs and triggered automated support information. As on-site therapy had already ended, a telephone consultation session allowed for the identification and discussion of the heightened intensity of suicidal thoughts, along with the development of specific additional help options. In the youth standalone case study, heightened tension in an adolescent with PTSD during trauma processing could be addressed in a telephone therapeutic session focusing on resource activation and emotion regulation. The referral to on-site treatment was supported. Overall, advantages of the NE management included automated procedures, multimodal assessment of a wide range of NEs, and standardized procedures tailored to different severity levels. Weaknesses included the use of single-item assessments for STBs and lack of procedures in case of deterioration or nonresponse to treatment.

Conclusion

This study provides practical insights and derives key learnings and recommendations regarding the management of NEs in different IMI contexts for both adults and youth.

Keywords: eMental Health, Internet intervention, Negative events, Suicidal thoughts and behaviors, Case report

Highlights

  • •

    Negative events in IMI studies require adequate management.

  • •

    Practical insights into negative event management in two IMI studies

  • •

    Case reports from two different IMI contexts with youths and adults

  • •

    Recommendations for standard operating procedures in IMI studies

  • •

    Discussion of weaknesses and important future research topics

1. Introduction

A common concern in studies on internet- and mobile-based interventions (IMIs) is how to adequately address participants in crisis remotely. Negative events (NEs) experienced adversely by participants may manifest as adverse events (AEs) marked by a more severe deterioration of target symptoms than expected during treatment. NEs can also manifest as serious adverse events (SAEs) demanding intensive intervention such as hospitalization due to worsened symptoms or suicidal behaviors (Rozental et al., 2014). NEs in research studies may arise from diverse factors, whether within appropriately administered, evidence-based treatment groups or independently of the intervention due to external causes (Herzog et al., 2019), and across various types of control groups (Goldberg et al., 2023; Rozental et al., 2017). There is substantial variation in how NEs are defined, measured, and reported in studies, affecting reported NE frequency (Herzog et al., 2019; Ladwig et al., 2014; Linden and Schermuly-Haupt, 2014). For adult face-to-face psychotherapy, reported NE occurrences vary widely from 0 % to 93.8 % (Gerke et al., 2020; Hoppen et al., 2022; Ladwig et al., 2014; McQuaid et al., 2021; Moritz et al., 2019; Schermuly-Haupt et al., 2018; Strauss et al., 2021). While the rates at which adult participants experience at least one NE attributed to an IMI range from 8.6 % to 51.9 % (Boettcher et al., 2014; Braun et al., 2021, Braun et al., 2022; Fenski et al., 2021; Oehler et al., 2021; Rozental et al., 2015), the rates of NEs unrelated to an IMI seem to be higher (Braun et al., 2021; Schlicker et al., 2020; Zarski et al., 2021). In youth, one study on NEs in face-to-face psychotherapy was identified, which indicated that 100 % reported at least one NE, of which 66.7 % were linked to treatment (Watson et al., 2023).

To effectively assess and manage NEs, it is strongly recommended to establish standard operating procedures (SOPs) with a particular emphasis on addressing suicidal thoughts and behaviors (STBs; Duggan et al., 2014; Horigian et al., 2010; Klatte et al., 2022; Rozental et al., 2014). Specific recommendations for studies in youth encompass the provision of age-appropriate information on potential NEs to children and their caregivers, along with systematic measurement of NEs via designated instruments (Bieda et al., 2018). In IMI studies on participants with STBs, it is for example important to include instructions for developing a crisis plan and encourage access to other treatments if needed (Sander et al., 2020; Seiferth et al., 2023). However, it is rarely reported how exactly these recommendations are operationalized and applied. A systematic review on the assessment and management of harmful effects in psychotherapy studies found that only 34 % (k = 39/115) of included study protocols had specified a systematic harm assessment and only 41 % (k = 47/115) reported standardized procedures for handling NEs (Klatte et al., 2022). Merely 10 % (k = 12/115) of the study protocols reported planned documentation of harmful events, and only 27 % (k = 31/115) of studies described active procedures, such as additional interventions or referral to adequate care (Klatte et al., 2022). A review on suicide risk management in IMI-based studies for depression found that less than two thirds of research teams (n = 8/13) applied step-by-step guidelines and only half of them (n = 7/13) collected contact details of participants or their general practitioners (Sander et al., 2020).

When dealing with NEs, research on IMIs offers the advantage of direct assessment and monitoring through diary entries or eCoach interaction, with the potential for immediate and automated procedures in response to identified NEs. However, implementing NE management in studies involving IMIs poses several challenges, which can be considered as contributing factors to the lack of attention to NE management. Participants may have specifically chosen an IMI study due to its low-threshold accessibility or the absence of human interaction. Consequently, they may not seek or desire additional on-site treatment or further supporting contacts from the study team, such as via phone (Werntz et al., 2022). There are limited options for referring participants to additional on-site treatment, as participants in IMI studies might live in remote areas without available on-site care facilities (Nielsen et al., 2017; van den Berg et al., 2021). Further, participants may experience mobility restrictions in seeking additional face-to-face treatment. Moreover, technical issues or participants' concerns about data privacy violations (Freund et al., 2022) can lead to situations where NEs are not detected or where important information is missed or inaccurately recorded in remote assessments. Another challenge is compiling all information assessed for each participant, as different study team members may be in contact with the same participant at different time points during the study, potentially collecting discrepant information.

This study aims to illustrate the assessment, monitoring, and procedures for addressing NEs in IMI studies. Two illustrative case examples from two different target groups are provided to elucidate the step-by-step procedures and discuss the strengths and weaknesses of the chosen approach for managing NEs. The first case study describes NE management procedures in a blended psychotherapy study, which integrates an IMI into routine face-to-face psychotherapeutic care for adults with diagnosed anxiety and depressive disorders (Baumeister et al., 2021). The second case study reports NE management procedures in a standalone, therapist-guided IMI study for post-traumatic stress disorder (PTSD) in youth (Schulte et al., 2022). Conclusively, the key learnings from our practical experience in both studies on NE management are summarized and aligned with existing evidence-based recommendations from the research context on IMIs to furnish a systematic and practical guide for future research.

2. Methods

2.1. Study design

The case reports were derived from two research studies (Baumeister et al., 2021; Schulte et al., 2022) on IMIs in different intervention contexts following the CARE guidelines (Gagnier et al., 2014). All participants who experienced at least one NE during these studies were identified. Based on practical considerations to be able to illustrate the management procedures for different NEs in a comprehensive manner, an exemplary case was selected in each study. The rationale for the selection of the two cases was to provide an exemplary and detailed illustration of the practical application of our NE management strategies in two different cases with different NEs. Therefore, the case selection was discussed and carefully weighed up by the involved study team members.

2.1.1. Study on a blended care intervention in adults (adult blended study)

The first case report originated from the three-armed, multicenter, cluster-randomized controlled PSYCHOnlineTHERAPY study (Baumeister et al., 2021). The study investigates two versions of blended psychotherapy compared to standard cognitive behavioral therapy (CBT). Within the blended intervention groups, CBT-based online sessions were combined with on-site CBT sessions either using a flexible number and order or a predefined alternating sequence, with a maximum of 16 sessions. Participants were at least 18 years old and had a depressive and/or anxiety disorder diagnosis assigned according to the International Classification of Diseases (ICD-10, World Health Organization, 2019) by their licensed on-site psychotherapist, who also delivered the treatment according to the study group after randomization. A sample of 498 participants was recruited between January 2021 and April 2023. Details of the study and the intervention can be found in the study protocol (Baumeister et al., 2021).

2.1.2. Study on a standalone IMI in youth (youth standalone study)

The second case report was derived from a one-arm, non-randomized feasibility study evaluating a therapist-guided standalone IMI for youth with PTSD (Schulte et al., 2022). The participants were between 15 and 21 years old and displayed clinically significant PTSD symptoms (Child and Adolescent Trauma Screen; CATS ≥21; Sachser et al., 2022), but no acute suicidality (Self-Injurious Thoughts and Behaviors Interview; SITBI; Nock et al., 2007) at baseline. All participants received access to the standalone IMI comprising nine online sessions. Guidance was provided by an eCoach, a master's-level psychologist in psychotherapy training, who gave feedback on completed sessions. A sample of 32 participants was recruited between July 2021 and January 2023. Details of the study and the intervention can be found in the study protocol (Schulte et al., 2022).

2.2. Definition of NEs

In both studies, NEs were defined following guidelines from Rozental et al., 2014, Horigian et al. (2010), and Duggan et al. (2014). NEs were classified as either AEs or SAEs. AEs were defined as physical or psychological symptoms more severe than would usually be expected during treatment, including non-acute suicidality (i.e., suicidal ideation without concrete plans). SAEs encompassed specific events: 1) emergency hospitalization because of a mental disorder, 2) termination of an important and close relationship, 3) intoxication with psychotropic substances requiring medical attention, 4) intentional self-harm requiring medical attention, 5) suicide or attempted suicide, and 6) acute psychosis. Additional SAEs in the youth standalone study included 7) aggression towards persons or objects and 8) insecure living conditions (e.g., threat at home). With the involvement of licensed psychotherapists and eHealth experts, and following recommendations on NE management (Bieda et al., 2018; Duggan et al., 2014; Horigian et al., 2010; Sander et al., 2020), a structured step-by-step manual (SOPs) for the assessment, monitoring, and procedures to deal with NEs was developed and approved by an ethics committee (adult blended study: Ethics Committee of the German Psychological Society, BaumeisterHarald2020–07-29VADM; youth standalone study: Ethics Committee of the German Psychological Society, EbertDavidDaniel2020-09-16-VA). Study team members including licensed psychotherapists received training on the respective SOPs for NE management according to the manuals.

3. Assessment, monitoring, and procedures for NEs

Both studies included self-reported assessments via online questionnaires using an internet-based survey tool (Limesurvey, 2020) and observer-based assessments via online reporting forms and telephone diagnostic interviews conducted by clinicians.

Adult blended study: Self-reported assessments of NEs took place at baseline (T0), 6 weeks (T1), 12 weeks (T2), 18 weeks (T3), 24 weeks (T4), and 12 months (T5) after study inclusion. Telephone interviews were conducted at baseline (T0) and 18 weeks (T3) after study inclusion by psychotherapists in training (i.e., clinical interviewers).

Youth standalone study: Self-reported assessments of NEs took place at screening (T0), baseline (T1), as well as 3 weeks (T2), 8 weeks (T3), and 3 months (T5) after receiving access to the intervention. Telephone interviews were conducted for study inclusion (T0) and at T3 by clinical interviewers who were licensed child- and adolescent psychotherapists. A qualitative interview took place 12 weeks (T4) after access to the intervention by a psychotherapist in training.

3.1. Participant baseline information

3.1.1. Informed consent and contact information

The informed consent comprised information about the study, including potential risks of participation and side effects. Contact information of participants (e.g., telephone number, e-mail-address) were collected within the consent form required for study participation. The participants were informed that the study team could be contacted on working days within working hours.

Adult blended study: All study participants provided informed consent for the whole study, including case reports, prior to intervention start. They agreed that information on diagnoses and occurrence of NEs, including STBs, would be shared with their on-site therapist.

Youth standalone study: In addition to the informed consent for all study participants, the participant and the caregivers consented again to the case reports. Participants were required to name an emergency contact who had to 1) be at least 18 years old, 2) live near to the participant's home, 3) have a close relationship with the participant, and 4) agree to get in touch with the participant if a NE was identified and the study team could not reach the participant via e-mail or phone.

3.1.2. Crisis plan and suicide prevention contract

In the youth standalone study, an individual crisis plan was developed during the baseline telephone interview. Therefore, the clinical interviewer explored the current emotion-regulation and self-help strategies of the participant and the ability to make use of these strategies in critical situations. Further, the clinical interviewer examined the participant's readiness to reach out to the indicated emergency contact and make use of general help offers, (e.g., emergency services and crisis phone services) during the study if needed. For study inclusion, participants provided consent to the crisis plan and a suicide prevention contract for the time of study participation. If the clinical interviewer rated the overall currently available support options as insufficient, additional support that best suited the participant was discussed. After the interview, the crisis plan, the suicide prevention contract, and the individualized information on on-site help offers were sent in a pdf file via e-mail to the participant.

3.1.3. Sociodemographic data

For both studies, information on age, gender, previous treatment, and medication are reported while omitting specific identifiers of the individuals mentioned in the case reports.

3.2. Diagnoses and symptom severity

Adult blended study: The Patient Health Questionnaire Anxiety and Depression Scale (PHQ-ADS; Kroenke et al., 2016) was used to measure depression and anxiety symptom severity. Internal consistency of the PHQ-ADS is reported to be between α = 0.88 to α = 0.92 (Kroenke et al., 2016). Lifetime suicidal and self-injurious thoughts and behaviors were assessed with a modified version of the Columbia Suicide Severity Rating Scale (C-SSRS), with internal consistency reported to range between α = 0.73 and α = 0.95 (Posner et al., 2011).

Youth standalone study: The Clinician-Administered PTSD Scale for the Diagnostic and Statistical Manual of Mental Disorders, (DSM-5, American Psychiatric Association, 2013) in its Child/Adolescent Version (CAPS-CA 5, Pynoos et al., 2015) was used to assess the PTSD diagnosis. Anxiety and depression symptom severity was measured with the PHQ-ADS (Kroenke et al., 2016) with a good internal consistency in the study sample (α = 0.88). A stress scale developed according to Hendricks et al. (2014), ranging from 0 to 10, assessed the felt tension level during each intervention session.

3.3. Suicidal thoughts and behaviors (STBs)

3.3.1. Self-reported assessment

The PHQ-ADS item 9 was used to assess the frequency of “Thoughts that you would be better off dead, or of hurting yourself” within the last two weeks (0=”not at all,” 1=”several days,” 2=”more than half the days,” and 3=”nearly every day;” Kroenke et al., 2016). Existing evidence indicates an association between scores on this item and the likelihood of suicide (Louzon et al., 2016; Simon et al., 2013). The severity of suicidal thoughts and wishes during the past two weeks was assessed with the Beck Depression Inventory (BDI-II) item 9, (0=”I don't have any thoughts of killing myself,” 1=”I have thoughts of killing myself, but I would not carry them out,” 2=”I would like to kill myself,” and 3=”I would kill myself if I had the chance;” Hautzinger et al., 2006). This item is recommended to use as a brief screen for suicide risk (Green et al., 2015). An algorithm based on predefined suicidality cut-offs was programmed in the online survey tool as follows.

Adult blended study: The algorithm combined the PHQ-ADS item 9 and BDI-II item 9 responses to distinguish three severity levels of STBs at all self-reported assessments time points (level 0: PHQ-ADS item 9 = 0 and BDI-II item 9 = 0; level 1: PHQ-ADS item 9 = 0–2 and BDI-II item 9 < 2; level 2: PHQ-ADS item 9 > 0 and BDI-II item 9 > 1 or PHQ-ADS item 9 = 3 and BDI-II item 9 ≥ 0).

Youth standalone study: The BDI-II item 9 was used in an algorithm to assess the severity level of STBs in self-reports at all time points (level 0: BDI-II item 9 = 0; level 1: BDI-II item 9 = 1; level 2: BDI-II item 9 ≥ 2). In instances of a positive screen on item 15 of the Negative Effects Questionnaire at T3 (NEQ, “I got thoughts that it would be better if I did not exist anymore and that I should take my own life”, Rozental et al., 2014), the BDI-II item 9 was additionally assessed to determine the severity level of STBs.

3.3.2. Observer-based assessment

Adult blended study: The Structured Clinical Interview (SCID-5; Beesdo-Baum et al., 2019) and the Quick Inventory of Depressive Symptomatology, clinician-rated version (QIDS-C; 16 items, Rush et al., 2003) were used to assess STBs within telephone interviews at T0 and T3. The internal consistency of the QIDS-C is reported to range between α = 0.81 and 0.95 (Trivedi et al., 2004). One question at T3 additionally assessed suicide attempt(s) since the beginning of the study intervention.

Youth standalone study: Selected questions from the SITBI were used to assess the occurrence, frequency, and characteristics of suicidal thoughts, plans, and non-suicidal self-injurious behavior in the telephone interview at T0 (9 of 169 items, Nock et al., 2007). Participants were informed before the start of the interview that they would be asked about STBs.

3.4. Other NEs

3.4.1. Observer-based assessment

In both studies, the occurrence of other NEs apart from STBs was assessed during the telephone interviews at T3 using the list of the predefined SAEs. Standardized online reporting forms were used to document the type, time of occurrence, duration, and severity, and to rate the association with treatment for each NE. All individuals involved in the studies including on-site therapists and eCoaches were instructed to use the online reporting form for documenting any NEs following contact with participants.

3.4.2. Self-reported assessment

Adult blended study: The NEQ (Rozental et al., 2014) was used as a self-report measure for other NEs at T3 but was not used in the monitoring of and procedures for NEs. Internal consistency of the NEQ is reported to be α = 0.95 (Rozental et al., 2016).

Youth standalone study: Participants could report additional predefined SAEs within an online questionnaire at T5. Further, they could report any NE via e-mail to the study team or via the online platform to the eCoach.

3.5. Procedures for NEs

In the instance of a NE, the predefined step-by-step procedures were applied depending on 1) assessment modality and context (i.e., interview, questionnaire, treatment sessions) and 2) the type and severity of the NE. For the adult blended study, Fig. 1 displays the flow chart for the SOPs regarding STBs and Fig. 2 regarding other NEs. Fig. A.1 and Fig. A.2 in Appendix A show the respective flow charts for the youth standalone study, and Appendix B provides a detailed description of the SOPs for dealing with NEs.

Fig. 1.

Fig. 1

Open in a new tab

Standard operating procedures (SOPs) for dealing with STBs in the adult blended study.

Note: STBs = suicidal thoughts and behaviors, P = participant, DSMB = Data Safety Monitoring Board. Level 0 is not followed by any procedures.

Fig. 2.

Fig. 2

Open in a new tab

Standard operating procedures (SOPs) for dealing with other NEs in the adult blended study.

Note: P = participant, SAE = serious adverse event, AE = adverse event, DSMB = Data Safety Monitoring Board. Level 0 is not followed by any procedures.

Adult blended study: The safety management procedures depended on the timing of the occurrence of NEs. If the study intervention delivered by the on-site therapist was ongoing, the on-site therapist was promptly informed upon the occurrence of a NE and was responsible for the adequate response to the NE. If the intervention had been completed according to the on-site therapist, the study team took responsibility for the safety management procedures.

Youth standalone study: The study team was responsible for the adequate response to any NE that occurred during the study and the intervention.

3.6. Documentation and reporting of NEs

Each study established an independent Data Safety Monitoring Board (DSMB) including experts in the fields of clinical psychology not otherwise involved in the studies. The DSMB received regular reports on all NEs and could give recommendations for additional safety management procedures or modifications. The DSMB's response could also encompass recommendations to modify or discontinue the respective study itself. The reports on NEs and recommendations from the DSMB were discussed in annual meetings.

Adult blended study: The DSMB was informed immediately in the case of an SAE and could offer recommendations whether further information was required, how to handle the SAE, and its follow-up management.

4. Case reports

4.1. Case report 1 (adult blended study)

4.1.1. Participant information

Case report 1 presents the case of a female participant in her thirties living in a small town in Germany. Her on-site therapist diagnosed a moderate depressive episode (ICD-10: F32.1). During the initial assessment (T0) of the study, the participant reported experiencing severe depressive and anxiety symptoms, as indicated in the self-report (PHQ-ADS score: 34/48). She also mentioned that she was undergoing mental health treatment for the first time and was not currently taking any medication. However, she did disclose a history of suicidal thoughts and a past suicide attempt during her youth but did not indicate any current intentions to act on such thoughts (C-SSRS). She participated in the flexible study condition with six on-site treatment sessions and ten online sessions assigned by her on-site therapist. The participant completed all but the final (T5) online and interview assessments assigned. At T3, she reported a concomitant antidepressant medication initiated during the study intervention.

4.1.2. Assessment and monitoring of NEs

The participant reported STBs in online questionnaires at four time points (T0, T1, T2, and T3). At all four events, she displayed a score of two on the PHQ-ADS item 9, expressing thoughts of being better off dead or self-harm on more than half the days over the last two weeks, and a score of two on the BDI-II item 9, indicating that she would like to kill herself, classified as STBs severity level 2. No STBs were reported in the interviews (T0, T3), and no other NEs were registered during her study participation.

4.1.3. Procedures for STBs

Via automated procedures, the participant received information on general help offers at all four events. At the first three events (T0, T1, and T2), the therapy was ongoing, and the on-site therapist was immediately informed by a research assistant, responsible for the administrative study management, via e-mail that the participant had reported STBs within an online questionnaire. At the fourth event (T3), the study therapy was already terminated. The study team informed the participant via e-mail that the on-call therapist, a licensed psychotherapist with on-call duty in case of severity level 2 STBs, would call her for a mandatory therapeutic consultation within the next 24 h. The participant responded to this e-mail two and a half hours later and confirmed her telephone number and suitable time periods for the telephone consultation. The on-call therapist called the participant on the same day and conducted the predefined and trained semi-standardized emergency plan (see Appendix B for further details). During the therapeutic consultation, the participant stated credibly that she did not have any plans of suicide and had not tried to attempt suicide. However, she expressed concerns that her suicidal thoughts had become more concrete and was worried about their continued worsening. After discussing possible help options, the participant said that she was considering inpatient treatment and mentioned having an appointment with her psychiatrist on the same day. She assured that she would inform her psychiatrist about her suicidal thoughts. It was agreed upon that her partner who supported her treatment plans would accompany her to the appointment. At the end of the therapeutic consultation, the participant was credibly distanced from acute suicidality and expressed a commitment to seek additional support services. Additionally, she acknowledged the importance of reaching out to emergency services in the event of an acute crisis. The on-call therapist assessed that the level of suicidality was not acute and determined that no further interventions or actions were required from the study personnel. All events and following procedures were documented by the on-call therapist and the research assistants.

4.2. Case report 2 (youth standalone study)

4.2.1. Participant information

Case report 2 is on an adolescent student who lives with his parents in a major city in Germany. At the time of study inclusion, the online screening indicated increased PTSD symptomatology (CATS: 31/60), and, in the subsequent telephone interview with the CAPS-CA-5, the clinical interviewer diagnosed PTSD with high severity due to multiple traumatic events (e.g., sexual violence and physical assaults). Within this interview, the participant also reported having difficulties interacting with other people since childhood. The clinical interviewer noted a high level of disease burden and emotion regulation difficulties in combination with a lack of coping skills, self-help strategies, and personal resources. The clinical interviewer developed a detailed crisis plan with the participant, including self-help and emotion-regulation strategies (e.g., engaging in sports, photography, taking a cold shower), contact information of online (e.g., a 24-h crisis hotline, crisis chat website) and offline counseling and treatment facilities (e.g., emergency numbers) and additional help options (e.g., on-site crisis counseling centers, psychological counseling, contact information for on-site psychotherapists) in case of worsening of symptoms during the intervention. The participant consented to the suicide prevention contract and received all documents discussed (crisis plan, suicide prevention contract, individualized help options) in a PDF-sheet via e-mail after the interview. The participant completed four out of five online questionnaires, two out of three interviews, and all nine online sessions within seven weeks.

4.2.2. Assessment and monitoring of NEs

In the interview for study inclusion, the participant disclosed a history of previous STBs. However, he credibly denied experiencing any current suicidal ideation. The participant also rated the likelihood of having suicidal thoughts in the future as “little” (SITBI: 2/4), and the likelihood of making a suicide plan as “very little” (SITBI: 0/4). The clinical interviewer concluded that suicidal and non-suicidal self-injurious behaviors did not pose a risk for study participation (severity level 1). The participant reported two NEs within the last sixth months prior to study participation (i.e., end of relationship with his girlfriend and aggression towards objects such as destroying his mobile phone amongst other items). While completing the intervention, the participant hardly made any written statements in the predefined text boxes, although the eCoach repeatedly encouraged him to report on his experiences with the exercises and his condition. The participant, however, did not respond to the eCoach's follow-up questions. After the fourth session on the trauma narrative, the participant reported a heightened stress level (8/10) due to his “lifelong suffering and difficulties with the intervention because he did not have the strength to formulate words for his feelings and thoughts.” The eCoach assessed the symptomatology to be elevated beyond expectations and subsequently communicated the occurrence of an AE to the study team using a standardized reporting form.

4.2.3. Procedures for NEs

The eCoach contacted the participant using the messaging function on the online platform and recommended focusing on the emotion regulation strategies learned before delving into the next session. Following the NE procedures, the eCoach also provided the participant with the option of a therapeutic consultation with a licensed child and adolescent psychotherapist. Despite making three attempts to reach the participant through email and the online platform within the specified timeframes, there was no response. Consequently, the child and adolescent psychotherapist reached out to the participant by phone. During the subsequent therapeutic consultation session, the participant expressed feeling “somewhat better” already. The psychotherapist assessed the participant's current stress level, and together, they explored factors causing his stress, which also included, apart from the trauma exposure, external factors such as schoolwork and rejection following a job application. The psychotherapist and participant subsequently worked on activating self-help strategies for emotion and stress regulation already discussed in the interview for study inclusion. The participant further reported that he had a first session scheduled at a counseling center and intend to continue with regular appointments. The psychotherapist encouraged the participant in making use of these help options and invited the participant to get in touch with the study team if he would need further support in finding additional treatment options. Afterwards, the participant completed the next two intervention sessions dealing with the trauma narrative on the online platform. After the sixth session, he indicated a heightened level of stress again (stress scale: 8/10) without leaving any further information in the text boxes of the session. The eCoach offered another therapeutic consultation via the online-platform, and after 24 h with no response, the study team sent an e-mail regarding the offer and with request for feedback. This time, the participant responded to the e-mail, and thanked for the offer, but denied further need for support. Afterward, he completed the remaining three online sessions (7–9), and no further NEs were identified. The participant completed the follow-up telephone interview at T3, where he stated having scheduled a face-to-face session with a licensed psychotherapist in an outpatient care setting. The clinical interviewer encouraged the participant in his efforts to seek care and offered support in finding additional help options if desired. During the qualitative interview at T4, the participant mentioned that the intervention did not prove helpful because he realized he would have preferred “interacting with a real person.” Additionally, he reported problems with “[making] sense of the free text fields” resulting in him “clicking around.” Despite these challenges, the participant acknowledged that the support provided in accessing on-site therapy was beneficial, and the process of initiating on-site therapy was successful. He expressed a positive perception of the appointments with the on-site psychotherapists and reported feeling optimistic about the upcoming on-site appointments.

5. Key learnings and recommendations

We derived key learnings and recommendations for the assessment, monitoring, and procedures for dealing with NEs (Table 1). The key learnings were derived from our overall experience with the development and implementation of NE management strategies in both studies as illustrated by the case reports, and the expertise of the involved researchers and practitioners. They were supplemented with existing best practice recommendations (Sander et al., 2020; Seiferth et al., 2023; Teismann et al., 2021) regarding the assessment, monitoring, and procedures for dealing with NEs.

Table 1.

Key learnings and recommendations for the management of negative events in studies on internet- and mobile-based interventions.

  • 1.

    Assessment and Monitoring

  • •

    Potential negative events (NEs) should be predefined with consideration of the specific study sample, targeted mental disorders, treatment goals, and setting. These predefined terms should also incorporate the latest findings from research on the negative effects of psychotherapy (e.g., Rozental et al., 2014). Furthermore, there should be a possibility for all study team members to record additional unforeseen NEs that may occur during a study but were not initially specified (e.g., via online reporting forms that can be used anytime and not only at predefined measurement time points).

  • •

    Assessment of NEs, especially suicidal thoughts and behaviors (STBs), should be implemented at different time points during the study and preferably be assessed and monitored using different assessment modalities (e.g., self- and observer-based rating) and specific instruments instead of single items. Participants should be informed about STB assessment in interviews and questionnaires (as recommended by Teismann et al., 2021).

  • •

    In the context of questionnaires evaluating broader constructs such as depression (e.g., Patient Health Questionnaire Anxiety and Depression Scale, PHQ-ADS; Beck Depression Inventory, BDI-II), or assessing potential adverse events (e.g., Negative Effects Questionnaire, NEQ) that include items related to STBs, it is essential to consider their relevance in the monitoring of STBs. Responses to these items should be monitored, and if necessary, appropriate help offers should be offered directly, followed by predefined procedures.

  • •

    Participants' contact information must be collected, and if feasible, an emergency contact should be provided during the time of study participation. The emergency contact should consent to be called in incidents of self-harm or harm to others, if the participant itself is not reached after a predefined number of contact attempts.

  • •

    We recommend making use of standardized reporting forms for all occurring NEs, which are accessible online and must be used by all study team members.

  • 2.

    Procedures

  • •

    Procedures should follow a standardized and predefined step-by-step manual incorporating specific instructions on when (e.g., with predefined time criteria) and how (e.g., with standardized text templates for e-mails) to act (standard operating procedures, SOPs). Specific adaptations may be necessary, depending on the nature of the intervention (e.g., standalone or blended) and the characteristics of the study sample (e.g., consent from caregivers for minority participants).

  • •

    SOPs on NE management should account for a range of scenarios involving NEs of differing types and severities that may arise at various stages of the study. It is essential to establish clear definitions of the responsibilities of each person involved in the NE management procedures, along with their availability and contact information. This clarity also facilitates rapid decision-making in unique situations not explicitly covered in the SOPs.

  • •

    We recommend implementing automated (e.g., cut-off-based) procedures to facilitate prompt responses when NEs occur. This may include the automated provision of information on help offers to participants with elevated scores on certain predefined items and questionnaires and automated notifications sent to the study team.

  • •

    The same SOPs on NE management should be applied for all study conditions (including control groups). Participants should be informed about every step of the NE management (e.g., via e-mail) to ensure transparency in procedures and participants autonomy.

  • •

    At the commencement of the study, participants should be provided with clear information regarding the availability and responsibilities of the study team in responding to NEs. This may include details such as the study team's availability during regular working hours on weekdays and the absence of emergency services. These arrangements should be consistently adhered to by all study team members. In cases where adverse events occur outside of the study team's available periods, predefined procedures should be in place, and participants should be informed accordingly.

  • •

    To promote the participants' ability to self-help in difficult situations, an individual crisis plan should be developed encompassing self-help and emotion-regulation strategies, general help offers, and individual emergency contacts. This plan can be established before study start, especially for vulnerable groups (e.g., youth), or when participants show elevated levels on STB-related items.

  • •

    If participants express additional requirements for treatment or counseling, such as crisis consulting or psychotherapy, they should be offered comprehensive support in finding suitable local care and encouraged to access these offers. This can involve discussing available options together over the phone and arranging for on-site contact if necessary.

  • •

    Involving a Data Safety Monitoring Board (DSMB) in the study may be beneficial to obtain an external independent expert opinion on the management procedure of NEs and an overall evaluation of the safety of the intervention under investigation.

  • 3.

    Studies on blended interventions

  • •

    For studies of blended interventions, on-site therapists should be involved in the development and implementation of the management of NEs. Responsibilities should be clearly defined, and on-site therapists should be trained regarding the identification of NEs and procedures in the respective study context.

  • 4.

    Human contact

  • •

    We recommend allocating sufficient time and personnel capacities to develop and test SOPs prior to the study start and to schedule study-personnel in SOPs for NE management during the study (e.g., responsible for checking e-mail notifications). It should be noted that these study team members cannot be blinded towards the study condition and therefore should not be involved in the outcome assessment.

  • •

    We recommend including licensed on-call therapists and supervisors who can be available for multiple studies simultaneously at predefined time slots in cases of NEs that need case advice. Supervision for study team members could be beneficial for their well-being and for upholding a high quality of NE management.

Open in a new tab

6. Discussion

We used one case study of blended therapy in adults and one case study of a standalone IMI in youth to illustrate the assessment, monitoring, and step-by-step procedures for addressing NEs in IMI research context. The inclusion of different age groups allowed for a comprehensive review of the strengths and weaknesses of the chosen NE management approach. The broad definition of possible AEs ensured a high sensitivity in NE recording considering not only STBs but also other NEs including uncharacteristically severe worsening of symptoms in the course of therapy. Furthermore, the use of multiple assessment modalities (i.e., questionnaires, interviews, reporting forms) appeared to be valuable in comprehensively detecting NEs. This was evident in the adult blended study, as suicidal thoughts were only identified via online questionnaires but not reported by the participant in both telephone interviews, which were conducted at the same time points when two of the four events occurred. Utilizing automated standardized messages enabled the promptly provision of information on help offers, while therapeutic consultations offered further personalized support as required. The comprehensive action plan, encompassing defined cut-offs, time schedules, and participant communication protocols, has proven particularly beneficial, primarily by establishing unequivocal task assignments within the procedural framework. Additionally, detailed documentation of NEs and close communication between involved study team members extending beyond standardized assessment methods were essential for gathering further important information and coordinating additional procedures as needed. In both studies, the NE management approach ensured that individuals were effectively referred for additional therapeutic support outside of the IMI study whenever the need arose. Supporting participants in overcoming personal barriers to accessing additional face-to-face therapeutic support if needed is a critical aspect. As, in some instances, participants opt for an IMI because they are worried about seeking therapy locally, e.g., due to difficulties in finding out offers on-site and fear of potential stigmatization.

Our study pinpointed areas for enhancing SOPs for NE management. As a limitation of our NE management strategies, assessments of deterioration and nonresponse via self-report were measured in both studies but not integrated in the automated NE management procedures. Nevertheless, responding to symptom deterioration and non-response by adapting the treatment may hold significant importance, because both deterioration and a lack of treatment effects do occur, and can potentially discourage participants from seeking further treatment (Ebert et al., 2016; Karyotaki et al., 2018; Lutz et al., 2015; Rozental et al., 2014, Rozental et al., 2017, Rozental et al., 2019). Further, self-assessment of STBs was conducted using single items such as BDI-II item 9. This approach aligns with the common practice observed in IMI studies. However, some single items like PHQ-ADS item 9 have been shown to inadequately assess suicidal thoughts, instead focusing on thoughts of death or self-harm (Sander et al., 2020). It is advisable to incorporate dedicated, comprehensive questionnaires specifically designed for assessing STBs, rather than relying solely on single items related to suicidality, especially when assessing STBs (Holmes et al., 2018; Sander et al., 2020; Teismann et al., 2021). However, it is also important to consider that questionnaire assessments, in general, have limited predictive value for STBs because there is no strong correlation between suicidal thoughts and other risk factors associated with suicide and the occurrence of actual STBs (Franklin et al., 2017). Moreover, depending on the population studied, between 25 % and 60 % of suicide attempts are considered “impulsive,” further limiting the predictive value of participants' momentary responses with respect to future STBs (Simon et al., 2001; Wojnar et al., 2009; Wyder and De Leo, 2007). Hence, solely relying on participants' responses to individual items and self-assessment questionnaires for tailoring NE management may not be effective (Franklin et al., 2017; Steeg et al., 2018) and it is necessary to explore and incorporate additional clarification options. Another disadvantage of our procedure was that additional input in case of STBs for participants despite the display of help offers was provided solely via telephone consultations, which may represent a barrier to utilization for participants. Neither of the online sessions in both studies included content aimed at reducing STBs (Sander et al., 2020; Seiferth et al., 2023; Teismann et al., 2021). Indeed, in the adult blended study the on-site therapists were responsible for implementing interventions to reduce STBs. However, we did not verify whether the on-site therapists implemented such NE procedures, nor if they adhered to current treatment recommendations, e.g., providing sessions twice a week in case of severe depressive symptoms (Bruijniks et al., 2020).

6.1. Limitations of the study

There are limitations to consider when interpreting the findings of this study. The case selection was based on practical considerations to describe the procedure for different NEs, which restricts the generalizability of the results to other cases. Furthermore, the sample was comprised of individuals who proactively chose to take part in a study examining an IMI, which might limit the sample's overall representativeness. Additionally, no systematic quantitative or qualitative analyses of all NE data of all participants in the two IMI studies were used for deriving recommendations. The recommendations rely on our practical experiences with the implementation of SOPs for NE management in the two studies, as well as the clinical and scientific expertise of the involved researchers and clinicians and existing research and guidelines regarding NE management. The case reports are an exemplary illustration of how NE management was implemented in two different IMI studies to address NE in all participants.

6.2. Future research

One future approach in NE management could involve a more personalized strategy, identifying individuals at increased risk of experiencing NEs early on, closely monitoring them, and offering tailored additional support. To achieve this objective, however, the initial step is to enhance the prediction of STBs (Franklin et al., 2017). Potential predictors of NEs, e.g., clinical or sociodemographic variables should be taken into account and further investigated in studies with adequate statistical power like individual patient data meta-analyses (Rozental et al., 2017). Further, the use of sensor technology, smartphone applications, and machine learning might be helpful to get deeper insights into the development and the course of NEs and to investigate new risk factors for the occurrence of NEs (Sels et al., 2021; Smith et al., 2023; Torous et al., 2018). To ensure timely support, automated processes in IMIs warrants further investigation as they represent a low-threshold, highly scalable, and potentially cost-effective means to assess and monitor NEs that can then be linked to stepped-up support in terms of synchronous and asynchronous measures. There is current evidence, that symptoms fluctuate frequently over time and therefore a more dynamic assessment of NEs and adaptive interventions through just-in-time adaptive interventions (JITAIs) shows great potential to better respond to changes like deterioration or relapse. However, JITAIs need further investigation in digital mental health (Smith et al., 2023). Moreover, alternative, data-secure chat programs for study communication through which individuals can potentially be reached more promptly should be further tested as adolescent participants in the youth standalone study for example seemed to be less likely to use e-mail on a regular basis (Schulte et al., submitted; Schulte et al., 2022). Regarding additional support options, integrating social support systems offline or on-site (e.g., emergency contacts, online support buddies, or self-help groups) should be investigated for its additive value. Another possibility to enrich the NE management might also be the active collaboration with existing providers of help offers, such as 24-hour hotlines (van Spijker et al., 2015). For a comprehensive NE management, future research might also investigate potential threats within the social context of the participants (e.g., intimate partner violence, childhood abuse or neglect; Micklitz et al., 2023) along with procedures for dealing with these NEs, as done in the youth standalone study. Furthermore, forthcoming research should assess NE management approaches within the framework of integrating IMIs into routine care settings. Such studies should also explore the balance between preserving the benefits of IMIs, such as enhanced accessibility and flexibility in terms of time and location, while providing essential support to participants at risk of encountering NEs.

6.3. Conclusion

This study illustrated the assessment, monitoring, and step-by-step procedures of dealing with NEs in two studies of IMIs using two case reports set in different intervention contexts, one involving a youth population and the other targeting adults. The procedures described represent an exemplary solution to the question of NE management in studies on IMIs discussing associated advantages and disadvantages. Key learnings and recommendations include, amongst others, the implementation of stepped automated and individualized procedures depending on the assessment modality, the type, and the severity of NEs, according to a predefined action plan.

Statement of ethics

The study protocol for the clinical trial PSYCHOnlineTHERAPY was reviewed and approved by the Ethics Committee of the German Association of Psychology (DGPs no. BaumeisterHarald2020-07-29VADM). The participants provided their written informed consent to participate in this study. The study protocol of the feasibility study was also reviewed and approved by the ethics committee of the German Association of Psychology (DGPs no. EbertDavidDaniel2020-09-16-VA). Written informed consent was obtained from all participants and from both carers in case of minor participants.

Funding sources

PSYCHOnlineTHERAPY was supported by the Innovation Committee (Innovationsausschuss) of the Joint Federal Committee (Gemeinsamer Bundesausschuss, gBA) [no: 01NVF18036]. The feasibility study is part of the BESTFORCAN project and was supported by the Federal Ministry of Education and Research [no: 01KR1804D].

CRediT authorship contribution statement

TS, ChS, HB, IT, CeS, LS, and ACZ were involved in the development of the structured manual for the assessment, monitoring, and procedures to deal with negative events in eHealth studies. TS, CS, and ACZ drafted the manuscript. All authors read, commented, and approved the final manuscript.

Declaration of competing interest

ACZ reports having received fees for lectures or workshops and for expert videos for an internet-based intervention for sexual dysfunction. IT reports having received fees for lectures/workshops in the e-mental-health context from training institutes and congresses for psychotherapists, but outside the submitted work. She was the project lead for the research project ImpleMentAll (funded by the European Commission) at GET.ON which aimed to investigate the effectiveness of tailored implementation strategies compared to implementation as usual for guided IMIs (11/2017–03/2021). HB received consultancy fees, reimbursement of congress attendance, and travel costs as well as payments for lectures from Psychotherapy and Psychiatry Associations as well as Psychotherapy Training Institutes in the context of E-Mental-Health topics. He has been the beneficiary of study support (third-party funding) from several public funding organizations. LBS received personal fees from Psychotherapy and Psychiatry Associations and Psychotherapy Training Institutes, including reimbursement of travel costs and payment for lectures in the context of E-Mental-Health but outside the submitted work. ChS, CeS, TS, and LS have no conflicts of interest to declare.

Acknowledgment

We thank Dr. Thorsten Sukale for his work on the projects. We thank the study team from both studies for their support in the study administration.

Footnotes

Appendix A

Supplementary data to this article can be found online at https://doi.org/10.1016/j.invent.2024.100710.

Appendix A. Supplementary data

Appendix A

Standard operating procedures youth standalone study.

mmc1.docx (332.7KB, docx)
Appendix B

Procedures.

mmc2.docx (26.7KB, docx)

References

  1. American Psychiatric Association . Fifth edition. 2013. Diagnostic and Statistical Manual of Mental Disorders. [DOI] [Google Scholar]
  2. Baumeister H., Bauereiss N., Zarski A.-C., Braun L., Buntrock C., Hoherz C., Idrees A.R., Kraft R., Meyer P., Nguyen T.B.D., Pryss R., Reichert M., Sextl T., Steinhoff M., Stenzel L., Steubl L., Terhorst Y., Titzler I., Ebert D.D. Clinical and cost-effectiveness of PSYCHOnlineTHERAPY: study protocol of a multicenter blended outpatient psychotherapy cluster randomized controlled trial for patients with depressive and anxiety disorders. Front. Psychiatry. 2021;12 doi: 10.3389/fpsyt.2021.660534. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Beesdo-Baum K., Zaudig M., Wittchen H.-U. SCID-5-CV: Strukturiertes Klinisches Interview für DSM-5-Störungen—Klinische Version. Hogrefe. 2019 [Google Scholar]
  4. van den Berg N., Fleßa S., Hoffman W. Gesundheitsversorgung im ländlichen Raum. bpb.de. 2021. https://www.bpb.de/themen/stadt-land/laendliche-raeume/334219/gesundheitsversorgung-im-laendlichen-raum/
  5. Bieda A., Pflug V., Scholten S., Lippert M.W., Ladwig I., Nestoriuc Y., Schneider S. Unerwünschte Nebenwirkungen in der Kinder- und Jugendlichenpsychotherapie – Eine Einführung und Empfehlungen. PPmP - Psychotherapie · Psychosomatik · Medizinische Psychologie. 2018;68(09/10):383–390. doi: 10.1055/s-0044-102291. [DOI] [PubMed] [Google Scholar]
  6. Boettcher J., Rozental A., Andersson G., Carlbring P. Side effects in internet-based interventions for social anxiety disorder. Internet Interv. 2014;1(1):3–11. doi: 10.1016/j.invent.2014.02.002. [DOI] [Google Scholar]
  7. Braun L., Titzler I., Terhorst Y., Freund J., Thielecke J., Ebert D.D., Baumeister H. Effectiveness of guided internet-based interventions in the indicated prevention of depression in green professions (PROD-A): results of a pragmatic randomized controlled trial. J. Affect. Disord. 2021;278:658–671. doi: 10.1016/j.jad.2020.09.066. [DOI] [PubMed] [Google Scholar]
  8. Braun L., Terhorst Y., Titzler I., Freund J., Thielecke J., Ebert D.D., Baumeister H. Lessons learned from an attempted pragmatic randomized controlled trial for improvement of chronic pain-associated disability in green professions: long-term effectiveness of a guided online-based acceptance and commitment therapy (PACT-A) Int. J. Environ. Res. Public Health. 2022;19(21) doi: 10.3390/ijerph192113858. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Bruijniks S.J.E., Lemmens L.H.J.M., Hollon S.D., Peeters F.P.M.L., Cuijpers P., Arntz A., Dingemanse P., Willems L., Oppen P. van, Twisk J.W.R., Boogaard M. van den, Spijker J., Bosmans J., Huibers M.J.H. The effects of once- versus twice-weekly sessions on psychotherapy outcomes in depressed patients. Br. J. Psychiatry. 2020;216(4):222–230. doi: 10.1192/bjp.2019.265. [DOI] [PubMed] [Google Scholar]
  10. Duggan C., Parry G., McMurran M., Davidson K., Dennis J. The recording of adverse events from psychological treatments in clinical trials: evidence from a review of NIHR-funded trials. Trials. 2014;15:335. doi: 10.1186/1745-6215-15-335. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Ebert D.D., Donkin L., Andersson G., Andrews G., Berger T., Carlbring P., Rozenthal A., Choi I., Laferton J.a.C., Johansson R., Kleiboer A., Lange A., Lehr D., Reins J.A., Funk B., Newby J., Perini S., Riper H., Ruwaard J.…Cuijpers P. Does Internet-based guided-self-help for depression cause harm? An individual participant data meta-analysis on deterioration rates and its moderators in randomized controlled trials. Psychol. Med. 2016;46(13):2679–2693. doi: 10.1017/S0033291716001562. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Fenski F., Rozental A., Heinrich M., Knaevelsrud C., Zagorscak P., Boettcher J. Negative effects in internet-based interventions for depression: a qualitative content analysis. Internet Interv. 2021;26 doi: 10.1016/j.invent.2021.100469. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Franklin J.C., Ribeiro J.D., Fox K.R., Bentley K.H., Kleiman E.M., Huang X., Musacchio K.M., Jaroszewski A.C., Chang B.P., Nock M.K. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol. Bull. 2017;143(2):187–232. doi: 10.1037/bul0000084. [DOI] [PubMed] [Google Scholar]
  14. Freund J., Buntrock C., Braun L., Thielecke J., Baumeister H., Berking M., Ebert D.D., Titzler I. Digital prevention of depression for farmers? A qualitative study on participants’ experiences regarding determinants of acceptance and satisfaction with a tailored guided internet intervention program. Internet Interv. 2022;29 doi: 10.1016/j.invent.2022.100566. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Gagnier J.J., Kienle G., Altman D.G., Moher D., Sox H., Riley D. The CARE guidelines: consensus-based clinical case report guideline development. J. Clin. Epidemiol. 2014;67(1):46–51. doi: 10.1016/j.jclinepi.2013.08.003. [DOI] [PubMed] [Google Scholar]
  16. Gerke L., Meyrose A.-K., Ladwig I., Rief W., Nestoriuc Y. Frequencies and predictors of negative effects in routine inpatient and outpatient psychotherapy: two observational studies. Front. Psychol. 2020;11 doi: 10.3389/fpsyg.2020.02144. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Goldberg S.B., Sun S., Carlbring P., Torous J. Selecting and describing control conditions in mobile health randomized controlled trials: a proposed typology. Npj Digit. Med. 2023;6(1):1. doi: 10.1038/s41746-023-00923-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Green K.L., Brown G.K., Jager-Hyman S., Cha J., Steer R.A., Beck A.T. The predictive validity of the beck depression inventory suicide item. J. Clin. Psychiatry. 2015;76(12):15048. doi: 10.4088/JCP.14m09391. [DOI] [PubMed] [Google Scholar]
  19. Hautzinger M., Keller F., Kühner C. 2006. Beck Depressions-inventar (BDI-II) [DOI] [PubMed] [Google Scholar]
  20. Hendricks A., Cohen J.A., Mannarino A.P., Deblinger E. Your very own tf-cbt workbook. 2014. https://mindfulecotherapy.org/wp-content/uploads/2017/01/TF-CBT-Workbook-Final.pdf
  21. Herzog P., Lauff S., Rief W., Brakemeier E.-L. Assessing the unwanted: a systematic review of instruments used to assess negative effects of psychotherapy. Brain Behav. 2019;9(12) doi: 10.1002/brb3.1447. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Holmes E.A., Ghaderi A., Harmer C.J., Ramchandani P.G., Cuijpers P., Morrison A.P., Roiser J.P., Bockting C.L.H., O’Connor R.C., Shafran R., Moulds M.L., Craske M.G. The lancet psychiatry commission on psychological treatments research in tomorrow’s science. Lancet Psychiatry. 2018;5(3):237–286. doi: 10.1016/S2215-0366(17)30513-8. [DOI] [PubMed] [Google Scholar]
  23. Hoppen T.H., Lindemann A.S., Morina N. Safety of psychological interventions for adult post-traumatic stress disorder: meta-analysis on the incidence and relative risk of deterioration, adverse events and serious adverse events. Br. J. Psychiatry. 2022;221(5):658–667. doi: 10.1192/bjp.2022.111. [DOI] [PubMed] [Google Scholar]
  24. Horigian V.E., Robbins M.S., Dominguez R., Ucha J., Rosa C.L. Principles for defining adverse events in behavioral intervention research: lessons from a family-focused adolescent drug abuse trial. Clin. Trials. 2010;7(1):58–68. doi: 10.1177/1740774509356575. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Karyotaki E., Kemmeren L., Riper H., Twisk J., Hoogendoorn A., Kleiboer A., Mira A., Mackinnon A., Meyer B., Botella C., Littlewood E., Andersson G., Christensen H., Klein J.P., Schröder J., Bretón-López J., Scheider J., Griffiths K., Farrer L.…Cuijpers P. Is self-guided internet-based cognitive behavioural therapy (iCBT) harmful? An individual participant data meta-analysis. Psychol. Med. 2018;48(15):2456–2466. doi: 10.1017/S0033291718000648. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Klatte R., Strauss B., Flückiger C., Färber F., Rosendahl J. Defining and assessing adverse events and harmful effects in psychotherapy study protocols: a systematic review. Psychotherapy. 2022;60(1):130–148. doi: 10.1037/pst0000359. [DOI] [PubMed] [Google Scholar]
  27. Kroenke K., Wu J., Yu Z., Bair M.J., Kean J., Stump T., Monahan P.O. The patient health questionnaire anxiety and depression scale (PHQ-ADS): initial validation in three clinical trials. Psychosom. Med. 2016;78(6):716–727. doi: 10.1097/PSY.0000000000000322. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Ladwig I., Rief W., Nestoriuc Y. Welche Risiken und Nebenwirkungen hat Psychotherapie? - Entwicklung des Inventars zur Erfassung Negativer Effekte von Psychotherapie (INEP) Verhaltenstherapie. 2014;24(4):252–263. doi: 10.1159/000367928. [DOI] [Google Scholar]
  29. Limesurvey GmbH. LimeSurvey: an open source survey tool (version 5.4.11) [software]. http://www.limesurvey.org siehe. 2020. https://community.limesurvey.org/licence-trademark
  30. Linden M., Schermuly-Haupt M.-L. Definition, assessment and rate of psychotherapy side effects. World Psychiatry. 2014;13(3):306–309. doi: 10.1002/wps.20153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Louzon S.A., Bossarte R., McCarthy J.F., Katz I.R. Does suicidal ideation as measured by the PHQ-9 predict suicide among VA patients? Psychiatr. Serv. 2016;67(5):517–522. doi: 10.1176/appi.ps.201500149. [DOI] [PubMed] [Google Scholar]
  32. Lutz W., De Jong K., Rubel J. Patient-focused and feedback research in psychotherapy: where are we and where do we want to go? Psychother. Res. 2015;25(6):625–632. doi: 10.1080/10503307.2015.1079661. [DOI] [PubMed] [Google Scholar]
  33. McQuaid A., Sanatinia R., Farquharson L., Shah P., Quirk A., Baldwin D.S., Crawford M. Patient experience of lasting negative effects of psychological interventions for anxiety and depression in secondary mental health care services: A national cross-sectional study. BMC Psychiatry. 2021;21(1):578. doi: 10.1186/s12888-021-03588-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Micklitz H.M., Glass C.M., Bengel J., Sander L.B. Efficacy of psychosocial interventions for survivors of intimate partner violence: a systematic review and meta-analysis. Trauma Violence Abuse. 2023 doi: 10.1177/15248380231169481. [DOI] [PubMed] [Google Scholar]
  35. Moritz S., Nestoriuc Y., Rief W., Klein J.P., Jelinek L., Peth J. It can’t hurt, right? Adverse effects of psychotherapy in patients with depression. Eur. Arch. Psychiatry Clin. Neurosci. 2019;269(5):577–586. doi: 10.1007/s00406-018-0931-1. [DOI] [PubMed] [Google Scholar]
  36. Nielsen M., D’Agostino D., Gregory P. Addressing rural health hhallenges head on. Mo. Med. 2017;114(5):363–366. [PMC free article] [PubMed] [Google Scholar]
  37. Nock M.K., Holmberg E.B., Photos V.I., Michel B.D. Self-injurious thoughts and behaviors interview: development, reliability, and validity in an adolescent sample. Psychol. Assess. 2007;19(3):309–317. doi: 10.1037/1040-3590.19.3.309. [DOI] [PubMed] [Google Scholar]
  38. Oehler C., Görges F., Hegerl U., Rummel-Kluge C. A closer look at negative effects in a guided web-based intervention for mild to moderate depression. Clin. Psychol. Sci. Pract. 2021;28(2):131–141. doi: 10.1037/cps0000004. [DOI] [Google Scholar]
  39. Posner K., Brown G.K., Stanley B., Brent D.A., Yershova K.V., Oquendo M.A., Currier G.W., Melvin G.A., Greenhill L., Shen S., Mann J.J. The Columbia–Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am. J. Psychiatry. 2011;168(12):1266–1277. doi: 10.1176/appi.ajp.2011.10111704. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Pynoos R.S., Weathers F.W., Steinberg A.M., Marx B.P., Layne C.M., Kaloupek D.G., Schnurr P.P., Keane T.M., Blake D.D., Newman E., Nader K.O., Kriegler J.A. 2015. Clinician-Administered PTSD Scale for DSM-5—Child/Adolescent Version. In Ptsd.va.gov. [Google Scholar]
  41. Rozental A., Andersson G., Boettcher J., Ebert D.D., Cuijpers P., Knaevelsrud C., Ljótsson B., Kaldo V., Titov N., Carlbring P. Consensus statement on defining and measuring negative effects of Internet interventions. Internet Interv. 2014;1(1):12–19. doi: 10.1016/j.invent.2014.02.001. [DOI] [Google Scholar]
  42. Rozental A., Boettcher J., Andersson G., Schmidt B., Carlbring P. Negative effects of internet interventions: a qualitative content analysis of patients’ experiences with treatments delivered online. Cogn. Behav. Ther. 2015;44(3):223–236. doi: 10.1080/16506073.2015.1008033. [DOI] [PubMed] [Google Scholar]
  43. Rozental A., Kottorp A., Boettcher J., Andersson G., Carlbring P. Negative effects of psychological treatments: an exploratory factor analysis of the negative effects questionnaire for monitoring and reporting adverse and unwanted events. PLoS One. 2016;11(6) doi: 10.1371/journal.pone.0157503. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Rozental A., Magnusson K., Boettcher J., Andersson G., Carlbring P. For better or worse: an individual patient data meta-analysis of deterioration among participants receiving Internet-based cognitive behavior therapy. J. Consult. Clin. Psychol. 2017;85(2):160–177. doi: 10.1037/ccp0000158. [DOI] [PubMed] [Google Scholar]
  45. Rozental A., Andersson G., Carlbring P. In the absence of effects: an individual patient data meta-analysis of non-response and its predictors in internet-based Cognitive Behavior Therapy. Front. Psychol. 2019;10 doi: 10.3389/fpsyg.2019.00589. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Rush A.J., Trivedi M.H., Ibrahim H.M., Carmody T.J., Arnow B., Klein D.N., Markowitz J.C., Ninan P.T., Kornstein S., Manber R., Thase M.E., Kocsis J.H., Keller M.B. The 16-Item quick inventory of depressive symptomatology (QIDS), clinician rating (QIDS-C), and self-report (QIDS-SR): a psychometric evaluation in patients with chronic major depression. Biol. Psychiatry. 2003;54(5):573–583. doi: 10.1016/S0006-3223(02)01866-8. [DOI] [PubMed] [Google Scholar]
  47. Sachser C., Berliner L., Risch E., Rosner R., Birkeland M.S., Eilers R., Hafstad G.S., Pfeiffer E., Plener P.L., Jensen T.K. The child and Adolescent Trauma Screen 2 (CATS-2) – validation of an instrument to measure DSM-5 and ICD-11 PTSD and complex PTSD in children and adolescents. Eur. J. Psychotraumatol. 2022;13(2):2105580. doi: 10.1080/20008066.2022.2105580. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Sander L.B., Gerhardinger K., Bailey E., Robinson J., Lin J., Cuijpers P., Mühlmann C. Suicide risk management in research on internet-based interventions for depression: a synthesis of the current state and recommendations for future research. J. Affect. Disord. 2020;263:676–683. doi: 10.1016/j.jad.2019.11.045. [DOI] [PubMed] [Google Scholar]
  49. Schermuly-Haupt M.-L., Linden M., Rush A.J. Unwanted events and side effects in Cognitive Behavior Therapy. Cogn. Ther. Res. 2018;42(3):219–229. doi: 10.1007/s10608-018-9904-y. [DOI] [Google Scholar]
  50. Schlicker S., Baumeister H., Buntrock C., Sander L., Paganini S., Lin J., Berking M., Lehr D., Ebert D.D. A web- and mobile-based intervention for comorbid, recurrent depression in patients with chronic back pain on sick leave (Get.Back): pilot randomized controlled trial on feasibility, user satisfaction, and effectiveness. JMIR Ment. Health. 2020;7(4) doi: 10.2196/16398. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Schulte, C., Sachser, C., Rosner, R., Ebert, D. D., Zarski, A.-C., n.d.. Internet- and mobile-based trauma-focused intervention for adolescents and young adults with posttraumatic stress disorder: Results of a proof-of-concept feasibility study. Eur. J. Psychotraumatol. [DOI] [PMC free article] [PubMed]
  52. Schulte C., Zarski A.-C., Sachser C., Rosner R., Ebert D.D. Internet- and mobile-based trauma-focused intervention for adolescents and young adults with posttraumatic stress disorder: a study protocol of a proof-of-concept feasibility study. Eur. J. Psychotraumatol. 2022;13(2) doi: 10.1080/20008198.2022.2101345. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Seiferth C., Vogel L., Aas B., Brandhorst I., Carlbring P., Conzelmann A., Esfandiari N., Finkbeiner M., Hollmann K., Lautenbacher H., Meinzinger E., Newbold A., Opitz A., Renner T.J., Sander L.B., Santangelo P.S., Schoedel R., Schuller B., Stachl C.…Löchner J. How to e-mental health: A guideline for researchers and practitioners using digital technology in the context of mental health. Nat. Mental Health. 2023;1(8), Article 8 doi: 10.1038/s44220-023-00085-1. [DOI] [Google Scholar]
  54. Sels L., Homan S., Ries A., Santhanam P., Scheerer H., Colla M., Vetter S., Seifritz E., Galatzer-Levy I., Kowatsch T., Scholz U., Kleim B. SIMON: a digital protocol to monitor and predict suicidal ideation. Front. Psychol. 2021;12 doi: 10.3389/fpsyt.2021.554811. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Simon T.R., Swann A.C., Powell K.E., Potter L.B., Kresnow M., O’Carroll P.W. Characteristics of impulsive suicide attempts and attempters. Suicide Life Threat. Behav. 2001;32(Supplement to Issue 1):49–59. doi: 10.1521/suli.32.1.5.49.24212. [DOI] [PubMed] [Google Scholar]
  56. Simon G.E., Rutter C.M., Peterson D., Oliver M., Whiteside U., Operskalski B., Ludman E.J. Does response on the PHQ-9 depression questionnaire predict subsequent suicide attempt or suicide death? Psychiatr. Serv. 2013;64(12):1195–1202. doi: 10.1176/appi.ps.201200587. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Smith K.A., Blease C., Faurholt-Jepsen M., Firth J., Daele T.V., Moreno C., Carlbring P., Ebner-Priemer U.W., Koutsouleris N., Riper H., Mouchabac S., Torous J., Cipriani A. Digital mental health: challenges and next steps. BMJ Ment. Health. 2023;26(1) doi: 10.1136/bmjment-2023-300670. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. van Spijker B.A., Calear A.L., Batterham P.J., Mackinnon A.J., Gosling J.A., Kerkhof A.J., Solomon D., Christensen H. Reducing suicidal thoughts in the Australian general population through web-based self-help: study protocol for a randomized controlled trial. Trials. 2015;16(1):62. doi: 10.1186/s13063-015-0589-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Steeg S., Quinlivan L., Nowland R., Carroll R., Casey D., Clements C., Cooper J., Davies L., Knipe D., Ness J., O’Connor R.C., Hawton K., Gunnell D., Kapur N. Accuracy of risk scales for predicting repeat self-harm and suicide: a multicentre, population-level cohort study using routine clinical data. BMC Psychiatry. 2018;18(1):113. doi: 10.1186/s12888-018-1693-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Strauss B., Gawlytta R., Schleu A., Frenzl D. Negative effects of psychotherapy: estimating the prevalence in a random national sample. BJPsych Open. 2021;7(6) doi: 10.1192/bjo.2021.1025. [DOI] [Google Scholar]
  61. Teismann T., Forkmann T., Sander L.B., Glaesmer H. Umgang mit suizidalem Erleben und Verhalten in Forschungsprojekten. PPmP - Psychotherapie · Psychosomatik · Medizinische Psychologie. 2021;71(9/10):423–424. doi: 10.1055/a-1542-9753. [DOI] [PubMed] [Google Scholar]
  62. Torous J., Larsen M.E., Depp C., Cosco T.D., Barnett I., Nock M.K., Firth J. Smartphones, sensors, and machine learning to advance real-time prediction and interventions for suicide prevention: a review of current progress and next steps. Curr. Psychiatry Rep. 2018;20(7):51. doi: 10.1007/s11920-018-0914-y. [DOI] [PubMed] [Google Scholar]
  63. Trivedi M.H., Rush A.J., Ibrahim H.M., Carmody T.J., Biggs M.M., Suppes T., Crismon M.L., Shores-Wilson K., Toprac M.G., Dennehy E.B., Witte B., Kashner T.M. The Inventory of Depressive Symptomatology, Clinician Rating (IDS-C) and Self-Report (IDS-SR), and the Quick Inventory of Depressive Symptomatology, Clinician Rating (QIDS-C) and Self-Report (QIDS-SR) in public sector patients with mood disorders: a psychometric evaluation. Psychol. Med. 2004;34(1):73–82. doi: 10.1017/S0033291703001107. [DOI] [PubMed] [Google Scholar]
  64. Watson P.N., Leroux E., Chowdhury M., Mehra K., Henderson J., Szatmari P., Hawke L. Unexpressed wishes and unmet needs: a mixed methods study of youth negative experiences in psychotherapy. J. Child Fam. Stud. 2023;32(2):424–437. doi: 10.1007/s10826-022-02431-w. [DOI] [Google Scholar]
  65. Werntz A., Silverman A.L., Behan H., Patel S.K., Beltzer M., Boukhechba M.O., Barnes L., Teachman B.A. Lessons learned: providing supportive accountability in an online anxiety intervention. Behav. Ther. 2022;53(3):492–507. doi: 10.1016/j.beth.2021.12.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Wojnar M., Ilgen M.A., Czyz E., Strobbe S., Klimkiewicz A., Jakubczyk A., Glass J., Brower K.J. Impulsive and non-impulsive suicide attempts in patients treated for alcohol dependence. J. Affect. Disord. 2009;115(1):131–139. doi: 10.1016/j.jad.2008.09.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  67. World Health Organization ICD-10 Version:2019. 2019. https://icd.who.int/browse10/2019/en
  68. Wyder M., De Leo D. Behind impulsive suicide attempts: indications from a community study. J. Affect. Disord. 2007;104(1):167–173. doi: 10.1016/j.jad.2007.02.015. [DOI] [PubMed] [Google Scholar]
  69. Zarski A.-C., Berking M., Ebert D.D. Efficacy of internet-based treatment for genito-pelvic pain/penetration disorder: results of a randomized controlled trial. J. Consult. Clin. Psychol. 2021;89:909–924. doi: 10.1037/ccp0000665. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Appendix A

Standard operating procedures youth standalone study.

mmc1.docx (332.7KB, docx)
Appendix B

Procedures.

mmc2.docx (26.7KB, docx)

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES