Abstract
Purpose:
The “It Gets Better” project (IGBP) features video narratives of lesbian, gay, bisexual, transgender, queer persons or persons with other sexual or gender minority identities (LGBTQ+) of overcoming coming-out-related difficulties. This is the first experimental study investigating effects of these videos.
Methods:
We conducted a double-blind randomized controlled trial on-site in Austria and online in German-language settings from January to November 2020 with LGBTQ+ youth (14–22 years; n = 483), randomized to an IGBP (n = 242) or control video (n = 241). Suicidal ideation (primary outcome), help-seeking intentions, hopelessness, mood, and sexual identity were assessed at baseline (T1), postexposure (T2), and 4-week follow-up (T3). We assessed differences among gender identities, sexual orientations, with regard to depressive symptoms, and the role of identification. Data were analyzed with linear mixed models and mediation analysis.
Results:
There was no overall effect on suicidal ideation, but nonbinary/transgender individuals experienced a small-sized improvement (T2: mean change [MC] from baseline MC = −0.06 [95% confidence interval {CI} −0.16 to 0.05], p = 0.60; mean difference [MD] to controls MD = −0.42 [95% CI −0.79 to −0.06], p = 0.02, d = −0.10). An indirect preventive effect on suicidal ideation at T2 through the degree of identification with the protagonist in the video was observed. There was improvement in help-seeking intentions in the intervention group (T2: MC = 0.25 [95% CI 0.15 to 0.35], p < 0.001; MD = 0.28 [95% CI 0.01 to 0.54], p < 0.05, d = 0.09).
Conclusion:
Video narratives featuring coping might have some potential to decrease suicidal ideation and encourage help-seeking among vulnerable youth identifying with videos, but effects are small and short-lived.
Study Registration:
German Clinical Trial Registry (DRKS00019913).
Keywords: it gets better, media, randomized controlled trial, suicidal ideation, suicide, suicide prevention
Introduction
Lesbian, gay, bisexual, transgender, or queer persons or persons with other sexual or gender minority identities (LGBTQ+) are important target groups for suicide prevention.1 Particularly, young people in the development phase of their sexual identity have an increased risk of suicidal ideation and behaviors compared to non-LGBTQ+ peers.2 This appears most pronounced for transgender and bisexual individuals.1,3
Interventions tailored to the needs of LGBTQ+ youth and evaluations are scarce.4 The “It Gets Better” project (IGBP) aims to bridge this gap and empower youth by featuring personal video narratives of coping with difficulties during coming out.5 Originally started in the United States, the project has spread to 17 countries, including Austria.5,6
The strategy of using personal narratives of hope and recovery is of increasing relevance to suicide prevention.7 Emerging evidence suggests that brief media interventions featuring narratives of hope can reduce suicidal ideation in the audience, the “Papageno effect.”8–10 Some studies suggest most pronounced effects for vulnerable people, especially people with depressive symptoms or a recent suicide attempt.10 Research on the “Papageno effect” in LGBTQ+ individuals, however, is lacking. This is a missed opportunity, as they appear to rely more on media models due to the underrepresentation of role models in the immediate social environment.11
There are no experimental studies available that analyzed the effects of IGBP videos. Therefore, we conducted this randomized controlled trial (RCT), hypothesizing that participants would benefit from selected IGBP videos in terms of a reduction in suicidal ideation (primary outcome), improvements in help-seeking intentions, identity challenges, and hopelessness. We assumed a mediation effect on suicidal ideation through identification with the featured individuals. We explored if (1) effects varied for different gender identities, (2) sexual orientations, and (3) participants with/without depressive symptoms.
Materials and Methods
Participants and recruitment
The RCT was conducted online and on-site between January and November 2020. German-speaking participants (14–22 years), who were LGBTQ+, were invited to take part in a study on video effects. The online part of the study was conducted in German-speaking settings (i.e., Austria and Germany), while the on-site part of the study was conducted in Vienna, Austria.
The study was promoted through LGBTQ+ organizations in Austria and Germany and shared on LGBTQ+ facebook groups, reddit, or tumblr pages targeting LGBTQ+ youth. A well-known Austrian LGBTQ+ influencer shared the study on her Instagram channel. We also collaborated with Romeo to recruit gay and bisexual men.12 All study materials were provided online using Social Science Survey.13 An intake call/e-mail exchange took place for on-site participants to arrange an appointment for T1 and check eligibility. Online participants had direct access to the questionnaire and eligibility was checked online. Informed consent was obtained from participants in accordance with ethics regulations. For minors participating on-site, we obtained additional parental consent.
Sample size calculation
The sample size was estimated using G*Power 3.1.9.2.14 Using a repeated measures analysis of covariance with an assumed correlation of 0.79 between three repeated measurements, and 16 groups (gender: male or female; study group: intervention or control; identification: median split into high/low; and baseline vulnerability: median split into high/low), a two-sided significance level alpha <0.05, and a power of 0.80, we required a sample size of n = 288 to detect an effect size of f = 0.23.9 A power analysis using General Linear Mixed Model Power and Sample Size, which handles mixed models similarly, indicated a sample size of n = 284.15
Videos
Two videos, featuring a young cisgender woman and man, respectively, were used for the experiment. These were selected by a jury of n = 19 lesbian, gay, bisexual, and queer adolescents, as well as adolescents with other sexual minority identities (LGBQ+), and n = 9 suicide and media experts. The jury was asked to rate videos for their perceived suitability to help prevent suicide and strengthen sexual identity. The videos judged to be most suitable by the jury were selected as intervention videos and are available on YouTube.16–18
Rudi's story
Rudi talks about his struggle of realizing and accepting that he is gay. He is afraid of coming out to his conservative family and talks about having had depression and attempting suicide. With his first boyfriend, things improve and Rudi comes out to his family, who are more supportive than he had thought. Rudi encourages viewers to come out to the people important to them. The video is 7-minutes long.
Alice's story
Alice talks about the conservative setting she grew up in. Her situation worsens when her mother catches her kissing a girl and threatens Alice to expel her from the family. Alice describes how music and support from her girlfriend kept her going. Eventually, Alice moves to an urban area, where she feels better. Alice encourages others to get in contact with LGBTQ+ youth groups. The video is 3-minutes long.
Alice and Rudi were asked to produce a video for the control group of the trial. Requirements were a similar length and setting. The content was about personal narratives of incorporating a healthy lifestyle into everyday life (i.e., how to eat healthy and exercise regularly), thus unrelated to difficulties with coming out.
Procedure
After obtaining informed consent, participants could select between watching a video featuring a female or a male protagonist. They were then randomly allocated to the respective intervention/control group using the random generator in Social Science Survey ensuring equal group sizes (1:1 allocation ratio).19 Participants and researchers were blinded to group assignment until data collection was completed.
The sociodemographic variables were assessed at baseline (before randomization, T1). All outcome variables were assessed at T1, after watching the video (T2) and at 4-week follow-up (T3). Identification with the featured protagonist was assessed at T2 and blinding success at T3. After participation, all individuals were informed about the study objective and received contact information to crisis support services.
Sociodemographics
To collect data on sociodemographics, we used purpose-designed questions. We assessed age, place of residence (Austria, Germany, and other, with an additional question to specify the country of residence), gender identity, and sexual orientation. Furthermore, we assessed any history of suicidal behavior in the past (e.g., “Did you ever attempt suicide?”) and current mental health treatment (yes/no).
The questions assessing the participants' sexual and gender identity were based on best practice examples.20,21 Both questions were intended to be as inclusive as possible with only as many answering options as necessary. In addition, participants had the possibility to describe their identity through an open text field (“I would best describe myself as…”). After data collection was finished, all open text fields were examined and, due to sample size constraints, some categories had to be merged. In particular, people who identified as gender fluid, genderqueer, demigender, or questioning were grouped together with nonbinary and transgender individuals.
Primary outcome measure
Reasons for Living Inventory-Adolescents
We assessed suicidal ideation with the 32-item Reasons for Living Inventory-Adolescents about reasons for not wishing to die by suicide (e.g., “I believe I can find other solutions to my problems”).22 The scale has five subscales: future optimism, suicide-related concerns, family alliance, peer acceptance and support, and self-acceptance. Participants answered on a Likert scale from 1 (not at all important) to 6 (extremely important). The scale correlates well with other measures of suicidal ideation.23 Scores were reverse-coded, with higher scores indicating higher suicidal ideation (Cronbach's α = 0.94).
Secondary outcome measures
The General Help-Seeking Questionnaire
The General Help-Seeking Questionnaire measures the likelihood of seeking help in case of suicidal thoughts.24 Participants were asked to rate sources (e.g., “friends,” “phone helpline”) on a Likert scale from 1 (extremely unlikely) to 7 (extremely likely). We grouped sources into personal (Cronbach's α = 0.67) and professional help (Cronbach's α = 0.65).25,26
Lesbian, Gay, and Bisexual Identity Scale
The Lesbian, Gay, and Bisexual Identity Scale measures sexual minority identity and internalized homonegativity.27,28 The 27-item scale (e.g., “My sexual orientation is a central part of my identity”) is rated on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). Subscales were merged into “Identity Challenges/Negative Identity” (Cronbach's α = 0.83).27 Higher scores indicate more unease with one's sexual identity (Cronbach's α = 0.83).
Mood subscale
The mood subscale (Affective State Scale) assessed the participants' mood. It uses eight adjectives (e.g., merry, sad) rated on a Likert scale from 1 (not at all) to 4 (highly).29 Higher scores indicate better mood (Cronbach's α = 0.88).
Beck Hopelessness Scale
We used the short version of the Beck Hopelessness Scale to assess hopelessness.30,31 This questionnaire has 10 items (e.g., “The future seems vague and uncertain to me”). It is rated on a scale from 1 (fully disagree) to 6 (completely agree, Cronbach's α = 0.83).
Additional measures
Patient Health Questionnaire-9
Participants indicated depressive symptoms in the past 2 weeks (e.g., little interest or pleasure in doing things).32 The questionnaire consists of nine items with a Likert scale from 0 (not at all) to 3 (nearly every day). High scores indicate more depressive symptoms (Cronbach's α = 0.85).
Identification
The Cohen's Identification Scale was used to assess identification with the protagonist.33 This questionnaire has 10 items. It is rated on a 5-point Likert Scale from 1 (do not agree) to 5 (fully agree) (e.g., “While viewing the video, I felt as if I was part of the action,” Cronbach's α = 0.90).
Blinding success
We also assessed blinding success. Participants were asked which group they thought they were allocated to (intervention group, control group, and do not know). This was assessed at T3.34
Data analysis
The data were analyzed with linear mixed models, which are appropriate to handle missing data by taking into account all available data from each participant.35 A group (intervention group, control group) × time (preintervention T1, postintervention T2, and follow-up 4 weeks later T3) model was built for the primary research question. To analyze variations with gender identity, we added gender (cisgender male, cisgender female, and nonbinary/transgender) to the basic model.
Variations in effects with severity of depressive symptoms (as indicated by a score of >14 on the Patient Health Questionnaire-9)32 were tested by adding severity of depressive symptoms (yes/no) to the model. Sexual orientation was categorized into homosexual, bisexual, and other, the latter containing queer, pan/omnisexual, asexual, and questioning participants.
Due to the low number of participants in combined subgroups, we tested effects in separate models. Because of age differences, all analyses, including gender identity or sexual orientation, were adjusted for age. Multiple testing was Bonferroni corrected. Stratified analyses for gender identity and sexual orientation were reported for significant interactions only. A mediation analysis was run on the entire sample to test mediation effects of identification regarding suicidal ideation at T2 using the SPSS Macro “Process v3.5” by Andrew F. Hayes (Model 4; 5000 bootstrapping samples).36
Sensitivity analyses
We conducted four sensitivity analyses to examine any differences of findings from the total sample. We ran subsample analyses with (1) online participants (n = 321), (2) participants resident in Austria (n = 264), (3) participants who watched the video featuring the female protagonist (n = 295), and (4) participants with complete participation (n = 297).
Ethics statement
Ethical approval was obtained from the Institutional Review Board of the Medical University of Vienna (1013/2019). All procedures contributing to this work complied with the Helsinki Declaration of 1975, as revised in 2013.37 Trial registration: German Clinical Trial Registry (DRKS00019913).38
Results
Overall, n = 242 (50.1%) participants were randomized to the intervention and n = 241 (49.9%) to the control group. A total of n = 297 (61.5%) completed the trial (Fig. 1). Following the intention-to-treat principle, all randomized participants (n = 483) were included in the statistical analysis.39
Descriptive characteristics
We tested differences in the baseline characteristics between the two groups. There were no differences regarding the participants' characteristics. This also applied to the outcome variables at baseline (Table 1).
Table 1.
Variable | Intervention group, n (%) | Control group, n (%) | χ2/F |
---|---|---|---|
Age, mean (SD) | 18.96 (2.24) | 19.16 (2.25) | 0.95a |
Nation | 0.20b | ||
Austria | 132 (54.5) | 132 (54.8) | |
Germany | 106 (43.8) | 106 (44.0) | |
Other | 4 (1.7) | 3 (1.2) | |
Gender identity | 1.09c | ||
Cisgender male | 70 (28.9) | 60 (24.9) | |
Cisgender female | 122 (50.4) | 131 (54.4) | |
Nonbinary/transgender | 50 (20.7) | 50 (20.7) | |
Sexual orientation | 3.75d | ||
Gay | 63 (26.0) | 54 (22.4) | |
Lesbian | 53 (21.9) | 52 (21.6) | |
Bisexual | 64 (26.4) | 69 (28.6) | |
Questioning | 13 (5.4) | 11 (4.6) | |
Queer | 30 (12.4) | 40 (16.6) | |
Pan/omnisexual | 13 (5.4) | 8 (3.3) | |
Asexual and romantic | 6 (2.5) | 7 (2.9) | |
Suicide attempt in past year | 0.51e | ||
Yes | 43 (17.8) | 37 (15.4) | |
No | 199 (82.2) | 204 (84.6) | |
Current mental health treatment | 0.001e | ||
Yes | 53 (21.9) | 53 (22.0) | |
No | 189 (78.1) | 188 (78.0) | |
Severe depressive symptoms | 0.91e | ||
Yes | 59 (24.4) | 50 (20.7) | |
No | 183 (75.6) | 191 (79.3) | |
Baseline assessment of outcomes, mean (SD) | |||
Suicidal ideation | 2.68 (0.88) | 2.72 (0.93) | 0.28a |
Hopelessness | 2.88 (0.85) | 2.93 (0.89) | 0.39a |
Mood | 2.73 (0.68) | 2.77 (0.65) | 0.50a |
Help-seeking personal contacts | 3.76 (1.35) | 3.73 (1.30) | 0.07a |
Help-seeking professional help | 3.67 (1.55) | 3.77 (1.60) | 0.47a |
Internalized homonegativity | 1.80 (1.12) | 1.76 (1.12) | 0.17a |
Identity challenges/negative identity | 2.90 (0.75) | 2.92 (0.76) | 0.08a |
ANOVA, df1 = 1, df2 = 481.
Fisher's exact test.
Chi-squared test, df = 2.
Chi-squared test, df = 6.
Chi-squared test, df = 1.
ANOVA, analysis of variance; df, degrees of freedom; SD, standard deviation.
Differences between survey completers and dropouts
Participants who dropped out after randomization (n = 186) were younger and more likely from Germany. Furthermore, they had more severe depressive symptoms. They also showed worse scores in mood, hopelessness, internalized homonegativity, and identity challenges at baseline compared to survey completers (Supplementary Table S1).
Differences among gender identities
Nonbinary/transgender participants (n = 100) were younger and more likely to have severe depressive symptoms. Furthermore, they were more likely to indicate a past suicide attempt and current mental health treatment than their cisgender peers (Supplementary Table S2). They also scored worse in suicidal ideation, mood, and hopelessness at baseline (Supplementary Table S2). Differences among study participants (1) with or without depressive symptoms and (2) of different sexual orientations are presented in Supplementary Tables S3 and S4.
Blinding success
More participants did not know their group assignment (n = 120; 40.4%) compared to participants who guessed correctly (n = 105, 35.4%) or incorrectly (n = 72; 24.2%). There was no difference between the intervention and control group (χ2 = 3.48, df = 2, p = 0.18). These observations suggest successful blinding.
Sensitivity analyses
The patterns identified in the sensitivity analyses of subsamples were generally comparable to the full sample. For participants who watched the video featuring the female protagonist and for online participants, however, interaction terms for suicidal ideation were not significant anymore compared to the full sample, although the estimates pointed in the same direction as in the full sample. This was likely due to the smaller size of the respective subgroups analyzed.
Main result: effects of videos on intervention group
No effect was observed in the group × time model for the primary outcome suicidal ideation (Table 2). Help-seeking intentions for personal contacts increased after exposure to the intervention video (T2: mean change [MC] from baseline MC = 0.25 [95% confidence interval {CI} 0.15 to 0.35], p < 0.001; mean difference [MD] compared with control video MD = 0.28 [95% CI 0.01 to 0.54], p < 0.05, d = 0.09). The difference between the two groups at T3 was close to zero (T3: MC = 0.17 [95% CI −0.01 to 0.36], p = 0.08; MD = 0.11 [95% CI −0.17 to 0.39], p = 0.45, d = 0.03). An improvement in mood and hopelessness was found in both groups (Table 3).
Table 2.
Study variable | Group | Time | Group × time | |
---|---|---|---|---|
Primary outcome | ||||
Suicidal ideation |
F (df1, df2) p |
0.12 (1, 479.85) 0.73 |
1.97 (2, 366.14) 0.14 |
0.56 (2, 366.14) 0.57 |
Secondary outcomes | ||||
Hopelessness |
F (df1, df2) p |
0.38 (1, 479.43) 0.54 |
14.45 (2, 377.05) <0.001 |
0.01 (2, 377.05) 0.99 |
Mood |
F (df1, df2) p |
0.25 (1, 465.98) 0.62 |
13.35 (2, 363.53) <0.001 |
1.55 (2, 363.53) 0.21 |
Help-seeking personal contacts |
F (df1, df2) p |
0.40 (1, 475.43) 0.53 |
23.91 (2, 362.20) <0.001 |
3.66 (2, 362.20) 0.03 |
Help-seeking professional help |
F (df1, df2) p |
0.72 (1, 475.20) 0.40 |
3.04 (2, 356.27) 0.049 |
0.19 (2, 356.27) 0.82 |
Internalized homonegativity |
F (df1, df2) p |
0.02 (1, 473.79) 0.89 |
3.26 (2, 355.29) 0.04 |
1.00 (2, 355.29) 0.37 |
Identity challenges/negative identity |
F (df1, df2) p |
0.14 (1, 476.00) 0.71 |
2.22 (2, 354.25) 0.11 |
0.16 (2, 354.25) 0.85 |
Note: F and p values with degrees of freedom (df1 and df2) given in parentheses from linear mixed models representing the change of the respective outcome variable with regard to group (intervention group and control group), time (T1, T2, and T3), and interactions between these factors. Significant p values (<0.05) are marked in bold; the analyses were adjusted for multiple testing using the Bonferroni correction.
Table 3.
Control group (n = 241) |
Intervention group (n = 242) |
|||||
---|---|---|---|---|---|---|
Mean score (95% CI) | MC (95% CI)a from baseline | Mean score (95% CI) | MC (95% CI)a from baseline | MD (95% CI)b | Cohen's d | |
Suicidal ideation total score | ||||||
After exposure (T2) | ||||||
Cisgender male | 2.63 (2.40 to 2.87) | 0.02 (−0.08 to 0.12) | 2.60 (2.39 to 2.82) | −0.04 (−0.13 to 0.05) | −0.03 (−0.35 to 0.29) | −0.01 |
Cisgender female | 2.57 (2.41 to 2.73) | −0.01 (−0.07 to 0.06) | 2.64 (2.48 to 2.81) | 0.04 (−0.03 to 0.11) | 0.07 (−0.16 to 0.30) | 0.03 |
Nonbinary/transgender | 3.22 (2.96 to 3.47) | −0.004 (−0.11 to 0.10) | 2.79 (2.53 to 3.05) | −0.06 (−0.16 to 0.05) | −0.42 (−0.79 to −0.06)* | −0.10 |
All groups combined | 2.81 (2.68 to 2.93) | 0.002 (−0.05 to 0.06) | 2.68 (2.55 to 2.80) | −0.02 (−0.07 to 0.03) | −0.13 (−0.31 to 0.05) | −0.06 |
4 weeks later (T3) | ||||||
Cisgender male | 2.82 (2.58 to 3.06) | 0.21 (0.02 to 0.39)* | 2.58 (2.36 to 2.80) | −0.06 (−0.23 to 0.11) | −0.24 (−0.56 to 0.09) | −0.07 |
Cisgender female | 2.60 (2.44 to 2.75) | 0.02 (−0.10 to 0.13) | 2.77 (2.60 to 2.93) | 0.16 (0.04 to 0.29)* | 0.17 (−0.06 to 0.40) | 0.07 |
Nonbinary/transgender | 3.08 (2.82 to 3.33) | −0.14 (−0.34 to 0.05) | 2.89 (2.64 to 3.15) | 0.05 (−0.15 to 0.23) | −0.18 (−0.55 to 0.18) | −0.05 |
All groups combined | 2.83 (2.70 to 2.96) | 0.03 (−0.07 to 0.12) | 2.75 (2.62 to 2.87) | 0.05 (−0.05 to 0.14) | −0.08 (−0.26 to 0.10) | −0.04 |
Suicidal ideation future optimism | ||||||
After exposure (T2) | ||||||
Cisgender male | 2.53 (2.23 to 2.83) | −0.01 (−0.18 to 0.16) | 2.38 (2.10 to 2.66) | 0.02 (−0.14 to 0.18) | −0.15 (−0.56 to 0.26) | −0.03 |
Cisgender female | 2.54 (2.34 to 2.74) | −0.01 (−0.13 to 0.10) | 2.50 (2.29 to 2.71) | 0.00 (−0.12 to 0.12) | −0.04 (−0.33 to 0.25) | −0.01 |
Nonbinary/transgender | 3.10 (2.77 to 3.43) | 0.01 (−0.18 to 0.20) | 2.56 (2.23 to 2.89) | −0.20 (−0.39 to −0.02)* | −0.54 (−1.01 to −0.08)* | −0.10 |
All groups combined | 2.72 (2.56 to 2.89) | −0.004 (−0.10 to 0.09) | 2.48 (2.32 to 2.64) | −0.06 (−0.15 to 0.03) | −0.24 (−0.47 to −0.01)* | −0.09 |
4 weeks later (T3) | ||||||
Cisgender male | 2.60 (2.28 to 2.92) | 0.06 (−0.23 to 0.35) | 2.24 (1.94 to 2.53) | −0.12 (−0.39 to 0.14) | −0.36 (−0.79 to 0.08) | −0.07 |
Cisgender female | 2.52 (2.31 to 2.72) | −0.03 (−0.21 to 0.15) | 2.59 (2.37 to 2.81) | 0.09 (−0.10 to 0.28) | 0.07 (−0.23 to 0.38) | 0.02 |
Nonbinary/transgender | 2.86 (2.53 to 3.20) | −0.23 (−0.53 to 0.06) | 2.78 (2.44 to 3.11) | 0.01 (−0.28 to 0.30) | −0.09 (−0.57 to 0.39) | −0.02 |
All groups combined | 2.66 (2.49 to 2.83) | −0.07 (−0.22 to 0.08) | 2.54 (2.37 to 2.70) | −0.01 (−0.15 to 0.14) | −0.12 (−0.36 to 0.11) | −0.05 |
Hopelessness | ||||||
After exposure (T2) | ||||||
Cisgender male | 2.66 (2.44 to 2.89) | −0.06 (−0.18 to 0.06) | 2.62 (2.41 to 2.82) | 0.08 (−0.03 to 0.20) | −0.05 (−0.35 to 0.25) | −0.01 |
Cisgender female | 2.77 (2.62 to 2.92) | −0.12 (−0.20 to −0.04)* | 2.80 (2.64 to 2.95) | −0.13 (−0.22 to −0.05)* | 0.03 (−0.19 to 0.24) | 0.01 |
Nonbinary/transgender | 3.19 (2.95 to 3.44) | −0.08 (−0.21 to 0.06) | 2.95 (2.71 to 3.19) | −0.22 (−0.36 to −0.09)** | −0.24 (−0.58 to 0.10) | −0.06 |
All groups combined | 2.88 (2.76 to 3.00) | −0.09 (−0.15 to −0.02)* | 2.79 (2.67 to 2.91) | −0.09 (−0.16 to −0.03)* | −0.09 (−0.26 to 0.08) | −0.05 |
4 weeks later (T3) | ||||||
Cisgender male | 2.76 (2.54 to 2.98) | 0.03 (−0.15 to 0.21) | 2.46 (2.26 to 2.67) | −0.07 (−0.24 to 0.09) | −0.29 (−0.59 to 0.01) | −0.09 |
Cisgender female | 2.78 (2.64 to 2.93) | −0.11 (−0.22 to 0.01) | 2.92 (2.77 to 3.08) | −0.01 (−0.13 to 0.12) | 0.14 (−0.07 to 0.35) | 0.06 |
Nonbinary/transgender | 3.13 (2.90 to 3.37) | −0.14 (−0.32 to 0.05) | 2.94 (2.70 to 3.17) | −0.24 (−0.42 to −0.06)* | −0.20 (−0.53 to 0.14) | −0.05 |
All groups combined | 2.89 (2.77 to 3.01) | −0.07 (−0.16 to 0.02) | 2.77 (2.66 to 2.89) | −0.11 (−0.20 to −0.01)* | −0.12 (−0.28 to 0.05) | −0.06 |
Mood | ||||||
After exposure (T2) | ||||||
Cisgender male | 2.81 (2.64 to 2.99) | −0.01 (−0.10 to 0.09) | 2.93 (2.77 to 3.09) | 0.10 (0.01 to 0.19)* | 0.12 (−0.12 to 0.35) | 0.04 |
Cisgender female | 2.86 (2.74 to 2.98) | 0.05 (−0.01 to 0.12) | 2.74 (2.62 to 2.87) | 0.04 (−0.03 to 0.10) | −0.12 (−0.28 to 0.05) | −0.06 |
Nonbinary/transgender | 2.61 (2.42 to 2.80) | 0.01 (−0.10 to 0.11) | 2.76 (2.58 to 2.95) | 0.13 (0.03 to 0.23)* | 0.16 (−0.11 to 0.43) | 0.05 |
All groups combined | 2.76 (2.67 to 2.85) | 0.02 (−0.04 to 0.07) | 2.81 (2.72 to 2.91) | 0.09 (0.04 to 0.14)** | 0.05 (−0.08 to 0.19) | 0.04 |
4 weeks later (T3) | ||||||
Cisgender male | 2.84 (2.63 to 3.05) | 0.01 (−0.22 to 0.24) | 3.07 (2.88 to 3.26) | 0.24 (0.03 to 0.45)* | 0.24 (−0.05 to 0.52) | 0.07 |
Cisgender female | 2.98 (2.84 to 3.11) | 0.17 (0.03 to 0.32)* | 2.74 (2.59 to 2.88) | 0.03 (−0.13 to 0.18) | −0.24 (−0.44 to −0.04)* | −0.11 |
Nonbinary/transgender | 2.81 (2.59 to 3.03) | 0.20 (−0.03 to 0.44) | 2.90 (2.68 to 3.11) | 0.26 (0.03 to 0.49)* | 0.09 (−0.22 to 0.40) | 0.03 |
All groups combined | 2.87 (2.76 to 2.98) | 0.13 (0.01 to 0.25)* | 2.90 (2.79 to 3.01) | 0.18 (0.06 to 0.29)* | 0.03 (−0.13 to 0.18) | 0.02 |
p < 0.05; **p < 0.01 (two-tailed).
Comparison of means with baseline with Bonferroni-corrected contrast tests.
Comparison of means for intervention group with the control group with Bonferroni-corrected contrast tests.
CI, confidence interval; MC, mean change; MD, mean difference.
Differences by gender identity
A significant group × time × gender identity interaction was revealed for suicidal ideation (Table 4). Scores were lower for nonbinary/transgender participants in the intervention group than the control group at T2 (MC = −0.06 [95% CI −0.16 to 0.05], p = 0.60; MD = −0.42 [95% CI −0.79 to −0.06], p = 0.02, d = 0.10). The effect was mainly due to an improvement on the subscale future optimism but not maintained at 4-week follow-up (Table 3).
Table 4.
Study variable | Group | Time | Gender identity | Group × time | Group × gender identity | Gender identity × time | Group × gender identity × time | |
---|---|---|---|---|---|---|---|---|
Primary outcome | ||||||||
Suicidal ideation |
F (df1, df2) p |
1.57 (1, 481.34) 0.21 |
1.40 (2, 366.11) 0.25 |
7.47 (2, 481.67) 0.001 |
0.46 (2, 366.16) 0.63 |
2.13 (2, 481.47) 0.12 |
1.27 (4, 366.14) 0.28 |
3.69 (4, 366.14) 0.01 |
Suicidal ideation—future optimism |
F (df1, df2) p |
2.98 (1, 477.64) 0.09 |
0.85 (2, 367.29) 0.43 |
4.62 (2, 477.99) 0.01 |
1.31 (2, 367.34) 0.27 |
0.93 (2, 477.80) 0.40 |
0.88 (4, 367.32) 0.48 |
2.60 (4, 367.33) 0.04 |
Secondary outcomes | ||||||||
Hopelessness |
F (df1, df2) p |
1.48 (1, 480.53) 0.22 |
12.29 (2, 377.97) <0.001 |
10.17 (2, 480.70) <0.001 |
0.22 (2, 378.00) 0.80 |
1.42 (2, 480.72) 0.24 |
4.26 (4, 377.99) 0.002 |
3.40 (4, 378.00) 0.01 |
Mood |
F (df1, df2) p |
0.11 (1, 467.14) 0.74 |
13.29 (2, 363.84) <0.001 |
1.96 (2, 467.52) 0.14 |
2.95 (2, 363.93) 0.05 |
2.59 (2, 467.84) 0.08 |
0.71 (4, 363.93) 0.59 |
2.47 (4, 363.94) 0.04 |
Help-seeking—personal contacts |
F (df1, df2) p |
1.81 (1, 474.47) 0.18 |
18.51 (2, 363.13) <0.001 |
3.77 (2, 474.86) 0.02 |
3.96 (2, 363.22) 0.02 |
2.52 (2, 474.93) 0.08 |
1.04 (4, 363.20) 0.39 |
0.41 (4, 363.21) 0.80 |
Help-seeking—professional help |
F (df1, df2) p |
0.52 (1, 475.14) 0.47 |
0.88 (2, 356.64) 0.42 |
0.56 (2, 475.53) 0.57 |
0.17 (2, 356.69) 0.84 |
0.08 (2, 475.59) 0.92 |
1.70 (4, 356.72) 0.15 |
1.53 (4, 356.73) 0.19 |
Internalized homonegativity |
F (df1, df2) p |
0.02 (1, 472.38) 0.88 |
1.24 (2, 354.93) 0.29 |
11.51 (2, 472.56) <0.001 |
1.40 (2, 354.99) 0.25 |
1.78 (2, 472.61) 0.17 |
1.42 (4, 355.01) 0.23 |
1.78 (4, 355.02) 0.13 |
Identity challenges/negative identity |
F (df1, df2) p |
0.29 (1, 475.80) 0.59 |
1.12 (2, 354.20) 0.33 |
3.01 (2, 475.99) 0.05 |
0.06 (2, 354.22) 0.94 |
0.34 (2, 476.02) 0.72 |
0.97 (4, 354.23) 0.43 |
0.31 (4, 354.23) 0.87 |
Note: F and p values with degrees of freedom (df1 and df2) given in parentheses from linear mixed models representing the change of the respective outcome variable with regard to group (intervention group and control group), time (T1, T2, and T3), gender identity (cisgender male, cisgender female, and nonbinary/transgender), and interactions between these factors controlled for age. Significant p values (<0.05) are marked in bold; the analyses were adjusted for multiple testing using the Bonferroni correction.
The influence of depressive symptoms
A group × time × gender identity × severity of depressive symptoms interaction emerged related to suicidal ideation (Supplementary Data). Nonbinary/transgender participants with severe depressive symptoms appeared to benefit more strongly from the intervention in terms of a short-term reduction of suicidal ideation compared to controls. This comparison was short of statistical significance (Supplementary Table S5).
Sexual orientation
No effect for suicidal ideation was found. For hopelessness, a group × time × sexual orientation interaction was present (Supplementary Table S6). There was a short-term decrease in hopelessness for participants identifying other than homosexual or bisexual (e.g., pansexual, queer) (T2: MC = −0.13 [95% CI −0.25 to −0.01], p = 0.03), although the MD to controls was not significant (MD = −0.12 [95% CI −0.42 to 0.19], p = 0.46, d = −0.03).
Identification with the protagonist
Assignment to the intervention group predicted greater identification with the video protagonist (B = 0.60, 95% CI 0.45 to 0.75, p < 0.001). Higher identification, in turn, was associated with lower suicidal ideation scores at T2 (B = −0.29, 95% CI −0.39 to −0.20, p < 0.001). Due to the indirect effect through identification, participants from the intervention group had suicidal ideation total mean scores that were 0.17 (95% CI −0.26 to −0.10) points lower than the control group (Supplementary Fig. S1).
Discussion
This is the first RCT about the effects of selected videos from the IGBP on LGBTQ+ youth. There was no effect of the videos on the total intervention group regarding suicidal ideation, although an indirect small-sized beneficial effect was observed through identification with the protagonist. Nonbinary/transgender individuals responded with a short-term decrease in suicidal ideation compared to the control group that appeared potentially most pronounced among those with severe depressive symptoms. For participants watching an IGBP video, there was a short-term small-sized improvement in help-seeking intentions compared to controls.
Our findings show some preliminary “proof of concept” that personal narratives of LGBTQ+ individuals targeting LGBTQ+ youth appear safe and might have some beneficial effect on vulnerable youth. The question of safety is relevant because of examples of well-intended messages related to suicide that have yielded unintended harmful effects.40,41 However, the present findings also indicate that the videos did not have any impact on suicidal ideation across all participants, and the effect on help-seeking was small and short-lived.
Some previous studies suggest a protective effect of short media interventions on suicidal ideation, particularly for individuals with some degree of vulnerability to suicide.9,10 Within LGBTQ+ youth, nonbinary/transgender participants are an important risk group,3 which was also reflected in their high depressive symptom scores in this study. However, as this subgroup was small, further replication studies are needed to examine and replicate any effect in larger samples.
Of note, gender diverse people were not featured in either of the videos. Furthermore, their representation was weak across the Austrian and American IGBP videos.16,42,43 A specific tailoring to gender diverse individuals appears crucial to better harness any protective potential.16,44
The finding of this study that videos primarily reduced suicidal ideation through identification is important. It provides a possible pathway to protective media effects on suicidal ideation. Although the variance that was explained by the predictors was small in size, identification appears key for suicide-protective effects, and its relevance was also brought up in focus groups with LGBQ+ youth.16
The effect sizes for help-seeking were small. This is consistent with previous studies in the area.9,10 Overall, small effects would still be meaningful from a public health perspective if the intervention is delivered to a large proportion of the target group.45
As indicated by the absence of change in suicidal ideation across the intervention group, the materials used in the IGBP appear at low risk of triggering increases in suicidal ideation. Their focus is on coping with adversity, and suicide prevention is not explicitly brought up in the videos. The absence of a focus on suicide prevention is a dilemma, because positive effects might be larger if videos addressed explicitly how to cope with suicidal ideation. A more explicit discussion would also address LGBQ+ youth's criticism about the lacking focus on suicidal ideation in the videos.16 Guidelines are available on how to discuss suicidality in a safe way, and future IGBP video makers should be encouraged to use them.46–48
The tested videos showed some potential to improve help-seeking intentions particularly from private sources. Speaking with trusted others is crucial in suicide prevention.49 The featured narratives included a supportive significant other, which might have triggered this effect. These videos, and IGBP videos in general, however, did not address professional help-seeking, which has been criticized by LGBQ+ youth and prevention experts.16,42 An emphasis on professional help-seeking might yield larger and potentially more sustainable effects.
Study limitations
These findings are not generalizable to all IGBP videos. Most IGBP videos do not emphasize the types of adversities youth had to face, or how things got better, which makes any impact unlikely.42 Only short-term effects of one-time exposure to the videos were tested. We did not assess sex assigned at birth or ethnicity; hence, stratified analyses by these variables were not possible.
Furthermore, there was a considerable loss to follow-up in online participants between T2 and T3. It is well established that online studies carry an increased risk of loss to follow-up, and this study needed to swiftly implement an ad-hoc online component to cope with restrictions to movement during the COVID-19 pandemic, which made it impossible to carry out the study on-site only as originally planned.50,51 Personalized strategies to improve retention, such as personal e-mail contact, phone calls, and the provision of monetary incentives, might have increased retention and should be considered for future trials.52,53
Finally, Cronbach's α levels were low for measuring help-seeking intentions. This limitation has been described before. It is found to be a common issue for short scales.26,54
Conclusion
This study indicates some positive potential for narratives of hope and recovery to increase help-seeking and decrease suicidal ideation in some groups of LGBTQ+ youth. The effects, however, were small and short-lived. More planning needs to be done regarding how to discuss suicidal ideation in the videos, increase diversity, and how to best facilitate identification to develop powerful narratives that resonate well with specific LGBTQ+ groups.
Supplementary Material
Acknowledgment
We thank Romeo for their support with recruitment.
Authors' Contributions
T.N., B.T., and M.P. conceptualized the study. S.K. collected the data. Data analysis and interpretation were done by all authors. S.K. drafted the article; T.N., B.T., and M.P. critically revised the article regarding important intellectual content. All authors read and approved the final article.
Availability of Data and Materials
Data from this study can be obtained from the authors upon reasonable request.
Author Disclosure Statement
No competing financial interests exist.
Funding Information
The authors disclosed receipt of the following financial support for the research and authorship, of this article: this research was funded by the Austrian Science Fund (FWF) [project number: P30918-B27, PI: Thomas Niederkrotenthaler].
Supplementary Material
References
- 1. Plöderl M, Tremblay P: Mental health of sexual minorities. A systematic review. Int Rev Psychiatry 2015;27:367–385. [DOI] [PubMed] [Google Scholar]
- 2. D'Augelli AR, Hershberger SL, Pilkington NW: Suicidality patterns and sexual-orientation-related factors among lesbian, gay, and bisexual youths. Suicide Life Threat Behav 2001;31:250–264. [DOI] [PubMed] [Google Scholar]
- 3. Valentine SE, Shipherd JC: A systematic review of social stress and mental health among transgender and gender non-conforming people in the United States. Clin Psychol Rev 2018;66:24–38. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Russell ST, Fish JN: Mental health in lesbian, gay, bisexual, and transgender (LGBT) youth. Annu Rev Clin Psychol 2016;12:465–487. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. It Gets Better: It Gets Better Project. 2010. Available at www.itgetsbetter.org Accessed September 29, 2021.
- 6. Es wird besser Österreich [It gets better Austria]: Es wird besser Österreich [It gets better Austria]. 2016. Available at http://www.eswirdbesser.at Accessed September 29, 2021.
- 7. National Action Alliance for Suicide Prevention: Suicide Attempt Survivors Task Force. The way forward: Pathways to hope, recovery, and wellness with insights from lived experiences. 2014. Available at https://theactionalliance.org/resource/way-forward-pathways-hope-recovery-and-wellness-insights-lived-experience Accessed May 11, 2021.
- 8. Niederkrotenthaler T, Voracek M, Herberth A, et al. : Role of media reports in completed and prevented suicide: Werther v. Papageno effects. Br J Psychiatry 2010;197:234–243. [DOI] [PubMed] [Google Scholar]
- 9. Till B, Tran US, Voracek M, Niederkrotenthaler T: Beneficial and harmful effects of educative suicide prevention websites: Randomised controlled trial exploring Papageno v. Werther effects. Br J Psychiatry 2017;211:109–115. [DOI] [PubMed] [Google Scholar]
- 10. Niederkrotenthaler T, Till B: Effects of suicide awareness materials on individuals with recent suicidal ideation or attempt: Online randomised controlled trial. Br J Psychiatry 2020;217:693–700. [DOI] [PubMed] [Google Scholar]
- 11. Gomillion SC, Giuliano TA: The influence of media role models on gay, lesbian, and bisexual identity. J Homosex 2011;58:330–354. [DOI] [PubMed] [Google Scholar]
- 12. Romeo: Gay dating- chat, meet, love. 2002. Available at www.romeo.com/auth/login Accessed January 20, 2022.
- 13. SoSci Survey GmbH: SoSci Survey—die Lösung für eine professionelle Onlinebefragung [SoSci Survey—the solution for a professional online survey]. Available at www.soscisurvey.de Accessed September 29, 2021.
- 14. Faul F, Erdfelder E, Lang AG, Buchner A: G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 2007;39:175–191. [DOI] [PubMed] [Google Scholar]
- 15. GLIMMPSE: GLIMMPSE—General Linear Mixed Model Power and Sample Size. Available at http://glimmpse.samplesizeshop.org Accessed September 29, 2021.
- 16. Kirchner S, Till B, Plöderl M, Niederkrotenthaler T: Perceptions of LGBQ+ youth and experts of suicide prevention video messages targeting LGBQ+ youth: Qualitative study. BMC Public Health 2020;20:1845. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. HOSI Wien: YouTube: Es wird besser Österreich—Rudi [YouTube: It Gets Better Austria—Rudi]. 2013. Available at https://www.youtube.com/watch?v=gdKb4PKD_0k Accessed September 29, 2021.
- 18. EswirdbesserAT: YouTube: #EsWirdBesser mit Alice Möschl I #Homophobie nein danke [YouTube: #itgetsbetter with Alice Möschl I #homophobia, no thanks]. 2017. Available at https://www.youtube.com/watch?v=n0shGHyczSk Accessed September 29, 2021.
- 19. SoSci Survey GmbH: Random Generator. 2017. Available at https://www.soscisurvey.de/help/doku.php/en:create:questions:random Accessed September 29, 2021.
- 20. Stonewall: Do ask, do tell- Capturing data on sexual orientation and gender identity globally. 2016. Available at https://www.stonewall.org.uk/sites/default/files/do_ask_do_tell_guide_2016.pdf Accessed January 18, 2022.
- 21. The GenIUSS Group: Best practices for asking questions to identify transgender and other gender minority respondents on population-based surveys. 2014. Available at https://williamsinstitute.law.ucla.edu/wp-content/uploads/Survey-Measures-Trans-GenIUSS-Sep-2014.pdf Accessed January 18, 2022.
- 22. Osman A, Downs WR, Kopper BA, et al. : The Reasons for Living Inventory for adolescents (RFL-A): Development and psychometric properties. J Clin Psychol 1998;54:1063–1078. [DOI] [PubMed] [Google Scholar]
- 23. Gutierrez PM, Osman A, Kopper BA, Barrios FX: Why young people do not kill themselves: The reasons for living inventory for adolescents. J Clin Child Adolesc Psychol 2000;29:177–187. [DOI] [PubMed] [Google Scholar]
- 24. Wilson C, Deane F, Ciarrochi J, Rickwood D: Measuring help-seeking intentions: Properties of the General Help-Seeking Questionnaire. Can J Couns 2005;39:15–28. [Google Scholar]
- 25. Kovacs RH: The impact of different suicide prevention messages: A randomized controlled online study [Unpublished master's thesis], Vienna (Austria): University of Vienna, 2019. [Google Scholar]
- 26. Till B, Tran US, Niederkrotenthaler T: The impact of educative news articles about suicide prevention: A randomized controlled trial. Health Commun 2020;36:2022–2029. [DOI] [PubMed] [Google Scholar]
- 27. Mohr JJ, Kendra MS: The Lesbian, Gay & Bisexual Identity Scale (LGBIS). Measurement Instrument Database for the Social Science. Available at www.midss.org Accessed April 29, 2021.
- 28. Mohr JJ, Kendra MS: Revision and extension of a multidimensional measure of sexual minority identity: The lesbian, gay, and bisexual identity scale. J Couns Psychol 2011;58:234–245. [DOI] [PubMed] [Google Scholar]
- 29. Becker P: Skalen für Verlaufsstudien der emotionalen Befindlichkeit [Scales for longitudinal studies of affective state]. Zeitschrift für experimentelle und angewandte Psychologie [Exp Psychol] 1988;35:345–369. [Google Scholar]
- 30. Beck AT, Steer RA: Manual for the Beck Hopelessness Scale. San Antonio, TX: Psychological Corporation, 1988. [Google Scholar]
- 31. Krampen G: Skalen zur Erfassung von Hoffnungslosigkeit (H-Skalen). Deutsche Bearbeitung und Weiterentwicklung der H-Skalen von Aaron T. Beck. [Scales for Assessment of Hopelessness (H-scales). German Adaption and Enhancement of the H-scales by Aaron T. Beck]. Göttingen, Germany: Hogrefe, 1994. [Google Scholar]
- 32. Kroenke K, Spitzer RL, Williams JBW: The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med 2001;16:606–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Cohen J: Defining identification: A theoretical look at the identification of audiences with media characters. Mass Commun Soc 2001;4:245–264. [Google Scholar]
- 34. Kolahi J, Bang H, Park J: Towards a proposal for assessment of blinding success in clinical trials: Up-to-date review. Community Dent Oral Epidemiol 2009;37:477–484. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Gueorguieva R, Krystal JH: Move over ANOVA. Progress in analyzing repeated-measures data and its reflection in papers published in the Archives of General Psychiatry. Arch Gen Psychiatry 2004;61:310–317. [DOI] [PubMed] [Google Scholar]
- 36. Hayes AF: Introduction to Mediation, Moderation, and Conditional Process Analysis, 2nd ed. New York: Guilford Press, 2017. [Google Scholar]
- 37. The World Medical Association: WMA Declaration of Helsinki—Ethical Principles for Medical Research Involving Human Subjects. 2021. Available at https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/ Accessed September 29, 2021.
- 38. German Clinical Trials Register: DRKS—German Clinical Trials Register. Available at https://www.drks.de/drks_web/setLocale_EN.do Accessed September 29, 2021.
- 39. Gupta SK: Intention-to-treat concept: A review. Perspect Clin Res 2011;2:109–112. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Klimes-Dougan B, Yuan C, Lee S, Houri AK: Suicide prevention with adolescents: Considering potential benefits and untoward effects of public service announcements. Crisis 2009;30:128–135. [DOI] [PubMed] [Google Scholar]
- 41. Schmidtke A, Häfner H: The Werther effect after television films: New evidence for an old hypothesis. Psychol Med 1988;18:665–676. [DOI] [PubMed] [Google Scholar]
- 42. Kirchner S, Till B, Plöderl M, Niederkrotenthaler T: “It gets better” videos for suicide prevention in LGBTIQ+ adolescents: Content analysis of German-language videos. Crisis 2020;19:1–8. [DOI] [PubMed] [Google Scholar]
- 43. Craig S, McInroy B, Alaggia R, McCready L: “Like picking up a seed, but you haven't planted it yet”: Queer youth analyze the It Gets Better project. Int J Child Youth Family Stud 2014;5:204–219. [Google Scholar]
- 44. Phillips LM: Offering hope and making attributions through YouTube: An exploratory ethnographic content analysis of the social-change oriented “It Gets Better Project.” Soc Media Soc 2013;2:30–65. [Google Scholar]
- 45. Rose G: Strategy of prevention: Lessons from cardiovascular disease. Br Med J 1981;282:1847–1851. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. National Suicide Prevention Lifeline: Storytelling for Suicide Prevention Checklist. 2018. Available at https://suicidepreventionlifeline.org/storytelling-for-suicide-prevention-checklist/ Accessed March 30, 2021.
- 47. World Health Organisation (WHO): Preventing suicide: A resource for media professionals—update 2017. 2017. Available at https://www.who.int/mental_health/suicide-prevention/resource_booklet_2017/en/ Accessed May 11, 2021.
- 48. World Health Organisation (WHO): Preventing suicide: A resource for filmmakers and others working on stage and screen. 2019. Available at https://www.who.int/publications/i/item/preventing-suicide-a-resource-for-filmmakers-and-others-working-on-stage-and-screen Accessed May 11, 2021.
- 49. Hom MA, Stanley IH, Joiner TE: Evaluating factors and interventions that influence help-seeking and mental health service utilization among suicidal individuals: A review of the literature. Clin Psychol Rev 2015;40:28–39. [DOI] [PubMed] [Google Scholar]
- 50. Dandurand F, Shultz TR, Onishi KH: Comparing online and lab methods in a problem-solving experiment. Behav Res Methods 2008;40:428–434. [DOI] [PubMed] [Google Scholar]
- 51. Jain A, Ross MW: Predictors of drop-out in an internet study of men who have sex with men. Cyberpsychol Behav 2008;11:583–586. [DOI] [PubMed] [Google Scholar]
- 52. Bailey JV, Pavlou M, Copas A, et al. : The sexunzipped trial: Optimizing the design of online randomized controlled trials. J Med Internet Res 2013;15:e278. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53. Gillies K, Kearney A, Keenan C, et al. : Strategies to improve retention in randomised trials. Cochrane Database Syst Rev 2021;3:MR000032. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54. Sijtsma K: On the use, the misuse, and the very limited usefulness of Cronbach's Alpha. Psychometrika 2009;74:107–120. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Data from this study can be obtained from the authors upon reasonable request.