Skip to main content
PMC home page
Internet Interventions logoLink to Internet Interventions
. 2023 Aug 30;34:100663. doi: 10.1016/j.invent.2023.100663

Development and codesign of flourish: A digital suicide prevention intervention for LGBTQ+ youth who have experienced online victimization

Candice Biernesser a,⁎, Emma Win a,b, César Escobar-Viera a, Rosta Farzan c, Morgan Rose a,d, Tina Goldstein a,d
PMCID: PMC10491770  PMID: 37693013

Abstract

Background

LGBTQ+ youth experience disproportionately high rates of online victimization (OV), referring to harmful remarks, images, or behaviors in online settings, which is associated with suicidal risk. Current services have gaps in supporting LGBTQ+ youth facing OV events. To address these gaps, this study aims to develop Flourish, a digital suicide prevention intervention for LGBTQ+ youth who have experienced OV.

Methods

Qualitative interviews were conducted with 20 LGBTQ+ youth with past-year history of OV and lifetime history of suicidality, 11 of their parents, and 10 LGBTQ+-serving professionals. Subsequently, an iterative codesign process was conducted with 22 youth through individual and group design sessions, followed by usability testing. Data were recorded and transcribed. Qualitative interviews were analyzed using a qualitative description approach, and data from design sessions and usability testing were analyzed using rapid qualitative techniques.

Results

Interviews with youth, parents, and professionals suggested preferences for Flourish to be a partially automated, text message intervention leveraging web-based content that is a safe space for LGBTQ+ youth to seek support for OV through education, coping skills, and help-seeking resources. School and mental health services professionals considered the potential for implementing Flourish within youth services settings. Usability testing, assessed through the System Usability Scale, yielded an average rating of 91, indicating excellent perceived usability.

Conclusions

Flourish has potential to be an acceptable intervention to support LGBTQ+ youth following OV. Future steps will include testing the feasibility and efficacy of Flourish and further examining Flourish's potential for implementation within services for LGBTQ+ youth.

Keywords: LGBTQ+, Suicide prevention, Victimization, Social media, Codesign, Text messaging

Highlights

  • •

    LGBTQ+ youth and those in their support systems have clear priorities for prevention after OV.

  • •

    Flourish may be an acceptable digital suicide prevention intervention for LGBTQ+ youth after OV.

  • •

    Flourish could augment school-based and mental health services to extend support for LGBTQ+ youth.

  • •

    Codesign approaches offer insights for the development of interventions for at-risk LGBTQ+ youth.

1. Introduction

Online victimization (OV), disparaging remarks, images, or behaviors that cause harm on an online platform (Tynes et al., 2010), is a growing problem linked to suicide risk among adolescents (Nesi et al., 2021) that disproportionately impacts LGBTQ+ youth (Aboujaoude et al., 2015). LGBTQ+ refers to youth identifying as lesbian, gay, bisexual, transgender, queer, questioning, and sexual orientations other than heterosexual/straight and/or gender identities other than one's sex assigned at birth. LGBTQ+ youth face stressors at individual, interpersonal, community, and societal levels (Hatzenbuehler and Pachankis, 2016). Coined as minority stressors by Meyer (Meyer, 2003), these experiences include exposure to discriminatory or prejudicial events and internalized feelings of rejection, concealment, and homophobia and/or transphobia. OV can extend minority stress to online settings. While social media can afford important benefits for LGBTQ+ youth (e.g., identity exploration, connection, and affirming resources) (Craig et al., 2021), it also poses risks to LGBTQ+ youth who have three times the rate of OV as non-LGBTQ+ youth (Aboujaoude et al., 2015). Akin to other minority stressors, OV is associated with a range of poor mental health outcomes, the most concerning of which are suicidal thoughts and behaviors (Nesi et al., 2021; Kowalski et al., 2014). This is of particular significance for LGBTQ+ youth, who are three times as likely to attempt suicide as non-LGBTQ+ youth (Raifman et al., 2020).

Despite the importance of this problem, existing services such as school-based anti-bullying programs and mental health treatment have clear gaps for LGBTQ+ youth experiencing OV. First, attaining services depends upon youth disclosing OV. Rates of reporting OV are notoriously low among youth (37 %) and even less among LGBTQ+ youth (18 %) (Cooper and Blumenfeld, 2012). Second, OV inherently occurs outside of in-person settings, which reduces services' reach. Schools in particular face ethical and legal complexities regarding involvement with OV that occurs off school grounds (Hinduja and Patchin, 2011). Third, OV often occurs outside of times services are available. Yet, intervention is essential proximally to OV events when youth are known to experience distress (Birkett et al., 2015) that can contribute to suicidal thoughts (Quintana-Orts et al., 2022). Fourth, extant services are limited by a lack of programming designed and evaluated for LGBTQ+ youth (Espelage and Rao, 2013) and a lack of LGBTQ+ competent providers (Bishop et al., 2022), contributing youth having hesitance to seek care and for one-size-fits-all programming to be insufficient to meet LGBTQ+ youth's needs (Marshall, 2016).

1.1. Digital Mental Health Interventions (DMHIs) for online victimized LGBTQ+ youth

When DMHIs are designed to be responsive to LGBTQ+ populations, they can address disparities (Cohen et al., 2022) through increasing the availability and accessibility of affirming services (Whaibeh et al., 2020). DMHIs can be triggered at times of high need (Coppersmith et al., 2022), offering a potential mechanism for real-time response to OV. Further, DMHIs have shown promise for self-management of mental health (Witt et al., 2017) through strengthening capacity for autonomous engagement in coping skills. Notably, self-management is preferred by LGBTQ+ over in-person services, due to concerns toward stigma from providers (McDermott et al., 2018). Further, DMHIs can improve motivation to seek help for mental health problems (Evans-Lacko et al., 2022) This may be especially beneficial for LGBTQ+ youth, who often go online for health information and to discuss their identity and health (Fish et al., 2022). These data suggest a DMHI that responds to LGBTQ+ youth's OV experiences could be beneficial. However, careful consideration of the sensitive nature of OV and the vulnerability of this population is essential.

Codesign is a best practice for DMHI design that underpins successful adoption and implementation and extends reach to marginalized populations (Bergin et al., 2020; Schueller et al., 2019). Codesign of DMHIs centers lived experience through iterative engagement with end users, referring to the intended recipients of an intervention, and additionally considers the lived experiences of those within end users' support systems (Bevan Jones et al., 2020). Engagement with end-users begins with building trust and awareness before the project starts by setting structures for shared decision-making and co‑leadership of the design process (Tindall et al., 2021) and continues throughout intervention development and implementation (Hodson et al., 2019). Culturally sensitive codesign involves engaging with diversity (Bevan Jones et al., 2020), referring to designing in a way that is responsive to end-users' culture, mindful toward social environments impacting marginalized populations (Dombrowski et al., 2016), and focused not only on end-users' needs but also their strengths (Wong-Villacres et al., 2020). In doing so, power imbalances between researchers and end-users are recognized and addressed with cultural humility (Tindall et al., 2021; Hodson et al., 2019). For codesign of DMHIs, end-users' ideologies toward technology should also be considered, in recognition that one's culture and experiences can influence comfort with using technology for health-related purposes (Schueller et al., 2019; Bevan Jones et al., 2020).

1.2. Objectives

This study aimed to develop a digital intervention, Flourish, through codesign with LGBTQ+ youth. Flourish was designed to bolster adaptive coping skills and reduce suicide risk among LGBTQ+ youth following OV experiences through three treatment targets based on best practices for suicide prevention interventions (Glenn et al., 2015; Brent et al., 2013). First, Flourish delivers psychoeducation to bolster social problem-solving, referring to processes for identifying and enacting solutions to problematic social situations (Adrian et al., 2011), to support adolescents' capacity for effective response to OV. Second, in recognition that distress often follows OV events, Flourish provides skills in distress tolerance, which refers to one's capacity to withstand distressing emotional states (Leyro et al., 2010). Third, based on principles of motivational interviewing, an evidence-based approach for behavior change (Miller and Rollnick, 2013), Flourish leverages brief motivational techniques to improve engagement in help-seeking following OV. Toward the development of Flourish, this study involved qualitative interviews and an iterative design process with LGBTQ+ adolescents with history of suicidality who have experienced OV and those in their support systems.

2. Methods

This study utilized the Discover, Design, Build, and Test (DDBT) model (Lyon et al., 2019), which provides a structure for DMHI development that features codesign with end-users. DDBT brings together principles of human centered design (HCD) through incorporating iterative design and development, participatory action research by centering partnerships with end-users to bring about meaningful change with them rather than for them, and implementation science through a focus on understanding how best to implement and sustain a DMHI. This manuscript reviews two of the DDBT model's three phases (see Fig. 1): 1) “Discover,” which focuses on exploring parameters for an acceptable DMHI, consisted of qualitative interviews with youth and their support system and 2) “Design and Build” involved engaging in co-design and iterative development with end-users. Future work will include the third phase, “Test,” through evaluation of Flourish's real-world feasibility.

Fig. 1.

Fig. 1

Open in a new tab

Discover, design, build and test model (developed by Lyon et al., 2019).

2.1. Participants and procedures

This study recruited a purposive sample of LGBTQ+ youth (N = 33), parents (N = 11) and LGBTQ+-serving professionals (N = 10). Of the youth who participated, 20 completed interviews in the Discover Phase. The Design and Build Phase included 22 youth, 9 of whom also participated in the Discover Phase. Parents and professionals only participated during the Discover Phase. Inclusion criteria for youth were ages 13–18, self-identified as LGBTQ+, had one or more past-year OV experiences determined via items modified from a cybervictimization measure developed by Hinduja and Patchin (Hinduja and Patchin, 2010), and a lifetime history of suicidal thoughts and/or behaviors measured through items from the Ask Suicide-Screening Questions (Horowitz et al., 2020). Parents of youth in the study who knew that their child experienced OV were invited to participate, regardless of their knowledge of their child's LGBTQ+ identity. Professionals who worked predominantly with LGBTQ+ youth and were experiencing engaging with youth about OV were invited to participate. Professionals were from schools (N = 4), outpatient mental health (N = 3), and LGBTQ+ behavioral health services (N = 3).

The majority of youth (27 out of 33) were recruited through advertisements on Instagram and Facebook from which they completed an online survey to indicate potential eligibility. The remaining youth (6 out of 33) were identified from a clinic for youth in treatment for anxiety, depression, or suicide risk in Pittsburgh, PA and a research participant registry at the University of Pittsburgh. Recruitment procedures were determined through consultation with recruitment and regulatory specialists within the University of Pittsburgh's IRB and Center for Clinical and Translational Science. Procedures included youth completing a survey indicating potential eligibility, after which the second author contacted their parent by phone to confirm eligibility. To maintain confidentiality and avoid outing youth to their parents, criteria related to LGBTQ+ identity were determined by the youth's report alone. Informed consent was provided by parents and assent by youth. Parents who met eligibility criteria were invited to complete a research visit on a separate Zoom call. Professionals within schools and mental health services in urban and rural settings within Western Pennsylvania and Northern West Virginia were invited to participate via an e-mail from the primary investigator. All participants were compensated, and the University of Pittsburgh Institutional Review Board approved this study.

2.2. Data collection

Data collection for the Discover Phase occurred between February 2022 and May 2022. Adolescents, parents, and professionals completed qualitative interviews exploring their preferences and priorities for an intervention that would support LGBTQ+ youth's mental health and reduce suicide risk following OV. Professionals also considered the potential to implement such a program within youth services. Guides were used to facilitate semi-structured discussion (see Appendix), while interviewers also remained open to emergent topics. Participants were enrolled until saturation was reached, referring to the point when adequate information is gathered and no new information is discovered to draw necessary conclusions (Morse, 1995), regarding key priorities for Flourish.

Data collection for the Design and Build Phase occurred between July 2022 and January 2023 (see Fig. 2). All sessions were conducted remotely on Mural, a digital whiteboard space. Prior to design sessions, youth were orientated to codesign. A brief research summary from the Discover Phase was used to prompt discussion about the need for Flourish. In doing so, power imbalances were addressed through open dialogue about the ways in which existing programming had gaps for LGBTQ+ youth, who rarely had a voice in program development. Structures and expectation were set for adolescents' involvement in leadership and decision-making. Youth were empowered to make decisions toward the design of Flourish, including its functionality, aesthetics, and voice (referring to Flourish's tone and person of speech) in collaboration with researchers who guided incorporation of best practices in addressing OV and management of suicide risk. Additionally, researchers facilitated youth setting ground rules for inclusive design processes e.g., maintaining privacy and respect and using a “yes and” approach, referencing a style of communication that builds upon other's ideas.

Fig. 2.

Fig. 2

Open in a new tab

Flow of design sessions and HCD methods.

First, two small group (N = 3–4 each) design sessions were held to build consensus around features and content for Flourish. These sessions began with creating persona profiles, fictional characters who experienced OV and came to Flourish for support. Youth spent 15 min creating persona profiles using a worksheet as a guide, which was modified from a version developed by the LUMA Institute (LUMA Institute, 2023). Following this exercise, youth had the option to refer to the persona they created in subsequent design tasks, rather than their personal OV experiences, to preserve their privacy. Next, adolescents engaged in storyboarding, a participatory design technique that allows end-users to imagine how a DMHI would act and function. Adolescents spent 20 min as a group creating storylines based on two prompts, in which youth were asked to consider how Flourish would respond following an OV event for a teen who needed to cope with stress and seek help, respectively. Researchers stayed present with audio and video turned off during this time to empower youth to make autonomous decisions about the design of storyboards.

Second, based on the feedback from group sessions, researchers created low-fidelity prototypes of Flourish, which were then evaluated by youth. Individual sessions were chosen to mitigate bias (e.g., social desirability or group think) (Nyumba et al., 2018) that could inhibit honest feedback. Participants spent on average 64 min offering line-by-line feedback on Flourish's content and language by engaging in think aloud sessions, a HCD method on which participants verbalize their thoughts on what they are viewing aloud. During this process, researchers reviewed ways in which youth feedback was incorporated into low-fidelity prototypes to increase transparency in the shared decision-making process and trust that youth voice was valued.

The Design and Build Phase ended with usability testing, an evaluation of Flourish, with a new group of 10 youth. High-fidelity prototypes of Flourish were created, based on the intervention's treatment targets, in which Flourish was simulated to support youth in help-seeking, engaging in coping skills, or problem-solving after an OV event. Adolescents were randomly selected to review 2 of the 3 prototypes. Adolescents evaluated prototypes quantitatively using the System Usability Scale (SUS) (Brooke, 1995), a 10-item questionnaire assessing perceived usability (e.g., ease of use and confidence in use), and qualitatively through openly elaborating on responses to the SUS. SUS scores range from 0 to 100. A score of 68, considered “above average,” was preset as a minimum threshold for adequate usability.

2.3. Analysis

For the Discover Phase, interviews were audiotaped, transcribed, and coded using NVivo, Version 1.6.1. Data were analyzed using a qualitative description approach (Sandelowski, 2000; Sandelowski, 2010). All transcripts were reviewed independently and coded by the second and fourth authors, using a codebook developed by the first author based on the interview guide and revised to include emergent codes. The codebook included codes based on constructs from the Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2022). Selected CFIR constructs were chosen to provide initial indicators of where and how Flourish could be implemented within youth services. Exploration focused on professionals' perceptions toward characteristics of Flourish (e.g., its alignment with needs in the field) and their work setting (e.g., fit with workflow), considerations that can influence capacity for implementation and scalability (Schueller and Torous, 2020). Coding discrepancies were reviewed in consensus meetings among the research team. Finalized analyses informed the Design and Build phase.

For the Design and Build phase, data were analyzed using rapid qualitative analysis techniques (Nevedal et al., 2021), which can produce contextually rich information while being time efficient, allowing for rapid iteration during intervention development. All design sessions were audio recorded. Group design sessions were transcribed verbatim, to account for fruitful dynamic group discussion that informed analyses. Individual design sessions were summarized directly from audio recordings, an approach that has shown success for rapid identification of themes (Neal et al., 2015). The first author created templates using domains from interview guides, which the fourth author used to summarize the data and then compile it within a matrix organized by domain and participant. The first author subsequently analyzed and interpreted the data. Findings were discussed among the research team and used to guide intervention development.

3. Results

3.1. Participant characteristics

Youth's characteristics across both study phases are shown in Table 1. Youth ranged in age between 13 and 18 (on average 15.7 years old), 42.4 % identified as being from a racial and/or ethnic minoritized background, 30.3 % resided in a non-urban county, and two-thirds reported having a disability. LGBTQ+ identities were distributed as 72.7 % transgender/gender-diverse, 97.0 % with sexual orientations other than straight, and 70.0 % who identified with sexual orientations other than straight and were transgender/gender-diverse. Parents ranged in age between 42 and 62 (on average 48 years old), 70 % identified as female, and 40 % were from racial and/or ethnic minoritized backgrounds. Professionals ranged in age between 34 and 55 (on average 43 years old), 70 % identified as female, and none from racial or ethnic minoritized backgrounds. One parent and one professional identified as non-binary.

Table 1.

Adolescent characteristics, N = 33.

Age (years; mean, standard deviation, range) 15.7 (±1.3, 13–18)
Sex assigned at birth
 Female, N (%) 31 (93.9 %)
 Male, N (%) 2 (6.1 %)
Gender Identity
 Nonbinary, N (%) 12 (36.4 %)
 Female, N (%) 8 (24.2 %)
 Male, N (%) 8 (24.2 %)
 Other, N (%)a 5 (15.2 %)
Sexual orientation
 Gay/Lesbian, N (%) 8 (24.2 %)
 Bisexual, N (%) 7 (21.2 %)
 Pansexual, N (%) 6 (18.2 %)
 Queer, N (%) 5 (15.2 %)
 Other, N (%)b 7 (21.2 %)
Ethnicity
 Hispanic, N (%) 6 (18.2 %)
 Non-Hispanic, N (%) 27 (81.8 %)
Race
 White, N (%) 22 (66.7 %)
 Multi-racial, N (%)c 5 (15.2 %)
 Asian, N (%) 3 (9.1 %)
 Black, N (%) 2 (6.1 %)
 American Indian or Alaska Native, N (%) 1 (3.0 %)
Geographic Location
 Population of ≥1,000,000, N (%) 18 (54.5 %)
 Population of 250,000–999,999, N (%) 5 (15.2 %)
 Population < 249,999, N (%) 10 (30.3 %)
Disability
 Mental health-related disability, N (%) 12 (36.4 %)
 Physical health-related disability, N (%) 4 (12.1 %)
 Mental and physical health-related disabilities, N (%) 3 (9.1 %)
 Undisclosed disability, N (%) 3 (9.1 %)
 No self-reported disability, N (%) 11 (33.3 %)
Open in a new tab
a

Androgynous (N = 1), genderqueer (N = 1), genderfluid (N = 1), unsure (N = 1), and female /nonbinary (N = 1).

b

Omnisexual (N = 2), asexual (N = 2), straight (N = 1), unlabelled (N = 1), and trixic (N = 1).

c

American Indian or Alaska Native and White (N = 2), Asian and Hawaiian Native or Other Pacific Islander (N = 1), Black and White (N = 1), American Indian or Alaska Native, White, and Black (N = 1).

3.2. Discover phase: interview results

Across participant groups, interviews included discussion about what support adolescents needed following OV experiences and preferences for Flourish's approach and functioning. Specifically, all participants were asked about their comfort level with Flourish being automated, and youth were asked about their preferences toward Flourish's mode of delivery.

3.2.1. Intervention priorities

Youth, parents, and professionals identified priorities for Flourish, which can be found ordered by frequency of endorsement within Table 2.

Table 2.

Perceived Priorities for Flourish.

Priorities Adolescents' (N = 20), Parents' (N = 11), and Professionals' (N = 10) Perspectives
#1: Strategies for social support and help-seeking Affirming contacts and resources, appropriate for youth with varying levels of parental support, were considered highly important by parents and professionals (N = 19). Aligned with others' thoughts, a LGBTQ+-focused psychologist called for “multiple layers of support” following OV.

Youth wanted immediate and flexible options for help-seeking, e.g., from peers, parents, teachers, therapists, or crisis hotlines (N = 15).
“If you're able to instantly go to it and talk to someone…see if you need distraction or help or if you need to be transferred to a suicide hotline or something.” -17-year-old pansexual, nonbinary person
#2: Coping skills to reduce stress following OV As echoed by other parents and professionals, one mother stated, it is “very important” to “counteract stress” in the moments after OV. They found healthy coping strategies to help youth regulate their emotions critical (N = 15).

Youth wanted brief, uplifting coping skills to reduce stress (N = 10).
“It's hard to think of automatic, healthy coping skills on the spot when something bad happens online.” – 14-year-old gay, nonbinary person
#3: A safe space that is accepting and private Parents and professionals advised creating a safe space for youth through validation, open dialogue, allyship, and assuring confidentiality (N = 11). As one mother described, a safe space could offer “something positive right off the bat,” that says, “I do matter.”

Youth wanted a space they could go to that felt comfortable, validating of their identity, and discrete. (N = 11)
“If there was a place where they could go and feel safer, I'd also suggest the app be discreet. That way, parents can't immediate fish out that this is a gay site.” -15-year-old lesbian and asexual, nonbinary person
#4: Education to aid response to OV Parents and professionals advocated for providing youth with education on how to respond to OV in a way that honors their autonomy (N = 10). As one middle school counselor explained, education could help youth define what is best of them by “giving them the opportunity to really think through next steps and what they could be.”

Youth described need for space to process and understand OV's seriousness and how to respond (N = 9).
“Maybe a way to help quantify the interaction, if they're not sure if it's bullying.” -17-year-old gay male
Open in a new tab

3.2.2. Implementation

When asked about where Flourish could be implemented, professionals identified a range of settings including schools, outpatient mental healthcare, pediatrics, and LGBTQ+ specialty services, the majority of whom pointed to schools and outpatient mental healthcare as top choices (N = 8). Table 3 highlights professional's thoughts regarding the potential to implement Flourish within youth services, organized by select constructs from CFIR.

Table 3.

Professionals' Perspectives on Implementation of Flourish in Youth Services.

Implementation domain and definition Professionals' perspectives (N = 10)
Relative Advantage - The degree to which an innovation is seen as better than the program it replaces Flourish's approach of offering accessible, in-the-moment support was thought to expand professionals' toolboxes for serving LGBTQ+ youth (N = 5) and offer advantage to extant services, which often lack cultural competence for LGBTQ+ youth (N = 8) and capacity to address OV (N = 5). Some (N = 2), however, worried about liability regarding need to report severe OV events, e.g., as maltreatment.
Adaptability - The innovation can be modified/refined to fit the context Professionals imagined Flourish could be adaptable to be flexibly disseminated across a range of youth services contexts (N = 5) and could tailor its delivery to optimally meet adolescents' needs (N = 5), e.g., modifying levels of parental involvement based on developmental context.
Compatibility – Consistency of the innovation with values/needs Professionals universally concurred with the target population for Flourish: LGBTQ+ and other minoritized youth (N = 10).
Complexity - How difficult the innovation is to use and implement Professionals who commented on complexity (N = 7) thought Flourish could be widely disseminated among teens, who are comfortable with technology-based programs, but some (N = 3) warned about complexities of implementation in schools due to shortages of trained mental health staff and policies that sometimes prohibit phone usage during school hours.
Infrastructure – Available resources and organization alignment with the innovation School professionals (N = 4) described how Flourish could be integrated within existing school-wide workflows for responding to OV. Mental health professionals (N = 6) thought Flourish could augment care by extending coping skills for managing OV-related stressors outside of therapy sessions.
Open in a new tab

3.2.3. Type of intervention

Participants were asked to describe their prior knowledge of automated interventions and preferences toward automation for Flourish on a spectrum ranging from no automation, i.e., a phone call with a counselor, to full automation, i.e., artificial intelligence. Youth expressed familiarity with automated health interventions, some of whom had positive experiences interfacing with chatbots prior to reaching a person on a crisis line. Parents' and professionals' knowledge of automation varied. Some parents described points of reference stemming from customer service chatbots that they felt were poorly designed, whereas professionals' knowledge related to automation within healthcare and/or schools.

Youth spoke positively toward automation. One adolescent preferred to have no automation, whereas all others preferred a fully (N = 4) or partially (N = 15) automated intervention. Those preferring partial automation imagined Flourish would deliver education and coping skills in an automated fashion but connect to a human when necessary. Youth felt this would combine the strengths of human interactions, e.g., validation, listening, and capacity to respond to complex situations, with the strengths of automation. Youth identified benefits of automation as the potential for help-seeking without concern about judgment or burdening others and skill development in self-management of their mental health.

Parents and professionals perceived weaknesses of automation, i.e., the capacity for frustration from unexpected responses and for communication to feel “cold and impersonal,” while many recognized that automation was sometimes necessary to extend human capacity for response. Two parents and professionals, respectively, advocated for no automation, whereas the majority of parents (N = 8) and professionals (N = 8) expressed a partially automated intervention could be acceptable if it triggered response from a trusted adult when needed and was programmed with a “personal touch” that left teens feeling “heard,” “affirmed,” and “supported.”

3.2.4. Mode of delivery

When youth were asked to consider how they preferred Flourish to be delivered, most either decisively preferred text messaging (N = 9) or were ambivalent between text messaging and another platform (N = 7). Others preferred an app (N = 2), direct messaging (N = 1), or multiple platforms (N = 1). Texting was viewed as easy to use, accessible, and discrete in comparison to direct messaging that is monitored by social media companies. Youth also appreciated that texting was separate from social media, where OV often occurs.

3.3. Design and build phase: group and individual sessions

Group design sessions began with youth creating persona profiles. They generated characters who experienced OV and had a variety of needs, i.e., for social support, advice on how to respond to OV, and ways to cope with stress. Next, adolescents created storyboards. Storyboards within both groups began with an adolescent launching Flourish and the intervention responding by gauging the adolescents' needs, including an option to reach out for help immediately if necessary. Both groups imagined Flourish would respond with personalized resources, such as a preferred list of coping skills or crisis contacts, and would be followed by a check-in, evaluating if the content improved their mood. After immediate needs were met, one group imagined offering education and goalsetting to prevent OV from reoccurring. Both groups, without prompting, wrote storyboards with messages in the first person, e.g., “How can I help?” Additionally, one group imagining Flourish would communicate through a friendly character to offset the seriousness of conversations about OV. An example persona profile and storyboard are shown in Fig. 3.

Fig. 3.

Fig. 3

Open in a new tab

Examples of persona profile and storyboard from group design sessions.

In subsequent individual sessions, youth reviewed low-fidelity prototypes created by the research team based on results of group design sessions. During think aloud sessions, youth expressed the overall content within the prototypes was aligned with their expectations, though they recommended changes to Flourish's tone. They desired Flourish to communicate in a way that was neither clinical and cold nor overly informal, likening their preferred tone to that of a trusted, supportive adult. They also advocated for Flourish to incorporate a variety of types of content to make interactions more dynamic, e.g., infographics, animations, images, uplifting quotes, and links to video and audio-based content. The research team reviewed their suggestions and revised the language and content before developing high-fidelity prototypes.

3.4. Design and build phase: usability testing

During usability testing, 10 new adolescents to the study reviewed high-fidelity prototypes of Flourish (see Fig. 4). Adolescents reviewed standardized back-and-forth text messaging conversations between a teen who experienced OV and a friendly character named Sprout that led them in identifying what kind of support they wanted (e.g., help with engaging in help-seeking, coping skills, or problem-solving after OV events) and ended with an evaluation of the helpfulness of the interaction. Adolescents had control of the prototype throughout end-to-end interactions by taking remote control of a shared screen that appeared as text messaging on a phone and clicked through the prototypes at their own pace. After reviewing prototypes, adolescents rated Flourish using the SUS. Total SUS ratings ranged from 85 to 100 and had a mean of 91. Using the SUS's scoring metrics, a score of 91 can be interpreted with a grade of A (on a scale from A to F) and adjective rating of Excellent (on a scale from Excellent to Awful). Youth then provided open feedback to contextualize their response on this measure. Themes from these interviews and subsequent actions taken by the research team before finalizing Flourish are shown in Table 4.

Fig. 4.

Fig. 4

Open in a new tab

Screenshots from the flourish prototype.

Table 4.

Summary of usability testing results.

Usability domain Themes from youth interviews
(N = 10)		 Action taken in response to youth feedback (if any)
Interest in Use All youth expressed interest in using Flourish for these primary reasons:
  • -

    Flourish's tone, which was described as positive (N = 3), caring (N = 2), empathetic (N = 3), and authentic/relatable (N = 2)

  • -

    the helpfulness (N = 5) and personalization (N = 3) of its coping skills

its capacity to offer a comfortable way to seek support (N = 3) that is always available (N = 2)		 N/A
Complexity to Use All youth thought Flourish would be easy to use and understand (N = 10). While the majority thought Flourish would be broadly accessible, some thought youth who were younger (N = 1) or who had attention-related difficulties (N = 1) may need support. Reading levels and burden were reduced.
Organization of Content Participants described Flourish's organization to be easy to use and logical (N = 4) and the content to be varied and accurate to LGBTQ+ youth's experience (N = 4). Others thought its navigation (N = 2) and tone (N = 2) could be improved in specific areas to improve clarity and responsiveness. Specific changes recommended by youth regarding navigation and tone of responses were implemented.
Ease to Learn All youth thought it would be easy to learn to use Flourish (N = 10), noting that adolescents were familiar with this type of technology. Some (N = 2) thought instructions would help to improve adolescents' understanding. A brief introduction to Flourish was created to aid understanding.
Confidence in Independent Use All youth thought Flourish would be easy to use without a parent (N = 10). N/A
Open in a new tab

4. Discussion

This manuscript delineates the process of codesign and development of Flourish, a digital intervention that aims to improve coping after OV events and reduce suicide risk among LGBTQ+ youth. Qualitative interviews with LGBTQ+ youth, their parents, and LGBTQ+-serving professionals identified priorities for Flourish, focusing on the need for a validating, affirming digital environment in which education, coping skills, and support can be delivered while maintaining adolescents' privacy. Professionals recognized gaps in existing care models for LGBTQ+ youth who have experienced OV and suggested potential for Flourish to augment school and mental health services. Youth expressed interest in Flourish being a partially automated, text message intervention that leveraged a range of dynamic web-based content. Subsequent individual and group codesign sessions established the content and functionality of Flourish. Final testing revealed Flourish to have excellent perceived usability, exceeding thresholds necessary for further testing. Overall, these data support Flourish as an acceptable intervention for LGBTQ+ youth who have experienced OV that could augment existing care in schools and mental health services with potential to increase access to supportive care for a marginalized and underserved group.

This study's approach, which leveraged best practices for participatory research with marginalized youth and ethical consultation to maximize participant safety and privacy, yielded important takeaways, elucidated in the following five points. These insights could inform approaches to codesigning digital interventions for at-risk LGBTQ+ youth with an eye toward implementation within youth services.

  • 1.

    Establishing trust through empathetic listening to LGBTQ+ youth's experiences is paramount. In line with prior work (Hodson et al., 2019), we found listening to adolescents' experiences to be an essential first step to codesign that facilitated their comfort in offering open, honest feedback.

  • 2.

    Representation of youth with multiple minoritized identities is essential. Inclusivity aided identification of priorities for Flourish and for gathering realistic and accurate feedback about representation of intersectional identities within Flourish's content.

  • 3.

    Avoiding outing adolescents' LGBTQ+ identities to their parents is critical and should be paired with procedures for safety management. Our approach combined engaging parents in consent and discussion of safety concerns, in keeping with best practices for suicide risk management (Pettit et al., 2018), with procedures to avoid outing adolescents' LGBTQ+ identity to their parents by omitting our focus on LGBTQ+ youth within all study-related materials and discussions with parents. Future codesign efforts with at-risk LGBTQ+ youth should navigate parental involvement carefully.

  • 4.

    Adolescents need options to comfortably communicate about their experiences in groups. Creating persona profiles gave youth the option to decide how or if they wished to discuss their personal experiences or, instead, to refer to a fictional character. Flexible approaches such as this could facilitate comfort for youth to share on their own terms.

  • 5.

    Engaging LGBTQ+ youth's support systems can shed light on implementation environments. Parents and LGBTQ+-serving professionals within this study offered valuable context for how and where Flourish could be best implemented.

Regarding limitations, this study included a small, purposive sample. While ideal for codesign with end-users, generalizability to a larger population of LGBTQ+ youth is limited. Further, youth were assured of procedures to avoid outing their identities; however, it is possible those not out to their parents may have opted not to participate, which could limit the interpretation of findings. Also, parents were aware their child had experienced OV, and professionals worked predominantly with LGBTQ+ youth. Their opinions may not be representative of parents without knowledge of their child's OV experiences and professionals less experienced in working with LGBTQ+ youth. Other limitations may exist regarding study design and methods. Though some adolescents participated in design sessions once and others repeatedly, the study was not designed to compare their perceptions toward Flourish. Finally, quantitative evaluation during usability testing was limited to the SUS, often considered a “quick and dirty” tool (Brooke, 1995). Usability testing was designed to maximize potential for feedback responsively to the adolescent study population, two-thirds of whom reported having a disability. Thus, the SUS was chosen as a brief measure with limited cognitive burden that has been validated among youth samples and supplemented by qualitative data collection gathered by an interviewer experienced in working with youth with varying intellectual abilities.

5. Conclusion

This study's intervention development process, which actively engaged LGBTQ+ youth through codesign, suggests Flourish is an acceptable digital suicide prevention intervention for LGBTQ+ youth who have experienced OV. This process brought to light considerations that could aid future development of digital interventions for at-risk LGBTQ+ youth. Next steps will include testing the feasibility and efficacy of Flourish through controlled trials and examining effectiveness and implementation outcomes within community systems serving LGBTQ+ youth (e.g., schools, mental healthcare, and crisis networks).

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Acknowledgements

We are thankful to the adolescents, parents, and professionals who participated in this study without whom this work would not be possible.

Funding information

Drs. Biernesser, Escobar-Viera, and Goldstein received funding for this work from American Foundation for Suicide Prevention grant (YIG-0-021-20) and a NIMH ALACRITY Center grant (P50 MH115838). Dr. Biernesser received funding from NIMH through a postdoctoral training grant (T32 MH018951) and career development award (K23 MH131759). Additionally, this project received support from the Clinical Translational Science Institute at the University of Pittsburgh, which is funded by the NIH (UL1 TR001857).

Footnotes

Appendix A

Supplementary data to this article can be found online at https://doi.org/10.1016/j.invent.2023.100663.

Appendix A. Supplementary data

Supplementary material

mmc1.pdf (324.4KB, pdf)

References

  1. Aboujaoude E., Savage M.W., Starcevic V., Salame W.O. Cyberbullying: review of an old problem gone viral. J. Adolesc. Health. 2015;57(1):10–18. doi: 10.1016/j.jadohealth.2015.04.011. [DOI] [PubMed] [Google Scholar]
  2. Adrian M., Lyon A., Oti R., Tininenko J. In: Encylopedia of Child Behavior and Development. Goldstein S., Naglieri J., editors. Springer; Boston, MA: 2011. Social problem solving. [DOI] [Google Scholar]
  3. Bergin A.D., Vallejos E.P., Davies E.B., Daley D., Ford T., Harold G., Hetrick S., Kidner M., Long Y., Merry S., Morriss R., Sayal K., Sonuga-Barke E., Robinson J., Torous J., Hollis C. Preventive digital mental health interventions for children and young people: a review of the design and reporting of research. NPJ Digit. Med. 2020;3:133. doi: 10.1038/s41746-020-00339-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bevan Jones R., Stallard P., Agha S.S., Rice S., Werner-Seidler A., Stasiak K., Kahn J., Simpson S.A., Alvarez-Jimenez M., Rice F., Evans R., Merry S. Practitioner review: co-design of digital mental health technologies with children and young people. J. Child Psychol. Psychiatry. 2020;61(8):928–940. doi: 10.1111/jcpp.13258. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Birkett M., Newcomb M.E., Mustanski B. Does it get better? A longitudinal analysis of psychological distress and victimization in lesbian, gay, bisexual, transgender, and questioning youth. J. Adolesc. Health. 2015;56(3):280–285. doi: 10.1016/j.jadohealth.2014.10.275. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bishop J., Crisp D.A., Scholz B. A systematic review to determine how service provider practises impact effective service provision to lesbian, gay and bisexual consumers in a mental health setting. Clin. Psychol. Psychother. 2022;29(3):874–894. doi: 10.1002/cpp.2699. [DOI] [PubMed] [Google Scholar]
  7. Brent D.A., McMakin D.L., Kennard B.D., Goldstein T.R., Mayes T.L., Douaihy A.B. Protecting adolescents from self-harm: a critical review of intervention studies. JAACAP. 2013;52(12):1260–1271. doi: 10.1016/j.jaac.2013.09.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Brooke J. In: Usability Evaluation in Industry. Jordan P.W., Thomas B., IL McClelland, Weerdmeester B., editors. 1995. SUS: a quick and dirty usability scale; pp. 189–194. [Google Scholar]
  9. Cohen J.M., Feinstein B.A., Fox K. The promise of digital mental health for LGBTQ+ youths. Psychiatr. Serv. 2022;73(9):1069–1072. doi: 10.1176/appi.ps.202100569. [DOI] [PubMed] [Google Scholar]
  10. Cooper R., Blumenfeld W. Responses to cyberbullying: a descriptive analysis of the frequency of and impact on LGBT and allied youth. J. LGBT Youth. 2012;9:153–177. doi: 10.1080/19361653.2011.649616. [DOI] [Google Scholar]
  11. Coppersmith D.D.L., Dempsey W., Kleiman E.M., Bentley K.H., Murphy S.A., Nock M.K. Just-in-time adaptive interventions for suicide prevention: promise, challenges, and future directions. Psychiatry. 2022;85(4):317–333. doi: 10.1080/00332747.2022.2092828. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Craig S.L., Eaton A.D., McInroy L.B., Leung V.W.Y., Krishnan S. Can social media participation enhance LGBTQ+ youth well-being? Development of the social media benefits scale. Soc. Med. Soc. 2021;7(1):1–13. doi: 10.1177/205630512198. [DOI] [Google Scholar]
  13. Damschroder L.J., Reardon C.M., Opra Widerquist M.A., Lowery J. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR outcomes addendum. Implement. Sci. 2022;17(1):7. doi: 10.1186/s13012-021-01181-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Dombrowski L., Harmon E., Fox S. Designing Interactive Systems; Brisbane, Australia: 2016. Social Justice-oriented Interaction Design: Outlining Key Design Strategies and Commitments. [Google Scholar]
  15. Espelage D., Rao M. In: Creating Safe and Supportive Learning Environments. Fisher E., Komosa-Hawkins K., editors. Routledge; New York, NY: 2013. Safe schools: Prevention and intervention for bullying and harassment; pp. 140–155. [Google Scholar]
  16. Evans-Lacko S., Hahn J.S., Peter L.J., Schomerus G. The impact of digital interventions on help-seeking behaviour for mental health problems: a systematic literature review. Curr. Opin. Psychiatry. 2022;35(3):207–218. doi: 10.1097/YCO.0000000000000788. [DOI] [PubMed] [Google Scholar]
  17. Fish J.N., Williams N.D., McInroy L.B., Paceley M.S., Edsall R.N., Devadas J., Henderson S.B., Levine D.S. Q Chat Space: assessing the feasibility and acceptability of an internet-based support program for LGBTQ youth. Prev. Sci. 2022;23(1):130–141. doi: 10.1007/s11121-021-01291-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Glenn C.R., Franklin J.C., Nock M.K. Evidence-based psychosocial treatments for self-injurious thoughts and behaviors in youth. J. Clin. Child Adolesc. Psychol. 2015;44(1):1–29. doi: 10.1080/15374416.2014.945211. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Hatzenbuehler M.L., Pachankis J.E. Stigma and minority stress as social determinants of health among lesbian, gay, bisexual, and transgender youth: research evidence and clinical implications. Pediatr. Clin. N. Am. 2016;63(6):985–997. doi: 10.1016/j.pcl.2016.07.003. [DOI] [PubMed] [Google Scholar]
  20. Hinduja S., Patchin J. Bullying, cyberbullying and suicide. Arch. Suicide Res. 2010;14:206–221. doi: 10.1080/13811118.2010.494133. [DOI] [PubMed] [Google Scholar]
  21. Hinduja S., Patchin J. Cyberbullying: a review of the legal issues facing educators. Prev. Sch. Fail. 2011;55(2):71–78. doi: 10.1080/1045988X.2011.539433. [DOI] [Google Scholar]
  22. Hodson E., Dadashi N., Delgado R., Chisholm C., Sgrignoli R., Swaine R. Co-design in mental health; mellow: a self-help holistic crisis planning mobile application by youth, for youth. Des. J. 2019;22(1):1529–1542. doi: 10.1080/14606925.2019.1594975. [DOI] [Google Scholar]
  23. Horowitz L.M., Wharff E.A., Mournet A.M., Ross A.M., McBee-Strayer S., He J.P., Lanzillo E.C., White E., Bergdoll E., Powell D.S., Solages M., Merikangas K.R., Pao M., Bridge J.A. Validation and feasibility of the ASQ among pediatric medical and surgical inpatients. Hosp. Pediatr. 2020 doi: 10.1542/hpeds.2020-0087. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kowalski R.M., Giumetti G.W., Schroeder A.N., Lattanner M.R. Bullying in the digital age: a critical review and meta-analysis of cyberbullying research among youth. Psychol. Bull. 2014;140(4):1073–1137. doi: 10.1037/a0035618. [DOI] [PubMed] [Google Scholar]
  25. Leyro T.M., Zvolensky M.J., Bernstein A. Distress tolerance and psychopathological symptoms and disorders: a review of the empirical literature among adults. Psychol. Bull. 2010;136(4):576–600. doi: 10.1037/a0019712. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. LUMA Institute Persona Profile: LUMA Workplace. 2023. https://www.lumaworkplace.com Available from:
  27. Lyon A.R., Munson S.A., Renn B.N., Atkins D.C., Pullmann M.D., Friedman E., Arean P.A. Use of human-centered design to improve implementation of evidence-based psychotherapies in low-resource communities: protocol for studies applying a framework to assess usability. JMIR Res. Protoc. 2019;8(10) doi: 10.2196/14990. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Marshall A. Suicide prevention interventions for sexual & gender minority youth: an unmet need. Yale J. Biol. Med. 2016;89(2):205–213. [PMC free article] [PubMed] [Google Scholar]
  29. McDermott E., Hughes E., Rawlings V. Norms and normalisation: understanding lesbian, gay, bisexual, transgender and queer youth, suicidality and help-seeking. Cult. Health Sex. 2018;20(2):156–172. doi: 10.1080/13691058.2017.1335435. [DOI] [PubMed] [Google Scholar]
  30. Meyer I.H. Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: conceptual issues and research evidence. Psychol. Bull. 2003;129(5):674–697. doi: 10.1037/0033-2909.129.5.674. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Miller W., Rollnick S. Guilford Press; New York: 2013. Motivational Interviewing: Helping People to Change (3rd Edition) [Google Scholar]
  32. Morse J.M. The significance of saturation. Qual. Health Res. 1995;5(2):147–261. doi: 10.1177/104973239500500201. [DOI] [Google Scholar]
  33. Neal J., Neal Z., VanDyke E., Kornblug M. Expediting the analysis of qualitative data in evaluation: a procedure for the rapid identification of themes from audio recordings (RITA) Am. J. Eval. 2015;36(1):118–132. doi: 10.1177/1098214014536601. [DOI] [Google Scholar]
  34. Nesi J., Burke T.A., Bettis A.H., Kudinova A.Y., Thompson E.C., MacPherson H.A., Fox K.A., Lawrence H.R., Thomas S.A., Wolff J.C., Altemus M.K., Soriano S., Liu R.T. Social media use and self-injurious thoughts and behaviors: a systematic review and meta-analysis. Clin. Psychol. Rev. 2021;87 doi: 10.1016/j.cpr.2021.102038. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Nevedal A.L., Reardon C.M., Opra Widerquist M.A., Jackson G.L., Cutrona S.L., White B.S., Damschroder L.J. Rapid versus traditional qualitative analysis using the Consolidated Framework for Implementation Research (CFIR) Implement. Sci. 2021;16(1):67. doi: 10.1186/s13012-021-01111-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Nyumba T., Wilson K., Derrick C., Mukherjee N. The use of focus group discussion methodology: insights from two decades of application in conservation. Methods Ecol. Evol. 2018;9(1):20–32. doi: 10.1111/2041-210X.12860. [DOI] [Google Scholar]
  37. Pettit J.W., Buitron V., Green K.L. Assessment and management of suicide risk in children and adolescents. Cogn. Behav. Pract. 2018;25(4):460–472. doi: 10.1016/j.cbpra.2018.04.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Quintana-Orts C., Rey L., Neto F. Beyond cyberbullying: investigating when and how cybervictimization predicts suicidal ideation. J. Interpers. Violence. 2022;37(1–2):935–957. doi: 10.1177/0886260520913640. [DOI] [PubMed] [Google Scholar]
  39. Raifman J., Charlton B.M., Arrington-Sanders R., Chan P.A., Rusley J., Mayer K.H., Stein M.D., Austin S.B., McConnell M. Sexual orientation and suicide attempt disparities among US adolescents: 2009-2017. Pediatrics. 2020;145(3) doi: 10.1542/peds.2019-1658. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Sandelowski M. Whatever happened to qualitative description? Res. Nurs. Health. 2000;23(4):334–340. doi: 10.1002/1098-240x(200008)23:4&#x0003c;334::aid-nur9&#x0003e;3.0.co;2-g. [DOI] [PubMed] [Google Scholar]
  41. Sandelowski M. What's in a name? Qualitative description revisited. Res. Nurs. Health. 2010;33(1):77–84. doi: 10.1002/nur.20362. [DOI] [PubMed] [Google Scholar]
  42. Schueller S.M., Torous J. Scaling evidence-based treatments through digital mental health. Am. Psychol. 2020;75(8):1093–1104. doi: 10.1037/amp0000654. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Schueller S., Hunter J., Figueroa C., Aguilera A. Use of digit.al mental health for marginalized and underserved populations. Curr. Treat. Options Psych. 2019;6:243–255. doi: 10.1007/s40501-019-00181-z. [DOI] [Google Scholar]
  44. Tindall R.M., Ferris M., Townsend M., Boschert G., Moylan S. A first-hand experience of co-design in mental health service design: opportunities, challenges, and lessons. IJMHNI. 2021;30(6):1693–1702. doi: 10.1111/inm.12925. [DOI] [PubMed] [Google Scholar]
  45. Tynes B., Rose C., Williams D. The development and validation of the online victimization scale for adolescents. Cyberpsychology. 2010;4(2) https://cyberpsychology.eu/article/view/4237 [Google Scholar]
  46. Whaibeh E., Mahmoud H., Vogt E.L. Reducing the treatment gap for LGBT mental health needs: the potential of telepsychiatry. J. Behav. Health Serv. Res. 2020;47(3):424–431. doi: 10.1007/s11414-019-09677-1. [DOI] [PubMed] [Google Scholar]
  47. Witt K., Spittal M.J., Carter G., Pirkis J., Hetrick S., Currier D., Robinson J., Milner A. Effectiveness of online and mobile telephone applications (’apps’) for the self-management of suicidal ideation and self-harm: a systematic review and meta-analysis. BMC Psychiatry. 2017;17(1):297. doi: 10.1186/s12888-017-1458-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Wong-Villacres M., Disalvo C., Kumar N., DiSalvo B. Computer-Human Interaction; Honolulu, HI: 2020. Culture in Action: Unpacking Capactities to Inform Assets-based Design. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary material

mmc1.pdf (324.4KB, pdf)

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES