Stakeholder perspectives on digital behavioral health applications targeting adolescent depression and suicidality: Policymaker, provider, and community insights

Abstract

Background

With adolescent depression and suicidality increasing, technology-based interventions may help address mental health needs. The Enhancing Treatment and Utilization for Depression and Emergent Suicidality (ETUDES) Center supports development of technology interventions to help primary care providers target these problems. To successfully develop and integrate such interventions into primary care, iterative engagement of stakeholders is necessary. This paper outlines our stakeholder engagement process, a qualitative analysis of feedback and outcomes, and how feedback was incorporated to develop Center interventions.

Methods

Stakeholder panels represented key end-users of ETUDES Center interventions (adolescents, young adults, parents, and healthcare providers) and their advocates at the organizational/policy level. Meetings were held throughout intervention development and at annual retreats. Detailed meeting notes were collected and summarized by the stakeholder engagement team in real time, after which action items were generated and reviewed by the full research team. Using a content analysis approach, all stakeholder feedback summaries were coded using a prespecified codebook organized by recurring topics. Codes were organized under the Consolidated Framework for Implementation Research (CFIR). Anonymous stakeholder feedback surveys assessed relevance of topics, meeting effectiveness, and stakeholder involvement.

Results

Stakeholder meetings provided feedback on topics such as representation, language, access to care, and stigma. Relevant feedback and recommendations were incorporated into subsequent iterations of the interventions and their implementation. Mean stakeholder ratings of meeting proceedings on a 0–3 Likert scale ranged from 1.70 (SD  =  0.10) for participation to 2.43 (SD  =  0.08) for effectiveness in addressing meeting agenda.

Conclusions

The iterative engagement approach yielded practical feedback from stakeholders about ETUDES Center interventions. The team organized feedback to identify barriers and facilitators to using Center interventions and to generate action items, which were transposed onto components of an implementation strategy, supplemented by the CFIR-ERIC Implementation Strategy Matching Tool. Stakeholder feedback will direct the future development of an integrated intervention and guide further stakeholder engagement in developing technologies for adolescent mental health.

Plain Language Summary: Depression and suicide in teens have been on the rise for the past several years. Primary care may be an ideal place to address these concerns because most teens have a primary care provider (PCP) who can offer a confidential place for both teens and their caregivers. Our Center develops technology-based interventions to help PCPs address teen depression and suicidality. Multiple barriers may hinder PCPs in addressing teen depression and suicidality. This paper describes how we engaged multiple types of stakeholders—or individuals with an interest in adolescent mental health—to ask for their feedback over a series of meetings, to help us improve the fit of our interventions to this population. Stakeholders included providers, community members like parents and mental health advocates, youth, and policymakers. We learned what was most important to them—including concerns about equitable access (e.g., providing broadband internet to families who do not have it so they can still participate) and legal risks or failures for the technology to identify suicidality. In this paper, we describe our process for not only recruiting and engaging stakeholders but also for planning action based on their feedback. Similar processes could be used by other researchers and clinical organizations seeking to incorporate technology into mental health interventions.

Introduction

Adolescent depression and suicidality have increased substantially over the past decade, with 15.7% of persons aged 12–17 experiencing a major depressive episode (Substance Abuse and Mental Health Services Administration, 2020). Primary care is an ideal place to address these issues, given the large numbers of adolescents seen (Rand & Goldstein, 2018) and the increasing role played by primary care providers (PCPs) in delivering behavioral health services (Foy et al., 2019). Currently, one-third of youth receiving outpatient mental healthcare receive care from their PCP alone (Anderson et al., 2015).

Pediatricians recognize the growing need to assume care for children's mental health (Foy et al., 2019). While thoughtful application of technology may be greatly beneficial, through expanding and supporting integration of behavioral health in primary care, promoting easy and ongoing access, decreasing stigma, allowing options for provider portals, and being highly scalable, it may also create additional barriers such as disruptions to workflow, need for training, and lack of technical support. Careful incorporation of iterative stakeholder feedback into implementation strategies may decrease barriers to technology use for adolescent mental health interventions (Montague et al., 2015; Newton et al., 2020). Improving fit between needs of patients, parents, and providers, and technical features or requirements can provide key solutions for enhanced delivery of effective behavioral health (Montague et al., 2015; van Gemert-Pijnen et al., 2011).

This Enhancing Treatment and Utilization for Depression and Emergent Suicidality (ETUDES) Center is funded by the National Institute of Mental Health (NIMH) to help professionals in pediatric primary care better support depressed or suicidal youth. The ETUDES Center is overseen by a Steering Committee consisting of primary investigators, a community member who leads a mental health non-profit serving families, and a PCP from the community. The Center Steering Committee (CSC) establishes scientific priorities, supports annual evaluations, and oversees studies, resources, human protection. The ETUDES Center supports three studies: (a) Screening Wizard, a screening decision support tool to assist providers in making treatment recommendations (Radovic et al., 2021); (b) Text2Connect, an automated text messaging intervention to increase adolescents’ motivation to attend mental health appointments (Suffoletto et al., 2021); and (c) BRITE, a personalized safety planning and emotion regulation app onto which providers initially onboard new adolescents (Kennard et al., 2018). The goal is to combine these into an integrated suite of technology interventions for addressing adolescent depression and suicidality in primary care.

Stakeholder engagement is crucial at all intervention design stages and their successful integration into primary care. It is important to iteratively engage multiple types of stakeholders in all stages of development processes and create a conduit between stakeholder feedback and researchers. Stakeholder feedback can elucidate perceptions about deficits in the current system, and initial barriers to the implementation or use of interventions (van Gemert-Pijnen et al., 2011), helping researchers troubleshoot challenges earlier, saving time and money. Given that the Center supports several studies with their own recruitment of individual patients, providers, and parents (or other caregivers), to determine specific usability of interventions, we developed a network of multiple stakeholder constituencies representing these specific groups as well as policy-level stakeholders. The main purpose of these stakeholder panels was to identify potential barriers, and facilitators of the large-scale integration of digital health tools into routine clinical practice to inform current and future implementation strategies.

To guide our stakeholder participation process, we have incorporated relevant frameworks and tools from stakeholder engagement and implementation science, many of which have been applied in our prior work (Badawy & Radovic, 2020; Radovic et al., 2018, 2019). For example, we incorporated aspects of stakeholder engagement models that offer methods for planning, evaluating, and reporting stakeholder engagement and recommendations (Ray & Miller, 2017). We used the Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2009) to organize stakeholder feedback regarding the perceived determinants of digital intervention across its five domains (e.g., barriers and facilitators; Keith et al., 2017), which are relevant to the implementation of primary care interventions (Lau et al., 2015). CFIR was also used to identify potential implementation strategies (Waltz et al., 2019) that are also based on the ERIC compilation (Powell et al., 2015). Our evaluation included tools and methods parameters that have been used to capture both quantitative (Salentine & Johnston, 2011) and qualitative feedback (Chambers, 2019; Hamilton & Finley, 2019) from stakeholders. These approaches are compatible with several recommendations for delivering technology-based applications and interventions targeting adolescent mental health (Boydell et al., 2014; Klesges et al., 2005; Odgers & Jensen, 2020).

This paper addresses three key interrelated aims. First, we describe the development and composition of four stakeholder groups designed to provide complementary perspectives on the use of digital technologies for adolescent depression and suicidality. Second, we discuss the methods and process for incorporating this stakeholder network to revise the design and use of these technologies in real time. Third, we present results from a quantitative survey of stakeholders’ experiences and a qualitative analysis of meeting proceedings that describe perceived barriers and facilitators of implementation (organized by CFIR domain), and actions taken by the research team in response to specific stakeholder feedback. Such information will be used to further tailor and contextualize our study interventions and may inform other mental health technology efforts in pediatric primary care.

Methods

Stakeholder Panel Development and Description

Purpose and Organization

Our stakeholder engagement team (SET) consisted of leadership and administrative support staff. The SET leaders were Dr. Radovic, a pediatrician and subspecialist in adolescent medicine, and Dr. Kolko, a clinical psychologist. The administrative support staff included Dr. Kirk-Johnson, at the time in doctoral training for Applied Developmental Psychology. Dr. Kirk-Johnson was responsible for coordinating and facilitating meetings, communicating with stakeholders between meetings, and collecting anonymous feedback. Ms. George-Milford is a medical anthropologist who serves as project coordinator for the study and supported coordination between the CSC and stakeholders. Other research staff assisted with note-taking during meetings. Ms. Coren, a graduate medical student, did not interact with stakeholders but assisted with qualitative analyses.

In coordination with the CSC, the SET developed an organizational scheme describing the functions of stakeholder panels across all research phases and their relationship to the overall Center mission. Depicted in Figure 1, this scheme highlights four steps for engaging stakeholders, presenting updates on research developments to identify areas for improvement, discussion of feedback during CSC meetings, and translation of key suggestions into implementable actions.

Figure 1.

How the Center organized and tasked each stakeholder panel.

The SET determined key stakeholders should represent all potential intervention end-users including adolescents and young adults who could reflect on their recent experiences (will be referred to as youth from now on), parents, and healthcare (primary care and mental health) providers. Additionally, key stakeholders should include representatives and advocates for these groups, as well as decision-makers who can affect organizational-level, payment-structure, and policy-level changes to integrate technology into primary care. To determine structure and membership, the SET considered potential stakeholders’ availability and anticipated comfort level in sharing perspectives in a group (e.g., parent may speak more freely in community group rather than in presence of county health official). Based on these considerations and our prior research, advocacy, and clinical interactions with specific stakeholders, outreach at community events, and recommendation of other Center members, the SET identified stakeholders for membership, with a goal of about 10 members per panel.

Member Composition

The SET created separate community, policy, and provider stakeholder panels (Table 1). The Team initially experienced difficulty recruiting youth for the community panel. A discussion with a young adult and high school teacher in that panel revealed the comfort level among adolescent participants within a majority adult group may be low, signifying these power differentials may be hindering discussion (Hamilton & Finley, 2019). Thus, the Team separated the community group into a mostly adult group and a separate youth-only (14–25) group. The Team recruited youth with a digital/physical flyer, by attending community events, and word of mouth recommendations, and through a locally established Youth Research Advisory Board (YRAB) (Navratil et al., 2015) that reviews research in clinical settings.

Table 1.

Stakeholder panel description.

Criteria	Area/type of involvement	Member description
Community stakeholder—adult
Young adult Parent Mental health advocate Other community members with lived experience with adolescent mental health or who advocate for them	Planning the study - Review initial intervention ideas - Provide an overview of their general concerns about the mental health system Conducting the study - Review recruitment materials - Provide feedback on proposed study changes Dissemination - Provide recommendations for how to engage with community members and how to disseminate findings more broadly to community	8 mental health advocates (communities of color, disability, schools, workforce development, peer support at local and state levels), most with lived experience High school educator leading a Teen Mental Health Program 2 parents with lived experience caring for a child with mental health disorder Young adult with lived experience interested in mental health field
Community stakeholder—youth
Adolescent or young adult	Planning the study - Provide feedback on their experiences with mental health screens in primary care and with technology use for health Conducting the study - Review recruitment materials - Provide feedback on intervention changes - Inform how to best approach youth	Individuals aged 15–26 offering youth perspectives to researchers and community partners (meetings attended by 5–7 individuals)
Provider stakeholder
Primary care provider or behavioral health provider who treats adolescent patients Represent a variety of practice networks/agencies (not only internal to research team's hospital system) Availability at least quarterly Willing to provide feedback	Planning the study - Define study population - Provide input on changes to study design that minimizes disruption to clinic workflow and clinic norms - Recommendations for engaging providers (PCPs, BH clinicians) Conducting the study - Review data collection experiences Dissemination - Provide recommendations for how to expand findings to other providers in new settings (e.g., rural)	6 behavioral health providers with variety of expertise including child development, pediatric primary care, family medicine, adolescent medicine and adolescent mental health, and one who is also a nurse practitioner at a rural site 4 primary care providers (family medicine, pediatrics, child psychiatry)
Policy stakeholder
Individuals in leadership positions representing decision and policy makers involved in health insurance, healthcare organizations, professional organizations, non-profits, foundations, and governmental agencies Availability at least biannually	Planning and conducting the study - Provide updates on current and upcoming policy changes in mental health - Think broadly to ensure that research fits with practice parameters, health care policies, and emerging initiatives to ensure their applicability and generalizability Dissemination - Work with government and state-wide agencies/organizations that can disseminate and implement the new findings	5 health plan leaders (behavioral health integration, digital interventions, research design and implementation, Medicaid managed care) 2 experts in governmental affairs and policy in healthcare 2 non-profit/foundation leaders working in mental health 2 physician leaders with expertise in research, adolescent medicine, and leadership in a state pediatric organization

Note. PCPs = primary care providers; BH = behavioral health.

Stakeholder membership grew and changed organically, as the SET, CSC members, and current stakeholder members referred new members, through respondent-driven sampling (Hamilton & Finley, 2019). Panel members made referrals based on needs for individuals with particular strengths or perceived gaps in perspectives as well as availability and interest. There was no advertised time commitment for participation. They participated based on their interest in attending, their availability during meeting time, and, for policy, their continued involvement in their organization. Youth were not expected to attend more than one meeting, but this was a possibility if they attended YRAB regularly. Seven members left groups because of (a) professional commitments, (b) leaving their organization, or (c) their organization closed. See Table 1 for stakeholder domains of expertise (e.g., parent, community advocate, pediatrician, social worker, etc.). The backgrounds of stakeholders were diverse, with the provider panel representing eight different practice settings.

Member Orientation

Stakeholders were encouraged to attend as many meetings as possible but could terminate their position at any time for any reason. The procedures (meeting participation and feedback interview) were reviewed by University of Pittsburgh Institutional Review Board and deemed as not research, although for youth under 18, IRB approval was obtained with waiver to obtain parental permission to maintain privacy about sensitive information discussed.

Methods and Process

Panel Meeting Structure, Process, and Agenda

An overview of the development, methods, process, and analysis conducted during this project is provided in Figure 2. The Team scheduled meetings at inflection points when particular feedback was needed or to report back information to stakeholders. Stakeholders were invited to annual Center retreats including all study investigators and the CSC. This was an opportunity for stakeholders to hear a summary of Center progress, remaining challenges, and decisions to be made, and to interact with the full Center group. The SET integrated relevant content and devoted input periods for stakeholders into the agenda. Seven meetings (including 2 retreats) were held for Providers, eight meetings (including 2 youth-only meetings and 2 retreats) were held for Community, and six meetings (including 2 retreats) were held for the Policy group over 2 years (Table 2). Meetings lasted 1.5 h. For full-day retreats, stakeholders could attend as much as they desired, but 1 h was reserved for stakeholder feedback discussions. This group-based approach spurred discussion among group members of shared versus unique experiences. While meetings began prior to the COVID pandemic, a remote option via the WebEx platform was available for those who could not attend in person, though many stakeholders joined in person.

Figure 2.

An overview of the development, methods, and analysis described.

Table 2.

Stakeholder meetings and attendees per panel.

Note. YRAB = Youth Research Advisory Board.

The COVID pandemic resulted in necessitating meetings to be conducted entirely remotely for stakeholder and staff safety. To accommodate remote discussions, the Team used University-sponsored online video and tele-conferencing services and used chat features and hand raising options. After each meeting, stakeholders completed anonymous feedback surveys about their participation. Stakeholders were offered (some declined) a monetary gift ($75 on a prepaid debit card) and a meal during in-person meetings.

During meetings, the SET mediated discussions based on questions proposed by the CSC and Center investigators, addressing intervention-specific and general questions about the Center's future goals. The SET research administrative staff recorded detailed written meeting minute notes, as opposed to audio recordings so that stakeholders could speak freely. SET administrative staff further summarized detailed meeting notes, after which SET Leadership (Drs. Kolko and Radovic) developed a high-level summary with corresponding action items the Center could take to address stakeholder concerns. This high-level summary was then presented to the CSC as the basis for discussions evaluating which action items were currently actionable and which could be addressed in the future. To enhance transparency and track progress, the SET developed a living spreadsheet of all action items and a corresponding color system, which was accessible to both the CSC and stakeholders on a shared drive. The color system is like a traffic light system, to indicate if the stakeholder feedback provided had been integrated by the Center (green), was in process of being integrated (yellow), or not yet acted upon due to the need for additional feedback or resources (red). This action item spreadsheet was iteratively updated, and stakeholders were reminded of its availability as a link on their section of the research website so they could provide feedback to the SET at any time.

Qualitative and Quantitative Data Management and Analysis

Qualitative Data Regarding Stakeholder Panel Feedback

All detailed meeting minutes and the action item spreadsheet were examined as a part of a qualitative analysis. A directed content analysis approach (Hsieh & Shannon, 2005) was used, where the main purpose was for qualitative description to provide a “comprehensive summary of events in the everyday terms of those events” (Sandelowski, 2000), or a high-level summary which would stay close to the intended meaning gleaned from stakeholder feedback. Two SET members reviewed all minutes and agreed on an initial high-level codebook based on meeting agenda, the final action item spreadsheet, and by reading through meeting minutes and making note of recurring topics. Largely driven by probes and questions from meeting agenda items, high-level codes included feedback specific to individual interventions (Screening Wizard, Text2Connect, and BRITE), as well as a “General/nonspecific intervention” code for feedback pertaining to all interventions more generally. Additional high-level codes included “Privacy/confidentiality,” “Perception of mental health in primary care,” “Mental health stigma,” “Parent involvement,” “Policy concerns,” “Barriers to mental healthcare,” “Stakeholder process feedback,” and “Effects of COVID-19.” One coder, who received training from a qualitative expert throughout the process, reviewed and coded all meeting minutes using NVivo (QSR International Pty Ltd, 2020), and added sub-codes around diversity, monitoring, technical issues, implementation and dissemination, recruitment, and personalization. The coder determined a sub-code was needed when it was a topic, which was brought up by stakeholder members organically (i.e., not listed in agenda) multiple times across meetings. The finalized codebook was reviewed with the SET for purposes of member-checking and triangulation.

Quantitative Survey Data Regarding Stakeholder Experience

Post-meeting surveys were administered after each meeting to gauge stakeholder feedback about meeting format and content. Surveys were chosen over individual feedback sessions to preserve anonymity and authenticity. Surveys included four questions rated on a Likert scale (0  =  none/not at all, 1  =  a little/somewhat, 2  =  a lot, 3  =  very much) with space for free response. The questions included: (a) How much did you participate in the meeting? (b) How effective was the conduct of the meeting in addressing the agenda (i.e., soliciting ideas, respecting differences of opinion, supporting the conversation/discussion, summarizing key results)? (c) How relevant (significant, important) was the discussion/content of the meeting to advancing care for depressed/suicidal youth? (d) How much do you think the discussion/content of the meeting can help to advance the care for depressed/suicidal youth? Descriptive statistics (means, standard deviations) were used to summarize these results. Surveys were also reviewed individually between meetings to inform improvements to meeting structure. Anonymity was maintained in two ways. If the attendee was in-person, we asked them to not include their name on a paper survey. If the attendee was remote, they emailed the document to one Stakeholder Team member who saved the document to a folder without their name, and then deleted the email. The Team member summarized the surveys only after all surveys were returned.

Results

Panel Attendance and Engagement

Attendance by group type is noted in Table 2. Overall, meetings were well attended by stakeholders, who offered many comments about study progress and valuable feedback as described below.

Qualitative Summary of Stakeholder Panel Feedback and Center Follow-Up in Response

Stakeholder meetings provided a rich level of diverse feedback on multiple questions posed by the SET over time. A common high-level code that occurred across individual interventions was the desire for personalization, examples being primary care practices desiring additional screening tools or adolescents personalizing the mobile safety-planning app with personal images and content. Feedback from different stakeholder groups built upon each other. Community stakeholders expressed irritation at the idea of being asked screening questions (Screening Wizard) at check-in to a visit if it seemed unrelated to the visit complaint or if the relevance of screening was not explained. Similarly, providers felt that false-positive screening results may prolong visits and that using the tool after visits may decrease screening fatigue. Stakeholders said their overall perception of how welcoming and competent a primary care setting is to addressing mental health would influence trust and engagement in treatment. Community stakeholders—both youth and adults—emphasized the role of stigma and the importance of normalizing addressing mental health concerns. They highlighted a gap in parental psychoeducation, which may decrease treatment engagement for depressed adolescents. Stakeholders identified multiple barriers to mental healthcare including access to school-based services, lack of transportation, unequal representation of underserved groups such as adolescents of color or those with disability or in rural areas, low health literacy, and a lack of access to technology; they also shared concerns about reimbursement and insurance, and other system-level barriers such as system legal concerns about suicidality being revealed. A full summary of feedback is provided in detail in a supplemental file.

Additionally, Tables 3–5 provide examples of some of the most salient stakeholder panel discussions, including the meeting agenda item, the stakeholder discussion summary notes, and how this feedback was proposed to be acted on by the Center; any feedback that was acted upon is highlighted in italics. We include the items in these tables that were novel or unexpected, or resulted in actionable items we planned to address. The full list is available in a supplemental document, “Content and Action Items.” We were able to respond to several important sets of feedback, whereas some feedback addressed issues outside of the study scope or a policy or organizational concern that could not be directly addressed. Still, the later feedback was informative and could be used to support future grant proposals and individual advocacy work, resource provision, and networking that could strengthen our relationships with stakeholders. A few examples of our responses to instrumental feedback are described below and highlighted in Tables 3–5.

Table 3.

Provider stakeholder feedback and Center actions example.

Agenda	Stakeholder responses	Center proposed action items (those implemented in italics)
1. Review BRITE and potential barriers to the use of a mobile application for suicidality	Approved of personalization for mobile application Gave feedback on use of distress ratings	Review distress ratings in BRITE
	Ensure all practices have internet connectivity (Wi-Fi hotspot)	Assure internet connectivity
2. Discuss how providers respond to suicide endorsements and their personal comfort addressing suicide	Carefully consider language for crafting messages on depression and suicidality	Use inclusive/destigmatizing language
	PCPs influenced by legal and administrative teams regarding routinely asking about suicide, recommend how the Center can use research findings to support advocacy to promote the importance of asking about and addressing suicidality in primary care	Address the utility of PHQ item-9 (suicidality question)
	Work with PCPs to help them enhance patients’ comfort in discussing suicidality and to dispel myths and misunderstandings	Support comfort in provider–patient conversations (provide suggested language)

Note. PCPs = primary care providers; PHQ = patient health questionnaire.

Table 4.

Adult community stakeholder feedback and Center actions example.

Agenda	Stakeholder responses	Center proposed action items (those implemented in italics)
1. Describe experiences with mental health services in primary care settings	Parents and youth are more likely engage with mental health interventions in primary care if the material (e.g., images in advertisements, content of interventions) and approach are culturally sensitive and inclusive (i.e., race, ethnicity, disability, body shapes, and sizes)	Create materials with diverse representation
2. Pros and cons of using Center interventions in primary care as opposed to other settings	Racial minority and rural communities require adapted interventions to address community context	Create materials with diverse representation Improve rural access to Center projects Use inclusive/destigmatizing language Outreach to those who cannot use technology
3. Reasons that depression is not identified during screening	Primary care clinics need to demonstrate they are welcoming of and comfortable with treating behavioral health Parental role is important in helping their child access mental health and to normalize help-seeking	Prepare primary care to be welcoming for mental health, especially to underserved populations Use inclusive/destigmatizing language Consider parent involvement in technology

Table 5.

Youth community stakeholder feedback and Center actions example

Agenda	Stakeholder Responses	Center Proposed Action Items (those implemented in italics)
1. Thoughts on youth not reporting they are experiencing depression to their primary care provider? How to make primary care more comfortable to address mental health?	Teens don’t identify primary care as place to address MH. Use normalizing/destigmatizing messages throughout practice for all and not just for those with MH symptoms. Teens interested in innovative ways to normalize seeking help for mental illness (e.g., visual aids like humorous memes).	Use inclusive/destigmatizing language Utilize many media types for intervention Revise language to build rapport
2. Request suggestions to overcome obstacles Center is experiencing (e.g., how to reduce mental health inequity for racial and ethnic minority populations; discomfort with data collection)	To connect with people of color as it is easier to relate to someone more similar to you (in race, experience, etc.) It is a norm for youth that commercial apps obtain personal information-youth are comfortable with providers accessing this information because they trust the healthcare system	Create materials with diverse representation Prepare primary care to be welcoming for mental health, especially to underserved populations

Note. MH = mental health.

Since the integrated intervention suite is designed to promote options for addressing suicidality in primary care, we wanted to address the concern that suicidality may be a difficult topic for providers to bring up in the context of a busy primary care practice (Table 3). Provider stakeholders shared that even if they desire to address suicidality, there are legal and administrative barriers to screening. For example, when screens are embedded in a health portal, there is a concern that suicidality may be expressed but the response not reviewed in a timely fashion, increasing risk to the practice. Providers shared that patients and families have myths and misunderstandings about suicidality which are difficult for them to address. Providers hoped Center research would provide evidence for low risk of using technology as an adjunct to address suicidality in primary care.

Both community and policy stakeholders emphasized the importance of diversity (i.e., race, disability, body type, and sexual and gender minorities) in marketing and recruitment materials (Table 4). Stakeholders at the retreat suggested considering how mental health is viewed in different cultures. Subsequently, recruitment and marketing materials were enhanced with more diverse images (see Supplemental File). Community stakeholders also requested more inclusive language (i.e., removing “fluent in English,” replacing “parent” with more inclusive terminology). They recommended multicultural training and suggested soliciting stakeholder feedback on research advertisements and other materials regarding diverse representation in between regularly scheduled stakeholder meetings. The next iteration of the Center proposal will have a specific focus on reaching underrepresented adolescent populations.

Youth community stakeholders expressed that for Screening Wizard, false negative screens may occur due to non-disclosure from adolescents’ concern about the implications of a positive screen (Table 5). They suggested normalizing participants’ feelings about stigma and providing statistics about the prevalence of mental health disorders before the screen to increase comfort and honesty with memes or other visuals to loosen the atmosphere. Youth felt that it was important to include a disclosure that adolescents can talk to their provider without their parent present. We incorporated this feedback in the next version of Screening Wizard by including images with accompanying statements relaying confidentiality assurance, anti-stigma messaging, and normalizing prevalence of mental illness symptoms in adolescents.

Policy stakeholders emphasized disparities in access to technology with regards to digital literacy and access to high-speed internet (Table 6). They saw potential for Center interventions to increase reach to rural populations, but emphasized policy changes to enhance internet access and to develop simple instructions for how to use technologies. In response, the next iteration of the integrated intervention will include funds for providing access to smartphones and internet for patients who do not have it.

Table 6.

Policy stakeholder feedback and Center actions example.

Agenda	Stakeholder responses	Center proposed action items (those implemented in italics)
1. What are best ways to get the word out about Center interventions?	Integrate with the Electronic Health Record, Engage with and Enhance the training of mental health professionals in primary care and in schools	Consult with healthcare experts in system-level changes
2. What policy/regulation issues would support/hinder the rollout of these interventions in other venues	Consider costs: learn about value-based payment models	Consult with healthcare experts in system-level changes
3. How could the Center interventions help during the COVID-19 pandemic?	COVID-19 pandemic helped people realize the importance and need for telehealth Some populations struggle with technology use, there is a need for simple instructions to use apps	Improve rural access to Center projects Assure internet connectivity Expanding contexts outside of primary care Outreach to those who cannot use technology Education about telehealth Interventions should simulate in-person healthcare interactions

Summary of Stakeholder Feedback by CFIR Domain with Recommendations for Implementation

After the qualitative analysis was complete, the SET used the implementation science framework, Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2009) to organize findings. All feedback obtained from each panel over time was summarized to provide a comprehensive overview of all identified barriers to, and facilitators of each research or clinical practice feature/function that was discussed. These determinants were then categorized into each of the 5 CFIR domains (i.e., [a] Intervention Characteristics, [b] Outer Setting, [c] Inner Setting, [d] Characteristics of Individuals, and [e] Implementation Process) to better understand the number and type of determinants at each level. There were four barriers which mapped onto the outer setting including concerns about patient mistrust, limited phone access, families mistrust of research, and limited diverse images and messaging. There were four barriers mapping onto the inner setting including workflow obstructions, poor timing of Screening Wizard, concerns about having to track a negative mood, and the clinical team not noticing distress or suicidality identified by the interventions.

Next, the SET mapped CFIR determinants onto the 72 implementation strategies identified in the ERIC final compilation by (Powell et al., 2015). Finally, the SET identified key specific tasks that could be used to carry out each strategy in the context of the next iteration of the Center, which was to focus on the integration of the three separate intervention applications into a multicomponent digital treatment app. Table 7 outlines all CFIR elements that were coded and how proposed action items from stakeholder meetings transpose onto components of an implementation strategy. Strategies that were identified more than once included audit and provider feedback, prepare consumers for services, and identify and prepare champions. For example, providers raised concerns that because Screening Wizard includes screens for mania and suicidality, some providers may feel underprepared to address these within primary care. The implementation strategy of identifying and preparing local provider champions to receive additional training to address these concerns may help decrease this unease, especially when accompanied by an incentive strategy to meet certification requirements.

Table 7.

Proposed implementation strategies and tasks organized by CFIR to address reported technology barriers.

CFIR domain	Barrier	Implementation strategya	Specific task example
Intervention characteristics	New technology not working	Provide technical assistance	Have customer support and technical liaison available
		Audit and provider feedback	Offer online tutorials with troubleshooting feedback
	Legal risk to provider participation	Review/address liability laws	Seek legal guidance to revise procedure/policy to lower risk
			Engage leadership to balance risk management & patient need
	Limited parent role in teen/own services	Intervene with patients/families	Give caregivers information/support for teen's services or family care
		Prepare consumers for services
Outer setting	Provider concerns about patient mistrust	Conduct outreach visits	Support facilitators of the integrated intervention (e.g., destigmatize, save time, enhance family communication, gain trust, personalized care, ease of use)
	Limited teen phone access or data plan	Assess for readiness/identify barriers/facilitators	Include/support families who do not have access to phone
		Intervene with patients to enhance uptake	Offer outreach if client cannot access or easily use technology
			Create simple and easily accessible app instructions
			Explore phone service options to enhance sustainability
	Some families mistrust research	Prepare consumers for active role in services	Involve consumers in recruitment and ongoing support efforts
			Review research risks, safeguards, and monitoring plans
	Limited diverse images/messaging	Get feedback from diverse stakeholders	Enrich ethnic/racial representation and language in digital media
			Offer multicultural recruitment, training, and treatment
Inner setting	Workflow obstructions/slow	Re-examine implementation, tailor strategies
			Refer to the integrated intervention through EHR; Better integrate this intervention into workflow with team feedback and EHR (offer examples, guidance, be a resource)
			Have all reports integrated into EHR
	Poor timing of SW	Change structure & equipment	Partner with EHR support system
			Solicit input from engaged leadership
	Concern about having to track low mood	Audit and provider feedback	Set up auto/EHR system alerts for crises or use of crisis services
	Not capturing or noticing distress or suicidality	Create a new clinical team	Establish a role for emergent/clinical monitoring
Individual characteristics	Discomfort with mental health referral, Unfamiliar with reports Lack of comfort with mania	Education (materials and meetings)	Teach clinicians when/how to use intervention/safety plan. Provide training/ongoing guidance via champion/ambassador
		Facilitation/consultation	Use QI/MOC Type 4 as incentive to participate and give feedback to providers via EHR to track follow-through with referral (quality metric).
		Identify & prepare champions	Provide personalized referral lists to each provider
Process	Understanding benefit of the integrated intervention	Create a learning collaborative	Connect insurer, school, MH community advocate, and MH expert to enhance service access and awareness of resources.
		Identify and prepare champions	Engage opinion-leaders/practice champions/hold stakeholder meetings

Note. CFIR = Consolidated Framework for Implementation Research; EHR = Electronic Health Record; SW = social work; QI/MOC = quality improvement/maintenance of certification; MH = mental health. ^aBased on ERIC compilation (Powell et al., 2015)

Stakeholder Experience Based on Quantitative Surveys

Survey response rate varied across groups over time: 50% (3/6) to 57% (4/7) for provider; 33% (3/9) to 88% (7/8) for policy; 50% (5/10) to 100% (7/7) for community; and 100% (7/7) for youth. For reference, all ratings are on a 0 to 3 scale with means calculated across groups and meetings. The mean response to level of participation was 1.70 (SD  =  0.104). Stakeholders cited being new to the group, strong personalities within the group, and technological challenges with virtual meetings as reasons for low participation. The mean response to effectiveness of addressing meeting agenda was 2.43 (SD  =  0.079). Stakeholders indicating a lower score for this question cited inadequate time and tangential discussions, which were still viewed as beneficial. The mean response to relevance of meeting discussions to advancing care for depressed/suicidal youth was 2.41 (SD  =  0.086). Reasons for low scores included focus on parent and provider concerns rather than those of youth, and an emphasis on identification/advertising/recruitment in the discussions as opposed to patient care. The mean response to the extent to which meeting discussions and content can advance care for depressed/suicidal youth was 2.10 (SD  =  0.098). Stakeholders cited stigma and a need for advocacy as reasons for low scores. Free text responses to these questions are highlighted in Table 8.

Table 8.

Selected free text responses about stakeholder experience.

Theme	Illustrative quote
Different types of stakeholders in one group led to productive discussions	“I think there's a nice mix of roles/jobs that the participants play in suicide prevention. The moderators do a good job of soliciting opinions from these various perspectives. (I feel this will not only help in the studies’ success, but also will help make clinical application of study findings more practical).” - Provider stakeholder Sep 18, 2019
Post-meeting free text responses were used to share a dissenting opinion that may not have been in line with group discussion	“I think there ended up being a lot of discussion about parents and providers but not as much focus on what really works for young people. (Ex: there are concerns about confidentiality with an app but teens will probably be much more likely to use it.)” - Adult Community stakeholder Apr 30, 2019
Both providers and youth felt anti-stigma messaging and education of the public can be instrumental to meaningful change	“It's clear that ‘educating’ the public – elected officials/consumers/patient families – is a key component of this process. We need to think about the best way to move forward to do that.” - Policy stakeholder Feb 8, 2019
	“I feel that depression & suicidal things for youth can only advance as much as the youth are willing to open up. This can only be changed by shifting public opinion about mental illness.” – Youth Community stakeholder Sep 5, 2019
Youth enjoyed being asked about the agenda topics and felt heard	“I liked how when they asked for our opinions they took everything we said into consideration.” - Youth Community stakeholder Sep 5, 2019
	“I think that the content that we discussed will help people like me who don’t like to open up or tell the truth.” - Youth Community stakeholder Sep 5, 2019
Stakeholders noted increased relevance of Center in improving technology integration in primary care surrounding mental health due to the COVID-19 pandemic	“The research is particularly appropriate given the recent transition to telehealth and need for virtual engagement options.” - Policy stakeholder Jun 11, 2020
	“Having a choice and or options is very important to youth. If we can take something positive out of this quarantine, I feel like the transitions we have made to virtual meetings could be used as a possible option for youth to choose to do virtual or in-person once we are cleared to return to normal. Although there are challenges with virtual meetings, I feel that this could be a great chance to give youth an option and the power to make that decision.” - Adult Community Stakeholder May 27, 2020

Discussion

This report describes an integrated approach to identifying, engaging, soliciting, and applying feedback obtained from four panels of stakeholders to critically inform the design and deployment of digital tools for addressing adolescent depression and suicidality in primary care. Our SET was able to synthesize feedback from multiple stakeholder panel discussions to present to the CSC, who then made decisions on which items were actionable. From this process, we were able to circle back to stakeholders with a living document of how we were collecting their feedback and acting on it, which we would also discuss at meetings. Some stakeholders attended Center-wide retreats and directly gave feedback to the CSC and study principal investigators during larger meetings which also included interactions with outside scientific advisory board members. Stakeholders had multiple venues for actively and meaningfully providing and discussing their feedback and understanding how it was being utilized.

Concerns raised by our stakeholder panels may generalize to other mental health technology interventions that could be integrated into pediatric primary care. For example, providers highlighted the need for human support, concern about confidentiality, time, inaccuracies in screening, comfort with behavioral health, and disruptions to workflow. Providers felt Center interventions could facilitate communication with families to encourage focused, timely, efficient, and relevant management conversations, increasing trust and destigmatizing mental health issues. Providers recommended limiting workflow disruption by increasing flexibility for intervention timing, utilizing existing strategies through the electronic health record, being alerted if a patient is in crisis; they also suggested the use of learning collaboratives and a technical liaison.

One particular concern raised by providers was related to using technology to identify suicidal risk and their legal responsibility to address risk in a timely manner. Although consensus statements exist for how researchers should intervene on suicide risk in context of digital monitoring studies (Nock et al., 2021), there is less guidance available for clinical practice, where rapid changes in healthcare due to the COVID-19 pandemic necessitated telehealth for patients at high as well as low risk (Society for the Advancement of Psychotherapy, n.d.). Challenges resulted including being ill equipped to rapidly respond to emergency situations (Stoll et al., 2020), triage the patient, or remotely assess level of suicidality (Gilmore & Ward-Ciesielski, 2019). A recent article (Johnson & Aldea, 2021) recommends obtaining multiple emergency contacts to provide assistance if patient safety becomes a concern, to plan for technical difficulties, and to confirm a patient's location. As use of mobile mental health interventions grows, researchers will need to stay abreast of guidelines and legal concerns, including those of malpractice, where the concern of breach is most relevant, meaning that a physician's care was below the standard, including that of properly managing a suicidal patient. This standard is more uncertain in the case of newer technologies, as the standard of care is based on how well other physicians understand their use, applications, and limitations (Wiley, n.d.). Implementing new technology tools in these settings will require briefing providers on potential liability concerns and ensuring patients clinically understand their risks, benefits, and limitations.

Community stakeholders who were also parents appreciated the potential value of the interventions, but they did have concerns about being left out of knowing about their child's mental health due to confidentiality, and recommended having accompanying interventions for themselves. In contrast, youth had concerns about confidentiality and their results being shared with a parent. Healthcare technologies for both parents and adolescents need to consider a delicate balance between providing each with information while protecting adolescent confidentiality (Society for Adolescent Health and Medicine et al., 2014). A stakeholder-informed design of a patient portal for adolescents encouraged adolescent-provider confidential communication without negative feedback from parents (Thompson et al., 2016).

Multiple stakeholders expressed how diverse groups may be left out of using new technology interventions. Some may have less trust in healthcare, especially in technology, due to poor prior experiences with healthcare or research due to discrimination. Other barriers include cost of a data plan, cost of high speed internet, reading literacy, language proficiency, and confidentiality concerns. Studies show racial and ethnic minorities are less likely to have broadband at home (Ralston et al., 2019). Existing implementation science models such as Extension of RE-AIM to Enhance Sustainability (Shelton et al., 2020) focus on the importance of measuring health equity with the goal of promoting equitable access and outcomes. While specific implementation strategies may be beneficial on average, understanding what additional strategies result in adequate reach to youth of color, low-income youth, non-English speakers, or sexual and gender minority youth and their families will result in more equitable implementation. As described in Table 7, different extensions of the implementation strategy such as funding for phone services or multicultural adaptations (Jonassaint et al., 2017) may enhance the reach of the integrated intervention. Other pediatric studies found formative input from stakeholders, especially when iterative (Salloum et al., 2018), can help researchers understand perceived benefits of technology interventions and potential challenges to including families with limited access to technology or high-speed internet (Kan et al., 2021).

Stakeholder groups raised concerns about mental health stigma preventing help-seeking, and resulting in false negative screens. Recent studies support routine depression screens such as the PHQ-9 may miss up to a third of adolescent medical inpatients (Horowitz et al., 2021) and 8.3% of Medicaid-enrolled primary care adolescents for suicidality (Kemper et al., 2021). Screening via technology may enhance reporting, as youth using a computerized screen in primary care settings have higher percentages (14%) (Gardner et al., 2010) of endorsing suicidality than with nurse-administered verbal screens (3%) (Roaten et al., 2021). Yet adolescents who answered “no response” (neither yes or no) on a tablet suicide screener were almost as likely to have clinically significant risk (84.5%) compared to those who answered “yes” to the screener (93.4%), but were less ready to engage in mental health treatment (Hengehold et al., 2019). Technology interventions—especially in settings where mental health symptoms are being discussed for the very first time, like in primary care or in schools—may additionally help ease concerns about readiness for treatment through the promotion of symptom normalization, anti-stigma messaging, and additional assessment (Szlyk & Tan, 2020).

Although stakeholder feedback varied, almost all focused on potential barriers to the successful implementation of Center interventions. We found the widely used CFIR provided a useful template for organizing feedback and that using the CFIR-ERIC tool was helpful to inform future implementation strategies, which could be prioritized and bundled into a future accompanying implementation strategy toolbox that can be evaluated by measuring specific implementation outcomes.

Stakeholders provided process-oriented feedback that indicated they felt their opinions were heard. Some mentioned barriers to participation included technology difficulties or others’ strong personalities, but they did use the post-meeting survey to provide missed feedback. Other barriers not mentioned but likely present are that adolescent mental health is a difficult topic to discuss in a group. Both youth and caregivers with lived experience (e.g., prior history of severe depression or suicide) may experience extreme distress (e.g., sadness, anxiety) and potentially traumatic reactions (e.g., flashbacks) when they are exposed to questions or discussions in a group context. Although no stakeholders raised this concern, offering multiple ways to provide feedback is therefore important.

The iterative approach to stakeholder engagement was useful because stakeholders’ overlapping concerns could be shared with different groups at subsequent meetings. Having separate groups presented scheduling challenges but allowed for richer iterative feedback over time without influences of power discrepancies. Stakeholder engagement can be facilitated by providing information in advance to improve efficiency and effectiveness, accommodating different needs by offering multiple ways to give feedback, and continuing to maintain rapport between meetings. Stakeholders in mental health may be sensitive to inclusion in research due to prior ethical research concerns that health professionals override their decisions due to perceptions of limited decision-making capacity. Researchers should be sensitive to these potential concerns while allowing stakeholders to present difficult topics such as being stigmatized. Likewise, earnestly eliciting feedback from PCPs and mental healthcare providers about real day-to-day barriers to providing care which may seem inconsequential to researchers, such as the time it takes to complete a screen down to the minute, may in the long run greatly enhance the scalability of technology interventions.

Limitations

There are several limitations. There was no audio recording at meetings, so interpretations are based on extensive team notes. We had some attrition and stakeholder contributions at meetings were limited by scheduling conflicts and meeting duration limited to 1.5 h. The coding and implications of our stakeholder meetings may have been improved had we initially used the CFIR framework as a guide to questions asked on the agendas. Since completing this analysis, our group has reconsidered using the word “stakeholder” due to a violent connotation in tribal organizations (Centers for Disease Control and Prevention, 2022, Last Reviewed) and has since queried our current members for alternative preferred terms.

Conclusion

This paper describes the process of engaging stakeholders to critique emerging technology interventions to enhance their utilization in the treatment of adolescent depression and suicidality in primary care, including the development and composition of the panels, the methods and processes used to evaluate them, and both quantitative and qualitative data analysis findings. These ongoing and iterative stakeholder panel meetings conducted with providers, policymakers, parents, and community advocates/youth helped us understand potential barriers and facilitators to using components of the integrated intervention technology in real-world primary care practice. Our findings overlaid on the CFIR implementation framework inform specific initial implementation strategies and tasks that will support the future implementation of the integrated intervention and provides a framework for how stakeholder findings can be applied to technology interventions in adolescent mental health. These recommendations are informed solely by stakeholders’ feedback, and determining their effectiveness will require further implementation studies.