Skip to main content
NCBI home page
Internet Interventions logoLink to Internet Interventions
. 2023 Sep 28;34:100677. doi: 10.1016/j.invent.2023.100677

From formative design to service-ready therapeutic: A pragmatic approach to designing digital mental health interventions across domains

Jonah Meyerhoff a,⁎, Rachel Kornfield a, Emily G Lattie b, Ashley A Knapp c, Kaylee P Kruzan a, Maia Jacobs a,d, Caitlin A Stamatis a, Bayley J Taple a, Miranda L Beltzer a, Andrew BL Berry b, Madhu Reddy e, David C Mohr a, Andrea K Graham b
PMCID: PMC10551833  PMID: 37808416

Abstract

As digital mental health interventions (DMHIs) proliferate, there is a growing need to understand the complexities of moving these tools from concept and design to service-ready products. We highlight five case studies from a center that specializes in the design and evaluation of digital mental health interventions to illustrate pragmatic approaches to the development of digital mental health interventions, and to make transparent some of the key decision points researchers encounter along the design-to-product pipeline. Case studies cover different key points in the design process and focus on partnership building, understanding the problem or opportunity, prototyping the product or service, and testing the product or service. We illustrate lessons learned and offer a series of questions researchers can use to navigate key decision points in the digital mental health intervention (DMHI) development process.

Keywords: Digital mental health, Human-centered design, Methodology

1. Introduction

Digital mental health interventions (DMHIs) have potential to address treatment-access gaps and enhance mental health care delivery (Mohr et al., 2021) but most evidence-based tools never make it from the research context into real-world use. As DMHIs proliferate, there is a growing need to understand the complexities of moving these tools from concept and design to service-ready products.

Too often, DMHIs fail to translate into real-world use because they have not been designed as service-ready products, meaning they have not been designed for the people who will be using or impacted by them or for the settings in which they will be used. Human-centered design (HCD) is a design philosophy, research area, and practice that centers end-users throughout the design process to ensure that a tool or service addresses their specific needs, constraints, and preferences, and fits into the settings in which they will use it. Models of HCD have been described (e.g., Graham et al., 2019) and commonly follow a phased approach: investigate (assess end-users' needs, preferences, goals, and constraints); ideate (brainstorm design ideas); prototype (iteratively create design options); evaluate (iteratively evaluate design options with end-users); refine and develop (integrate design feedback and develop the tool); and validate (when the tool is made available and tested in practice). This “final” phase often generates new insights that inspire revisions to the tool's design.

Designing service-ready products also means understanding the setting(s) in which the DMHI will be delivered (e.g., Proctor et al., 2021). It is important to engage with people in the service setting (e.g., people delivering DMHIs, people incorporating DMHIs into their workflows, organization leadership, clinical supervisors) to ensure the DMHI can be implemented and sustained. Indeed, because HCD centers end-users in the design process, partnership building and maintenance are foundational throughout the design process. While partnership building is not formally one of the design phases, as described above, it is imperative during or even before embarking on the investigate phase.

Many methods can be used when developing a service-ready tool, and the role of the designer and clinical scientist is to make decisions – often in collaboration with the relevant interested parties (e.g., community organizations, healthcare systems, end-user communities) – around which methods to apply. For example, designers and clinical scientists, in collaboration with interested parties looking to build or adapt a DMHI, may engage potential end-users to understand the preferences and constraints that will guide the DMHI development and in so doing must make decisions about the type of study methods to employ (e.g., focus groups, surveys, interviews, co-design workshops, among others). They must then make decisions about how to take learnings and translate them into testable prototypes that inform software requirements for developers to subsequently build. Choices must then be made about how to evaluate these prototypes. For example, evaluation methods can be moderated usability testing sessions in which a user is tasked with performing actions while engaging with the prototype, unmoderated or self-guided usability testing where users are free to use the tool with minimal guidance or instruction, and/or extended usability testing deployments (“field testing”, “pilot testing”) where the prototype is deployed to a series of test users to use in situ, among others. Within each of these methods, there are similarly myriad design choices to be made that can affect the developed DMHI (e.g., appropriate number of participants to recruit, appropriate length of each study) - the decisions for which should be grounded in the research questions the clinical scientist aims to answer to inform next steps, and if possible, aligned with the needs of the end-users and deployment setting. Because HCD is iterative, it is important that clinical scientists who are interested in HCD understand how to select and apply these methods and when to advance the design process.

In this paper we outline pragmatic approaches for clinical scientists to design and develop service-ready products. We use case examples from research projects conducted at the Center for Behavioral Intervention Technologies (CBITs) to demonstrate the application of design methods at different phases of the HCD process. We highlight pragmatic considerations that scientists make in selecting and applying methods, and at what point they move onto future phases.

2. Materials and methods

Five case studies were compiled that illustrate different phases of the design process. Key decision points and the relevant case studies were identified by consensus from the authors who have experience conducting community-engaged research, designing, developing, implementing, and evaluating DMHIs in both research and real-world settings.

3. Results

3.1. Partnership building

The first decision in the design-to-product pipeline is aimed at determining who will be involved in the design process and begin determining the DMHI's scope. Because DMHIs are ideally designed for the people who will use them and the settings and contexts in which they will be used (Mohr et al., 2017), it can be critical to involve not just the individuals who will use the eventual DMHI but also the organizations who will implement it. Consequently, building a partnership with an organization can be an important precursory endeavor before embarking on the HCD process. Partnership building is about building credibility, earning trust, redistributing power, demonstrating commitment to the organization of interest, and fostering mutual learning about the researchers' and potential partners' goals, priorities, and perspectives – and continues throughout the design-to-product pipeline. Although the researcher may want to eventually work with an organization in a research capacity, partnership building is a non-research endeavor. With that said, some HCD methods and practices can be vital precursors to conducting research with organizations of interest.

Case Study 1 details how a community-research partnership was established between a local, Chicago-area public library, their teen patrons, and a research team.

  • The library's Teen Services program is designed for teens, by teens, and focuses on their social-emotional and academic development. In response to the high mental health needs of its patrons, the library expanded its social services within the last five years to include mental health services. Some of these services included hiring social workers as employees and providing free mental health assessments to library patrons. During this expansion, our team reached out to library leadership to learn about the library's mental health services and to explore the fit between the library's priorities and research teams' expertise in teen DMH services. At the initial meetings, it became apparent that the library leadership and research teams shared the common mission of providing accessible, evidence-based mental health resources to Chicagoland teens, and that there was high potential for partnership around designing accessible mental health resources with teens most experiencing health inequities (Knapp et al., 2023).

  • To support partnership building activities, the community and research leads applied for, and were awarded, an internal partnership development grant to financially support the library's teen services and partnership building activities. This included funds allocated for both a teen advisory board comprising teen library patrons and teens from the community the library serves, and an adult advisory board comprising teachers, parents, library workers, local government officials, and other community leaders. Both advisory boards were led by Teen Investigators hired by the researchers. The purpose of establishing the advisory boards was to facilitate bi-directional relationships between the researchers and the community the library serves and to build a shared vision for whether and how a DMHI could help provide mental health resources.

  • In the advisory board meetings, members discussed community strengths, the insufficiency of current teen mental health resources especially for Black, Indigenous, and teens of color (e.g., exclusion of Black, Indigenous, and teens of color from mental health narratives), accessibility issues (e.g., difficulty accessing mental health resources without disclosure to adults), and considerations for future design and implementation of mental health resources. Researchers commissioned graphic recordings – artistic representations of topics discussed in the meetings – to serve as a record of the meetings and a product to share in community and academic outlets.

  • Throughout work with the library, we (the researchers) aimed to earn community trust, re-distribute power typically delegated to researchers, and deepen our understanding of the community's priorities and concerns. While not an exhaustive list, we outline several methods we used. We prioritized “showing up” beyond research. For example, we attended teen-led community events including town hall presentations and performance art shows. These activities were beneficial for earning the trust of our library and teen partners, showing commitment to the community, and learning more about the community the library serves. To support our teen partners' academic and occupational pursuits, we wrote letters of recommendation on their behalf, which enabled us to formally demonstrate gratitude and acknowledge their many strengths. We attempted to adopt a stance of humility by asking for feedback, expressing our gratitude for critical feedback, and being transparent about how their input and feedback resulted in action or changes to the project. Further, we had transparent discussions around how research has caused harm to many Chicagoland communities, and to their community in particular. We intentionally owned these harms, even if not personally responsible, and were able to have productive discussions on methods of mitigating future harms. Additionally, in interactions with our community partners and in academic meetings and presentations, we explicitly celebrated and stated our appreciation for the expertise community partners brought to the project through their lived experience of mental health challenges and deep knowledge of the library's infrastructure. When funds were available, we showed our appreciation via compensation; for example, sharing honorariums for researcher-delivered invited talks that included pre-made videos by our partners.

  • These partnership building activities set the stage for, and eventually facilitated, the design process of a DMHI for teens to be incorporated into the library's Teen Services program because our research team had built strong working relationships and mutual trust with our community partners. In our experience, developing this strong partnership has been crucial for creating more equitable research dynamics and has improved the design process by facilitating mutual trust and enabling the design process to elicit and address the unique needs of the community and target end-users.

  • In this case study researchers and community partners evaluated fit and mutual benefit of partnering to address a shared goal. Additionally, this case study highlights methods of centering community expertise and a community partner's goals.

3.2. Understanding the problem or opportunity – investigate phase

The next key decision point is understanding the problem to solve or the design opportunity. This is where design research begins. The researcher must consider whether a DMHI is the appropriate designed outcome. Determining the answer can come from both formal and informal data sources and requires the researcher to embrace the partnership with the organization or community of interest. It is also important to assess constraints that might limit the efficacy and equitable delivery of a particular designed tool.

In Case Study 2, a researcher partnered with a mental health advocacy organization to understand their needs and the challenges the organization cares most about. The researcher's understanding of the organization's priorities evolved using iterative methods of inquiry.

  • This project builds on an existing partnership between our research center, CBITs, and Mental Health America (MHA), a mental health advocacy organization with a strong web presence. Our goals were to develop a digital tool for young adults that engage in non-suicidal self-injury (NSSI) that could be deployed to the individuals experiencing NSSI coming to the MHA website. To inform the design of such a tool, we first needed to understand the needs of young adults who engage in NSSI, their current self-management practices, and their interest in digital resources. We also needed to plan, with our partners, how this tool would be implemented by MHA to its constituents. Consequently, we conducted elicitation activities to identify needs, concerns, constraints, and preferences, with end-users and with MHA.

  • Most young adults do not seek treatment for NSSI so our understanding of how they self-manage NSSI behaviors is limited. To identify ways in which a digital tool may (or may not) help, we conducted one-on-one semi-structured interviews with 20 young adults to understand their NSSI-related goals and how they self-managed NSSI thoughts and behaviors. Interviews were chosen in place of a survey or focus group at this early stage because it is a method that affords free expression and the ability for researchers to flexibly follow-up on important insights while minimizing the number of people involved and, therefore, the need to worry about others' judgments.

  • Iterative HCD methods led the researchers to an increased understanding of design opportunities. For example, in interviews, some young adults discussed wanting to stop NSSI altogether, and others described goals to reduce the frequency or severity of NSSI over time (Kruzan et al., 2022b). The nuances arising from these interviews underscored the importance of the tool being tailored to young adults' readiness to change NSSI behavior, and incorporating language they were most comfortable with (e.g., “self-harm” instead of “NSSI”). The results were used to create app prototypes, which were shown to 10 additional young adults in individual interviews to gather feedback on specific features, desired ways of interacting, and content needs (Kruzan et al., 2022c). Between interviews, the prototypes were updated to reflect expressed needs and preferences. The prototypes were then used to create a requirements document for software developers.

  • We also explored young adults' needs with MHA, our community partner. We conducted an informal needs assessment through virtual meetings with MHA where we asked questions about MHA's mission and vision for the future, the needs they have observed in their visitors, and key areas for growth. We learned they had vested interest in diversifying their resources for NSSI.

  • We began to explore what MHA's infrastructure is like and what resources are available to integrate a digital tool. Through several meetings with our partners, we learned about MHA's workflows and existing resources, and how these related to their vision for expanding self-harm services. After considering several ways of deploying the digital tool through MHA (e.g., ads, additions to their Resource page) we recognized that while MHA has many self-report mental health screening tools on their website to assist people in accessing digital support tools, NSSI is a transdiagnostic problem that is not specifically assessed in any one screening. We determined collaboratively that a brief NSSI screening assessment was needed to identify and route interested individuals to NSSI tools. Thus, we worked with MHA leadership and young adult MHA website visitors to develop a screening tool that would conform to the structure of other screenings on the website and in a language appropriate for visitors.

  • Our team developed an initial set of items based on empirical evidence and theory, and we shared the items with MHA leadership in a Zoom session, where we collaborated on a Google doc and synchronously refined the language of items and response options based on consensus. Once our team and MHA partners were comfortable with the items, they were shared in focus groups (n = 12 participants) with MHA website visitors who had a history of NSSI. The focus group involved eliciting information on participants' reactions to items, clarifying language, and identifying items deemed to be too personal or arousing. Changes were made to reflect young adults' preferences, and when the initial screening tool was ultimately posted, over 21,000 visitors with experience of self-harm responded in just four months.

  • By engaging with our community partner and the population of interest, we have designed and developed a digital tool to increase access to information about NSSI, as well as developed a pathway for MHA visitors with NSSI to access that tool.

  • This case study highlights that by relying on the expertise of both community partners and target populations it is possible to identify addressable service gaps that might be appropriately served by digital health solutions. Additionally, this case study articulates the rationale for choosing particular research methods based on the research stage and open research questions.

3.3. Prototyping the product or service – ideate, prototype, evaluate, and refine and develop phases

Another potential decision point occurs when prototyping the DMHI. Researchers must determine the format of the DMHI and the affordances and drawbacks of different designs. For example, a text messaging based DMHI may be more accessible in regions where data access is limited but can be more rigid than app- or web-based digital interventions. Feedback from the end-users and organizations who will implement the DMHI will critically inform these decisions. Specific DMHI features and elements must also be considered, as well as the potential unintended harms that can be consequences of specific design choices (Pendse et al., 2022).

In Case Study 3 a researcher partnered with college-age young people and engaged them in visual co-design activities to adapt an existing DMHI to better fit their needs, goals, constraints and preferences.

  • For this project, we worked to adapt a DMHI (IntelliCare), originally designed for a general adult population, to meet the needs and preferences of college students. Adapting the intervention for college students required deciding on the clinical problems to focus on and the features to build. To make these decisions, we needed to understand the key clinical problems college students faced, and how a DMHI could begin to address them. We partnered with two local universities and began with a series of co-design workshops in which college students and college counseling center clinicians met jointly in a two-hour long workshop format (Cohen et al., 2022; Lattie et al., 2020a). Thirty college students and counseling center clinicians participated across four separate workshops (2 at each university). At each workshop, co-design participants engaged in three phases of work: (1) a warm-up activity to spark free thinking and creativity, (2) brainstorming and affinity diagramming, and (3) storyboarding.

  • For the warm-up, participants were tasked with a health focused exercise: mocking up an “enhanced bandage.” Participants were instructed to think as broadly as possible about how a bandage could be re-designed to be better. The goal was to get participants to begin thinking creatively about ideas for a health solution using a familiar prompt, and to increase their comfort in creative ideation. Then, researchers introduced the aim of the design workshop: helping design a tool that students can freely access to complete brief mental health assessments, receive feedback, and build upon their own stress management/coping skills by receiving access to interactive, clinical tools, and learn about mental health resources on campus.

  • Participants engaged in a two-part brainstorming activity. First, participants were given a stack of sticky notes and were instructed to spend 5 min writing down as many potential program features as they could think of. After 5 min, participants were prompted to share all of their ideas with the group and put their sticky notes up on a board. The timed independent brainstorm aimed to prevent the potential group dynamic in which some members are more participatory than others, and instead enable all members to equally contribute to the discussion. Second, we conducted affinity diagramming by grouping together the sticky notes with program features based on similarity (Harboe and Huang, 2015). This activity allowed for similar ideas and preferences to be rapidly elicited, and facilitated discussion of the meaning and value of proposed features and strategies. This allowed us to identify core needs for the clinical problems faced by college students and to identify solutions. For example, suggestions for an “on-demand chat feature” was elaborated upon to indicate that sometimes students just wanted to connect with a real human rather than be prompted with psychoeducation and self-management tools. While staffing a chat feature was outside of the scope of the project's budget, this suggestion prompted including information about on-demand hotlines and existing campus services that could meet those identified needs. Indeed, it is not uncommon for end-user recommendations to be outside the scope of what is feasible for a particular project, but eliciting these suggestions can uncover smaller-scale solutions that build towards the larger goal.

  • Following the brainstorming activity, participants engaged in a storyboarding exercise in which they were instructed to “tell a story of a hypothetical student's journey in using this mobile app, including: (1) how they learned about the mobile app, (2) why they decided to download it, (3) what types of mental health questions are asked, (4) what the feedback would look like, (5) what stress management/coping skills would be included, (6) when and in what context would they use it, (7) what information they got about on-campus services, and (8) whether or not they followed up with counseling center services and why.” This activity allowed the research team to understand the key problems that college students faced and to gain deeper insights into the social, environmental, and cultural context of college student life on campus.

  • Co-design activities conducted with college students and college counseling center clinicians facilitated adapting the existing DMHI to a population with unique needs within a specific context.

  • This case study highlights how iterative design ideation activities allowed nuanced features and content areas to be included in the DMHI adaptation and set up evaluative studies that could assess efficacy and acceptability.

Because designing a DMHI can follow different trajectories, we present another case study to highlight the varied paths tools can take depending on decisions such as the types of design activities used. In Case Study 4, a researcher iteratively prototyped and evaluated a text messaging based DMHI in partnership with young adult end-users.

  • This project is another collaboration with community partner Mental Health America (MHA), the community advocacy organization described above. MHA offers self-screening surveys to assess for mental health problems; however, despite high rates of detected mental health problems, the organization learned the surveys are accessed largely by non-treatment seeking young people. Therefore, MHA and CBITs collaborated to develop a DMHI to help non-treatment seeking young adults manage common mental health concerns. Consequently, we engaged young adults in an iterative design process.

  • First, we conducted asynchronous elicitation activities to understand end-users potential needs for a DMHI. Researchers and end-users engaged in collective discussions over approximately one month. Asynchronous methods were chosen to decrease barriers to research participation, especially among young people who often balance myriad temporal constraints. Our goal was to understand end-users' process of deciding whether to adopt a DMHI after self-screening, and to discuss ideas for how such a tool might support their self-management processes. We also sought feedback on automated text messaging as a DMHI delivery strategy, which we had identified as a promising option due to its low expense, accessibility (e.g., does not require a smartphone or data plan), and alignment with how young people use their phones. We convened two groups of young adults who responded to prompts in anonymous text-based discussion forums, based on asynchronous remote community methods (MacLeod et al., 2016, MacLeod et al., 2017). After agreeing to a code of conduct, participants created pseudonymous accounts on the study platform, FocusGroupIt.com, through which we released a new prompt every few days to which participants could respond. This type of forum can create conditions where individuals feel comfortable and unrushed sharing their perspectives and ideas, while also allowing for collaborative dynamics to emerge between the participants as they respond to, build on, and extend one another's ideas. We also engaged a subset of participants individually in phone interviews to follow-up on their responses and to gather reflections or ideas they did not share in a group setting.

  • Based on findings from this formative work, we generated examples of interactive dialogues that an automated text messaging DMHI could have with users. We conducted synchronous design workshops, remotely using Zoom, with several end-users who had previously participated in the asynchronous remote community elicitation workshops to get feedback on these initial dialogues, and to engage them in proposing improvements and alternatives. These workshops allowed us to refine the content, as well as provided guidance on message tone, scheduling, gathering feedback and context (e.g., message ratings, mood), interactivity, and personalization (Bhattacharjee et al., 2022, Bhattacharjee et al., 2023; Kornfield et al., 2022a; Kornfield et al., 2022b). Based on our formative work and other design work with young people (Kornfield et al., 2022a; Kornfield et al., 2022b; Liverpool et al., 2020; Ranney et al., 2014), we gathered input on both active and passive personalization, with active personalization involving direct customization of schedules and strategies, whereas passive personalization involved integrating reinforcement learning (a class of machine learning) algorithms that prospectively adapt content based on user engagement.

  • These design and prototyping activities enabled us to build a non-functional abbreviated prototype of the intervention, which we assessed for usability in a series of 1- and 2-week field trials with 42 young adults. We tested a non-functional prototype by implementing the Wizard-of-Oz design method, which involves a researcher playing the part of an automated system (Dahlbäck et al., 1993; Maulsby et al., 1993). In our case, a research team member manually sent text messages and processed user responses based on a detailed script (Bhattacharjee et al., 2023). This method facilitated getting the feedback that enabled the researchers to refine the dialogues and the DMHI logic without committing development resources and incurring technical debt.

  • The intervention, Small Steps SMS, was subsequently developed by a software development company, and delivers daily interactive text messaging dialogues over eight weeks to help users learn and apply psychological strategies from eclectic psychotherapy orientations (e.g., cognitive-behavioral therapy, acceptance and commitment therapy). Beginning in September of 2022, MHA publicly deployed an early version of this program reaching over 4000 users so far (August of 2023), allowing us to further optimize the system and improve the machine learning algorithms. Small Steps SMS will next be evaluated through efficacy trials.

    This case study illustrates processes of iterative user feedback and how findings from formative work translate into concrete design decisions when prototyping. Finally, this case study details two methods of eliciting prototype feedback from end-users (i.e., Wizard-of-Oz field trials and large open deployments) and the type of feedback each method enabled.

3.4. Testing the DMHI in practice – design validate phase

Once the DMHI is built, it is time to test it in practice. While the randomized controlled trial (RCT) remains the gold standard for the evaluation of tools, some modifications to methodology are needed for digital technologies. Unlike a pharmacologic agent, which is static and would remain the same for tens of millions of people over decades, the nature of digital technologies is change. They are continuously updated and changed to respond to changes in user preferences, new technologies and methods, and lessons learned from prior users. To account for this characteristic of digital technologies, RCTs of DMHIs can allow changes during the trial to correct problems and improve design (Mohr et al., 2015). When the DMHI is fixed and locked during the RCT, changes are inevitably made subsequent to the trial based on lessons learned in the trial and on the needs for the next deployment. It is both more efficient and more consistent with digital technologies to incorporate learnings acquired during trials into the DMHI.

The primary issue that must be addressed in allowing the DMHI to change during an RCT is the threat to internal validity. More specifically, if the thing being tested changes during an RCT, what is an RCT evaluating? Because a DMHI will continue to change over time (regardless of whether it is held constant during the RCT or not), the value of the RCT is not in testing a product, rather it is in evaluating a set of principles. Therefore, ensuring internal validity requires that the principles being tested remain constant, rather than the product itself, which will change. The Trials of Intervention Principles (TIPs) methodology (Mohr et al., 2015), provides a methodological framework and reporting standards to ensure internal validity and transparency. TIPs requires a principle statement that clarifies the clinical purpose of the DMHI, and a detailed description of the behavioral, design, and implementation principles being tested. Confining any modifications within these constraints allows meaningful and valid interpretations to be extracted from the RCT results regarding the defined principles. The effects of changes made over the course of the trial can be objectively evaluated. Transparency is created by reporting all changes in trial results, consistent with the EHEALTH CONSORT Guidelines (Eysenbach and CONSORT-EHEALTH Group, 2011).

Case Study 5 illustrates how design methods that were applied during a randomized clinical pilot trial (design validation phase) yielded ongoing iteration to a DMHI.

  • FoodSteps is a DMHI designed to address binge eating and weight management. Substantial formative HCD research with end-users informed its design (Graham et al., 2023), resulting in a 16-week DMHI whose primary focus is to empower users to set and track a goal each week to facilitate behavior change over time.

  • As participants progressed through the trial, clinical observations of users' DMHI data and text message communications with their coach as well as design interviews helped the research team understand users' preferences and recommendations regarding the intervention's design. Early data from these methods were showing that users were not practicing prior weeks' goals once they moved onto a new week. This was a design flaw. Clinically, we intended for users to keep practicing previously set goals in subsequent weeks, like they would in a typical intervention where behavior change is scaffolded over time. However, we realized the DMHI lacked a feature to facilitate continued practice. Because behavior change rarely works successfully after only one week, this oversight had potential to hinder users' clinical progress.

  • The researchers and software developer used participants' feedback to design how to enhance FoodSteps' goal-setting feature to more effectively facilitate this behavior change principle. We ideated and created a wireframe prototype in which, in addition to setting a goal each week, users would have the option of also selecting prior week's goals to continue to practice in their current week. The researchers solicited feedback on this feature from new users during design feedback interviews, which reinforced the decision to add this feature. Within a few weeks, the new feature was implemented.

  • We note that design activities during the trial yielded insights and user recommendations for other features that were outside the scope of our pre-defined behavior change principles that were being tested in this trial. These features were not considered for implementation in the current trial, but were saved for consideration in future projects with an expanded pre-defined scope.

  • Our new feature was “validated” by clinical observations showing subsequent users consistently using it. Design interviews also generated new learnings: the new feature, while helpful, was limited in that it did not allow users to track their progress completing the prior weeks' goals they selected. This created confusion for what users should be monitoring and marking progress on, as well as frustrations that they could not use the tool to monitor progress across multiple domains. We used the feedback interviews to engage users in ideating and prototyping an updated design. This update will be deployed in an upcoming trial.

  • This case study demonstrates the importance of continually seeking feedback from users even after the DMHI is deployed to users.

4. Discussion

In this manuscript we aimed to explicate the processes researchers in our center have adopted to design and develop DMHIs using HCD. The case studies show how HCD phases and decision points were navigated when developing service ready DMHIs. They also illustrate how the design process is intentionally open-ended and flexible to accommodate different scenarios, solutions, and sociotechnical environments. However, this open-endedness can be a barrier to building and sustaining service-ready tools in the absence of a framework to evaluate the different decisions that arise across the design-to-product pipeline. To reduce this barrier and facilitate evaluation of key decision points, we present questions that arose in our work, examples of matched-HCD methods that may uncover useful insights when creating service-ready DMHIs, as well as exemplar references that describe or illustrate the HCD methods (Table 1).

Table 1.

Key decision points, questions to consider, and HCD methods to facilitate progress in the DMHI design-to-product pipeline.

Key decision point (design phase) Questions to consider Example HCD methods (exemplar references that describe or illustrate the HCD method)a
Partnership building
  • •

    What kind of organization(s) are you partnering with as the deployment setting?

  • •

    What are the organization's missions and values? How does it align with the researcher's goals?

  • •

    How does the organization further the priorities of its constituents?

  • •

    What does the organization and its constituents view as priorities that should be the focus of a partnership?

  • •

    If a good fit, what are the organization's short- and long-term expectations of the partnership?

  • •

    What are the products community partners deem beneficial to the community from the research and partnership?
Understanding the problem or opportunity (investigate)
  • •
    What is the primary priority of the target user group and/or setting?
    • ◦
      What is the design opportunity?
    • ◦
      How might a DMHI/digital solution help address the primary priority?
  • •

    What tool or service will address the problem or opportunity and have a positive impact?

  • •

    Are there constraints that would limit the efficacy and equitable delivery of a particular designed solution?

  • •

    What is the environmental and digital infrastructure like?

  • •

    What resources are available for the community to adopt/integrate/sustain the DMHI?

  • •

    What are the community/organization strengths?

  • •

    Are there relevant legal, regulatory, or ethical considerations?

  • •

    Do the data you plan to collect or omit have implications for the kinds of targets the intervention will address?

  • •

    What are the workflows in the organization(s) that we need to account for when designing or implementing the DMHIs?

  • •

    What are the determinants that may affect implementation of the DMHI and strategies that address those determinants?
Prototyping the product or service (ideate, prototype, evaluate, and refine & develop)
  • •

    What is the ideal format of, or platform for, the DMHI?

  • •
    What kinds of use pathways exist when using the DMHI?
    • ◦
      What journey does a user follow as they use the DMHI?
  • •

    What design elements would be meaningful and usable to users?

  • •

    What potential harms could be consequences of specific design elements or the DMHI?
Testing the DMHI in practice (design validate)
  • •
    Does the product or service work the way we expect?
    • ◦
      Is it effective, and relative to what exists already?
    • ◦
      Does it improve access?
    • ◦
      Is it feasibly implemented into the deployment setting and acceptable?
  • •

    How do people actually use the DMHI?

  • •

    Were gaps or considerations missed during the design process?

  • •

    How well do the implementation strategies for the DMHI address implementation determinants?

  • •

    Does the product gain reach beyond the lab to make an impact?

  • •

    Are there iatrogenic effects of the DMHI once deployed?
Open in a new tab
a

Note: the HCD methods included in this table (as well as exemplar references) are not exhaustive and are included to illustrate useful examples and applications of these methods to digital mental health intervention design and testing. Additionally, we refer the reader to Martin and Hanington (2012) for an excellent resource on many of the methods listed.

4.1. Limitations

Strengths of this manuscript are the use of diverse case studies to illustrate the application of HCD in clinical research. However, this manuscript is limited in that we present work exclusively from our center and thus represent only one group's approach to this work. Additionally, the field of HCD is quite broad, and, as Table 1 shows, other HCD techniques can be deployed along the design-to-product pipeline than we have captured in this paper.

5. Conclusion

In this manuscript we present ways researchers in our center have navigated key decision points that frequently arise when developing a service-ready product. We offer concrete case studies to make transparent how and why many of those decisions were made. Our intent is that by sharing these case studies, we contribute to the evolving discourse on how to partner with communities, effectively design and improve service ready DMHIs to increase access to care and provide researchers with a framework to guide their work across key decision points.

Funding

This work was supported by grants from the National Institute of Mental Health [Grants: P50MH119029, K01MH125172, R34MH124960, K08MH128640, K08MH112878, K01MH121854, T32MH115882, R34MH128410], Grants from the National Institute of Diabetes and Digestive and Kidney Diseases [Grant: K01DK116925], and a seed grant from Northwestern University's Alliance for Research in Chicagoland Communities.

Declaration of competing interest

Jonah Meyerhoff has accepted consulting fees from Boehringer Ingelheim and Shirley Ryan Ability Lab. David C. Mohr has accepted honoraria and consulting fees from Otsuka Pharmaceuticals, Optum Behavioral Health, Centerstone Research Institute, and the One Mind Foundation, royalties from Oxford Press, and has an ownership interest in Adaptive Health, Inc. Emily Lattie and Miranda Beltzer are currently employed by Lyra Health. Ashley Knapp has accepted consulting fees from Woebot, Inc. Caitlin A. Stamatis has received salary and equity support from Akili Interactive Labs and Google. Bayley Taple is currently employed by BehaVR, Inc. Madhu Reddy has accepted consulting fees from Boehringer Ingelheim.

Acknowledgements

Special thanks and shoutout to our library worker and teen patron partners at the Oak Park Public Library. It has truly been a joy working alongside you all, and we are excited for what is to come! We would also like to acknowledge our colleagues at Mental Health America, including Theresa Nguyen and Kevin Rushton for their contributions to the tools and services presented here and their ongoing support. Additionally, we thank our colleagues who helped us design, build and test the Small Steps SMS intervention including Joseph J. Williams, Ananya Bhattacharjee, Harsh Kumar, Bei Pang, Jehan Vakharia, Veronica Bergstrom, Justice Tomlinson, Karrie Chou, Alvina Lai, Arissa Du, Yu-Chun Chien, Helena Jovic, Angelina Liu, Elexandra Tran, Samuel Maldonado, Tong Li, Jiakai Shi, Charles Krause, Sarah Popowski, and Chris J. Karr.

References

  1. Alliance for Research in Chicagoland Communities (ARCC) Northwestern University Alliance for Research in Chicagoland Communities (ARCC) Resources Directory. Resources Directory. 2023. https://www.feinberg.northwestern.edu/sites/cch/resources-directory/index.html
  2. Berry A.B.L., Lim C.Y., Hartzler A.L., Hirsch T., Ludman E., Wagner E.H., Ralston J.D. “It’s good to know you’re not a stranger every time”: communication about values between patients with multiple chronic conditions and healthcare providers. Proc. ACM Hum. Comput. Interact. 2017;1(CSCW):23:1–23:20. doi: 10.1145/3134658. [DOI] [Google Scholar]
  3. Berry A.B.L., Lim C.Y., Hirsch T., Hartzler A.L., Kiel L.M., Bermet Z.A., Ralston J.D. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 2019. Supporting communication about values between people with multiple chronic conditions and their providers; pp. 1–14. [DOI] [Google Scholar]
  4. Bhattacharjee A., Williams J.J., Chou K., Tomlinson J., Meyerhoff J., Mariakakis A., Kornfield R. “I kind of bounce off it”: translating mental health principles into real life through story-based text messages. Proc. ACM Hum. Comput. Interact. 2022;6(CSCW2):398:1–398:31. doi: 10.1145/3555123. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bhattacharjee A., Williams J.J., Meyerhoff J., Kumar H., Mariakakis A., Kornfield R. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023. Investigating the role of context in the delivery of text messages for supporting psychological wellbeing; pp. 1–19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Cohen K.A., Graham A.K., Lattie E.G. Aligning students and counseling centers on student mental health needs and treatment resources. J. Am. Coll. Heal. 2022;70(3):724–732. doi: 10.1080/07448481.2020.1762611. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Curran G.M., Landes S.J., McBain S.A., Pyne J.M., Smith J.D., Fernandez M.E., Chambers D.A., Mittman B.S. Reflections on 10 years of effectiveness-implementation hybrid studies. Front. Health Serv. 2022;2 doi: 10.3389/frhs.2022.1053496. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Dahlbäck N., Jönsson A., Ahrenberg L. Proceedings of the 1st International Conference on Intelligent User Interfaces - IUI ’93. 1993. Wizard of Oz studies: why and how; pp. 193–200. [DOI] [Google Scholar]
  9. De Vito Dabbs A., Myers B.A., Mc Curry K.R., Dunbar-Jacob J., Hawkins R.P., Begey A., Dew M.A. User-centered design and interactive health technologies for patients. Comput. Inform. Nurs. 2009;27(3):175–183. doi: 10.1097/NCN.0b013e31819f7c7c. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Eysenbach G., CONSORT-EHEALTH Group CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions. J. Med. Internet Res. 2011;13(4) doi: 10.2196/jmir.1923. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Faily S., Flechais I. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2011. Persona cases: a technique for grounding personas; pp. 2267–2270. [DOI] [Google Scholar]
  12. Graham A.K., Wildes J.E., Reddy M., Munson S.A., Barr Taylor C., Mohr D.C. User-centered design for technology-enabled services for eating disorders. Int. J. Eat. Disord. 2019;52(10):1095–1107. doi: 10.1002/eat.23130. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Graham A.K., Greene C.J., Kwasny M.J., Kaiser S.M., Lieponis P., Powell T., Mohr D.C. Coached mobile app platform for the treatment of depression and anxiety among primary care patients: a randomized clinical trial. JAMA Psychiatry. 2020 doi: 10.1001/jamapsychiatry.2020.1011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Graham A.K., Kosmas J.A., Massion T.A. Designing digital interventions for eating disorders. Curr. Psychiatry Rep. 2023;25(4):125–138. doi: 10.1007/s11920-023-01415-x. [DOI] [PubMed] [Google Scholar]
  15. Harboe G., Huang E.M. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2015. Real-world affinity diagramming practices: bridging the paper-digital gap; pp. 95–104. [DOI] [Google Scholar]
  16. Harrington C., Erete S., Piper A.M. Deconstructing community-based collaborative design: towards more equitable participatory design engagements. Proc. ACM Hum. Comput. Interact. 2019;3(CSCW):1–25. doi: 10.1145/3359318. [DOI] [Google Scholar]
  17. Hayes G.R. In: Ways of Knowing in HCI. Olson J.S., Kellogg W.A., editors. Springer; 2014. Knowing by doing: action research as an approach to HCI; pp. 49–68. [DOI] [Google Scholar]
  18. Jacobs M., Hopkins J., Mumber M., Mynatt E. AMIA Annual Symposium Proceedings, 2019. 2020. Usability evaluation of an adaptive information recommendation system for breast cancer patients; pp. 494–503. [PMC free article] [PubMed] [Google Scholar]
  19. Kaikkonen A., Kekäläinen A., Cankar M., Kallio T., Kankainen A. Usability testing of mobile applications: a comparison between laboratory and field testing. J. Usability Stud. 2005;1(1):4–16. [Google Scholar]
  20. Knapp A.A., Cohen K., Nicholas J., Mohr D.C., Carlo A.D., Skerl J.J., Lattie E.G. Integration of digital tools into community mental health care settings that serve young people: focus group study. JMIR Ment. Health. 2021;8(8) doi: 10.2196/27379. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Knapp A.A., Carroll A.J., Mohanty N., Fu E., Powell B.J., Hamilton A., Burton N.D., Coldren E., Hossain T., Limaye D.P., Mendoza D., Sethi M., Padilla R., Price H.E., Villamar J.A., Jordan N., Langman C.B., Smith J.D. A stakeholder-driven method for selecting implementation strategies: a case example of pediatric hypertension clinical practice guideline implementation. Implement. Sci. Commun. 2022;3(1):25. doi: 10.1186/s43058-022-00276-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Knapp A.A., Hersch E., Wijaya C., Herrera M.A., Kruzan K.P., Carroll A.J., Lee S., Baker A., Gray A., Harris V., Simmons R., Kour Sodhi D., Hannah N., Reddy M., Karnik N.S., Smith J.D., Brown C.H., Mohr D.C. “The library is so much more than books”: considerations for the design and implementation of teen digital mental health services in public libraries. Front. Digit. Health. 2023;5 doi: 10.3389/fdgth.2023.1183319. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Kornfield R., Meyerhoff J., Studd H., Bhattacharjee A., Williams J.J., Reddy M., Mohr D.C. CHI Conference on Human Factors in Computing Systems. 2022. Meeting users where they are: user-centered design of an automated text messaging tool to support the mental health of young adults; pp. 1–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kornfield R., Mohr D., Meyerhoff J., Ranney R., Williams J., Lattie E., Reddy M. Involving Crowdworkers with lived experience in content-development for push-based digital mental health tools: lessons learned from crowdsourcing mental health messages. Proc. ACM Hum. Comput. Interact. 2022;6(CSCW1):1–30. doi: 10.1145/3512946. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Kruzan K.P., Meyerhoff J., Nguyen T., Reddy M., Mohr D.C., Kornfield R. CHI Conference on Human Factors in Computing Systems. 2022. “I wanted to see how bad it was”: online self-screening as a critical transition point among young adults with common mental health conditions; pp. 1–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kruzan K.P., Mohr D.C., Reddy M. How technologies can support self-injury self-management: perspectives of young adults with lived experience of nonsuicidal self-injury. Front. Digit. Health. 2022:4. doi: 10.3389/fdgth.2022.913599. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Kruzan K.P., Reddy M., Washburn J.J., Mohr D.C. Developing a mobile app for young adults with nonsuicidal self-injury: a prototype feedback study. Int. J. Environ. Res. Public Health. 2022;19(23) doi: 10.3390/ijerph192316163. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kruzan K.P., Ng A., Stiles-Shields C., Lattie E.G., Mohr D.C., Reddy M. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023. The perceived utility of smartphone and wearable sensor data in digital self-tracking technologies for mental health; pp. 1–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Lattie E.G., Kornfield R., Ringland K.E., Zhang R., Winquist N., Reddy M. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020. Designing mental health technologies that support the social ecosystem of college students; pp. 1–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Lattie E.G., Nicholas J., Knapp A.A., Skerl J.J., Kaiser S.M., Mohr D.C. Opportunities for and tensions surrounding the use of technology-enabled mental health services in community mental health care. Adm. Policy Ment. Health Ment. Health Serv. Res. 2020;47(1):138–149. doi: 10.1007/s10488-019-00979-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Lattie E.G., Burgess E., Mohr D.C., Reddy M. Care managers and role ambiguity: the challenges of supporting the mental health needs of patients with chronic conditions. Comput. Support. Coop. Work. 2021;30(1):1–34. doi: 10.1007/s10606-020-09391-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Le Dantec C.A., Fox S. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. 2015. Strangers at the gate: gaining access, building rapport, and co-constructing community-based research; pp. 1348–1358. [DOI] [Google Scholar]
  33. LeRouge C., Ma J., Sneha S., Tolle K. User profiles and personas in the design and development of consumer health technologies. Int. J. Med. Inform. 2013;82(11):e251–e268. doi: 10.1016/j.ijmedinf.2011.03.006. [DOI] [PubMed] [Google Scholar]
  34. Liverpool S., Mota C.P., Sales C.M.D., Čuš A., Carletto S., Hancheva C., Sousa S., Cerón S.C., Moreno-Peral P., Pietrabissa G., Moltrecht B., Ulberg R., Ferreira N., Edbrooke-Childs J. Engaging children and young people in digital mental health interventions: systematic review of modes of delivery, facilitators, and barriers. J. Med. Internet Res. 2020;22(6) doi: 10.2196/16317. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Lu A.J., Sannon S., Brewer S., Jackson K.N., Green J., Reeder D., Wafer C., Dillahunt T.R. Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 2023. Organizing community-based events in participatory action research: lessons learned from a photovoice exhibition; pp. 1–8. [DOI] [Google Scholar]
  36. MacLeod H., Jelen B., Prabhakar A., Oehlberg L., Siek K., Connelly K. 10th EAI International Conference on Pervasive Computing Technologies for Healthcare. 2016. Asynchronous remote communities (arc) for researching distributed populations. [Google Scholar]
  37. MacLeod H., Jelen B., Prabhakar A., Oehlberg L., Siek K., Connelly K. A guide to using asynchronous remote communities (ARC) for researching distributed populations. EAI Endorsed Trans. Pervasive Health Technol. 2017;3(11) doi: 10.4108/eai.18-7-2017.152898. [DOI] [Google Scholar]
  38. Makosky Daley C., James A.S., Ulrey E., Joseph S., Talawyma A., Choi W.S., Greiner K.A., Coe M.K. Using focus groups in community-based participatory research: challenges and resolutions. Qual. Health Res. 2010;20(5):697–706. doi: 10.1177/1049732310361468. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Martin B., Hanington B.M. Rockport Publishers; 2012. Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions (Digital ed) [Google Scholar]
  40. Maulsby D., Greenberg S., Mander R. Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems. 1993. Prototyping an intelligent agent through Wizard of Oz; pp. 277–284. [DOI] [Google Scholar]
  41. Millen D.R. Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques. 2000. Rapid ethnography: time deepening strategies for HCI field research; pp. 280–286. [DOI] [Google Scholar]
  42. Mohr D.C., Schueller S.M., Riley W.T., Brown C.H., Cuijpers P., Duan N., Kwasny M.J., Stiles-Shields C., Cheung K. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J. Med. Internet Res. 2015;17(7) doi: 10.2196/jmir.4391. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Mohr D.C., Lyon A.R., Lattie E.G., Reddy M., Schueller S.M. Accelerating digital mental health research from early design and creation to successful implementation and sustainment. J. Med. Internet Res. 2017;19(5) doi: 10.2196/jmir.7725. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Mohr D.C., Azocar F., Bertagnolli A., Choudhury T., Chrisp P., Frank R., Harbin H., Histon T., Kaysen D., Nebeker C., Richards D., Schueller S.M., Titov N., Torous J., Areán P.A. Banbury forum consensus statement on the path forward for digital mental health treatment. Psychiatr. Serv. 2021 doi: 10.1176/appi.ps.202000561. appi.ps.202000561. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Muller M., Kogan S. 2010. Grounded Theory Method in HCI and CSCW. [Google Scholar]
  46. Newman S.D., Andrews J.O., Magwood G.S., Jenkins C., Cox M.J., Williamson D.C. Community advisory boards in community-based participatory research: a synthesis of best processes. Prev. Chronic Dis. 2011;8(3):A70. [PMC free article] [PubMed] [Google Scholar]
  47. Pendse S.R., Nkemelu D., Bidwell N.J., Jadhav S., Pathare S., De Choudhury M., Kumar N. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022. From treatment to healing: envisioning a decolonial digital mental health; pp. 1–23. [DOI] [Google Scholar]
  48. Powell R.A., Single H.M., Lloyd K.R. Focus groups in mental health research: enhancing the validity of user and provider questionnaires. Int. J. Soc. Psychiatry. 1996;42(3):193–206. doi: 10.1177/002076409604200303. [DOI] [PubMed] [Google Scholar]
  49. Proctor E.K., Toker E., Tabak R., McKay V.R., Hooley C., Evanoff B. Market viability: a neglected concept in implementation science. Implement. Sci. 2021;16(1):98. doi: 10.1186/s13012-021-01168-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Ranney M.L., Choo E.K., Cunningham R.M., Spirito A., Thorsen M., Mello M.J., Morrow K. Acceptability, language, and structure of text message-based behavioral interventions for high-risk adolescent females: a qualitative study. J. Adolesc. Health. 2014;55(1):33–40. doi: 10.1016/j.jadohealth.2013.12.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Rose A., Shneiderman B., Plaisant C. Proceedings of the 1st Conference on Designing Interactive Systems: Processes, Practices, Methods, & Techniques. 1995. An applied ethnographic method for redesigning user interfaces; pp. 115–122. [DOI] [Google Scholar]
  52. Ryu H., Berry A.B.L., Lim C.Y., Hartzler A., Hirsch T., Trejo J.I., Bermet Z.A., Crawford-Gallagher B., Tran V., Ferguson D., Cronkite D.J., Tiffany B., Weeks J., Ralston J. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023. “You can see the connections”: facilitating visualization of care priorities in people living with multiple chronic health conditions; pp. 1–17. [DOI] [Google Scholar]
  53. Stiles-Shields C., Cummings C., Montague E., Plevinsky J.M., Psihogios A.M., Williams K.D.A. A call to action: using and extending human-centered design methodologies to improve mental and behavioral health equity. Front. Digit. Health. 2022;4:848052. doi: 10.3389/fdgth.2022.848052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Thach K.S., Lederman R., Waycott J. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023. Key considerations for the design of technology for enrichment in residential aged care: an ethnographic study; pp. 1–16. [DOI] [Google Scholar]

Articles from Internet Interventions are provided here courtesy of Elsevier

RESOURCES