Skip to main content
NCBI home page
UKPMC Funders Author Manuscripts logoLink to UKPMC Funders Author Manuscripts
. Author manuscript; available in PMC: 2021 Nov 10.
Published in final edited form as: J Child Psychol Psychiatry. 2020 Jun 22;61(8):928–940. doi: 10.1111/jcpp.13258

Practitioner review: Co-design of digital mental health technologies with children and young people

Rhys Bevan Jones 1,2,3,✉, Paul Stallard 4, Sharifah Shameem Agha 1,2,3, Simon Rice 5,6, Aliza Werner-Seidler 7, Karolina Stasiak 8, Jason Kahn 9, Sharon A Simpson 10, Mario Alvarez-Jimenez 5,6, Frances Rice 1,2, Rhiannon Evans 11, Sally Merry 8
PMCID: PMC7611975  EMSID: EMS137683  PMID: 32572961

Abstract

Background

There is increasing interest in digital technologies to help improve children and young people’s mental health, and the evidence for the effectiveness for these approaches is rising. However, there is concern regarding levels of user engagement, uptake and adherence. Key guidance regarding digital health interventions stress the importance of early user input in the development, evaluation and implementation of technologies to help ensure they are engaging, feasible, acceptable and potentially effective. Co-design is a process of active involvement of stakeholders, requiring a change from the traditional approaches to intervention development. However, there is a lack of literature to inform the co-design of digital technologies to help child and adolescent mental health.

Methods

We reviewed the literature and practice in the co-design of digital mental health technologies with children and young people. We searched Medline, PsycInfo and Web of Science databases, guidelines, reviews and reference lists, contacted key authors for relevant studies, and extracted key themes on aspects of co-design relevant to practice. We supplemented this with case studies and methods reported by researchers working in the field.

Results

We identified 25 original articles and 30 digital mental health technologies that were designed/developed with children and young people. The themes identified were as follows: principles of co-design (including potential stakeholders and stages of involvement), methods of involving and engaging the range of users, co-designing the prototype and the challenges of co-design.

Conclusions

Co-design involves all relevant stakeholders throughout the life and research cycle of the programme. This review helps to inform practitioners and researchers interested in the development of digital health technologies for children and young people. Future work in this field will need to consider the changing face of technology, methods of engaging with the diversity in the user group, and the evaluation of the co-design process and its impact on the technology.

Keywords: Child, adolescent, mental health, digital, technologies, e-health, development, co-design

Introduction

Mental health difficulties are common in children and young people (CYP), but most are not getting any help (Neufeld et al., 2017). Digital mental health technologies (i.e. resources and interventions to support and improve mental health) have been identified as a potential way to improve reach and access to therapies, at relatively low cost. There is growing evidence to support the use of some technologies (Hollis et al., 2017), with guidelines recommending, for example, digital cognitive behavioural therapy (CBT) for depression (National Institute for Health and Care Excellence., 2019). Furthermore, many CYP have access to the internet and mobile technologies, including in low and middle-income countries (LMIC; Naslund et al., 2017). However, a major challenge in this area is the low user engagement, uptake and adherence to these programmes outside research settings (Fleming et al., 2019; Hollis et al., 2017).

Over recent years, there has been an increase in the publication of general guidance for intervention development. Guidance specific to digital health (e.g. Mohr et al., 2014; Yardley et al., 2015) reflects a broader direction of travel in the development and evaluation of complex health interventions, which stress the importance of the development phase and of user input from the initial stages (Craig et al., 2008; Hawkins et al., 2017; Wight et al., 2016). Co-design is a process of ‘collective creativity’ or ‘partnership’ with potential users and stakeholders, who are actively involved across the entire development of the technology - helping to ensure it meets the user’s needs and preferences. A rigorous process involving users, relevant theory and research evidence is more likely to produce an intervention that is evidence-based, engaging, acceptable and feasible to deliver (Thabrew et al., 2018).

Whilst the principle of co-design of digital health technologies is becoming accepted, there is limited guidance and literature on how this can optimally be undertaken. Co-design requires a shift from the traditional practice of expert-led development work where interventions are designed ‘for’ to one where they are designed ‘with’ CYP (Hodson et al., 2019). Designing and producing with CYP helps to ‘humanise’ the field of digital technologies which some have criticised for being overly structured, rigid and unresponsive.

The development of technologies for child and adolescent mental health requires particular considerations, and they should not merely be adaptations from ‘adult programmes’. A developmental or age-appropriate approach is needed regarding the content and design of a programme, and accounting for the range of interests and tastes of CYP. The presentation and management of mental health difficulties in this age group are also different to that of adults (WHO, 1993).

We provide a practitioner review of the literature on the approaches to the design and development of digital mental health technologies in collaboration with CYP and other stakeholders. We will map the existing evidence and practice for the co-design with CYP and use case studies and exemplars to illustrate key points throughout. The review offers an overview of an emerging research area to practitioners and researchers and concludes with practice points to help guide the planning, reporting and analysis of co-design activities.

Methods

Relevant articles were identified through computer searches in Medline, PsycInfo and Web of Science databases to July 2019, with no restriction regarding publication dates. The key search terms and methodology are outlined in Figure S1. Studies were appraised against the following inclusion criteria: articles with information on the (co-) design/development/production, of digital mental health technologies with and for CYP (up to 18 years); papers published or translated into English in a peer-reviewed journal. There was a focus on programmes/ applications to help with depression, anxiety, sleep, self-harm and suicide. We searched reviews, guidelines and reference lists, and contacted key authors with expertise in the development of digital mental health interventions for CYP, especially where it was unclear whether CYP were involved in the design/ development of certain technologies. Papers were excluded if technologies were developed for adults or primarily for physical health, or were diagnostic, screening, monitoring, communication or data management tools.

Titles and abstracts, and then full texts, were screened by RBJ and SSA. As this was an exploratory and descriptive review, an inclusive approach was taken to assessing full texts. Study quality was not appraised, as we aimed to map the emerging literature (rather than verifying an evidence base of effects) given the heterogeneous nature of the field. Data were extracted by RBJ and SSA on the co-design processes for a range of technologies and the findings were discussed with other authors. The data were categorised by the two authors into key overarching and recurring themes that help to understand the practical processes of involving CYP in the co-design of technologies. These themes were illustrated through case studies on specific technologies. The ‘Practice points’ section was based on the review findings and our methods, and reflections on future developments as researchers working in the field.

Results

Summary of studies

The original searches yielded 5891 articles after duplicates were removed, and 292 articleswere assessed from full text (see Figure S1). We identified 25 original articles and 30 digital mental health technologies that met the inclusion criteria (summarised in Table 1). Fifteen of the technologies (50%) were developed to help with several mental health difficulties, with eighteen of the technologies targeting depression (60%), nine targeting anxiety (30%), three targeting self-harm or suicidal ideation (10%), one targeting sleep (3%), and six helping with ‘general mental health’ difficulties or crises (20%). Twenty-nine technologies were developed with adolescents (97%), whilst five involved children (under 12 years; 17%).

Table 1. Digital mental health technologies for CYP and their development approaches.

Digital programme (Authors, countiy) MH difficulties Stakeholder involvement (ages of CYP in years, if stated) Technological approaches Psychological & other theories
Adventures of DoReMiFa (Shum et al., 2019; Hong Kong) Anxiety, general MH FGs - children (8-12), parents, teachers, practitioners Gamification, storytelling CBT, positive psychology
LifeBuoy (Han et al., 2019; AWS-authora; Australia) Suicidal thoughts Surveys/FGs - YP (16-25) Interactive modules, gamification DBT, ACT
Mellow (Hodson et al., 2019; Canada) General MH, crises Interviews/workshops/surveys (‘design charrettes/jams’ - YP (13-24), friends, families/carers, practitioners Joumal/planning companion tool Holistic crisis planning
Quest-Te Whitianga (Christie et al., 2019; Fleming et al., 2019; SM, KS-authorsa; New Zealand) Anxiety, depression Interviews/FGs/workshops (‘wall storms’/ Think-Alouds’ - YP (12-25), designers, practitioners Modular activities, gamification CBT, positive psychology, mindfulness, interpersonal skills
Bluelce (Stallard et al., 2018; PS-authora; England) Self-harm Meetings/workshops - YP (12-17), practitioners, developers Mood diary, mood-lifting activities, safety checks CBT, DBT
HABITs (Thabrew et al., 2018; SM-authora; New Zealand) Emotional health, substance use Surveys/FGs - YP, practitioners, cultural advisors Different user groups: games, chatbots, intrinsic motivators; digital eco-system CBT, positive psychology, harm minimisation
MoodHwb (Bevan Jones, Thapar, Rice, et al., 2018; RBJ, FR, SSA, PS, SM, SAS-authorsa; Wales) Depression Interviews/workshop/FGs - YP (13-19), parents/carers, practitioners, designers Illustrations / animations, profile-builder, moodmonitor, goal-setting Psychoeducation, CBT, social support
Rebound (Rice et al., 2018; SR, MAJ-authorsa; Australia) Depression Workshops/FGs/consultations - YP (15-25), families, professionals, writers/artists, designers Social media-enabled platform CBT, mindfulness, positive psychology, social support
SOVA (Radovic et al., 2018; USA) Depression, anxiety Interviews (‘Think-Alouds’/FGs - YP (13-26), parents, advocates, professionals Moderated social media Social support, psychoeducation
BeSafe (Huggett et al., 2017; Canada) General MH, addictions, crises ‘Design studio’/meetings - YP, practitioners Navigation, safety plans, decision aid Empowerment, social support
Sleep Ninja (Werner-Seidler et al., 2017; AWS-authora; Australia) Sleep, depression Interviews/FGs/consultations - YP (12-16), parents, professionals, designers Chatbot, gamification CBT-I
Social media messages (Robinson et al., 2017; Australia) Suicidal thoughts Closed social media/surveys/workshops - YP (16-18), creative agency Social media messages/videos Psychoeducation, social support
Thought Spot (Wiljer et al., 2017; Canada) General MH ‘Crowdsourcing’/ ‘hackathon’/ workshops, FGs - YP (15-24), practitioners, designers Information sharing, networking Peer/social support
SPARX (Shepherd et al., 2015; SM, KS-authorsa; New Zealand) Depression Workshops/FGs - YP (13-18), families, clinicians, designers, cultural advisors Gamification, avatars CBT
CLIMATE Schools (Teesson et al., 2014; Australia) Depression, anxiety, substance misuse FGs - YP (13-15), practitioners, designers Interactive modules, illustrated storylines Psychoeducation, CBT, harm minimisation
CURB (Saulsberry et al., 2013; USA) Depression Surveys/workshops - YP (15-18), parents, practitioner Interactive modules CBT, IPT
Grasp the opportunity (Sobowale et al., 2013; Hong Kong) Depression Questionnaires/FGs/discussions - YP, parents, teachers, practitioners Interactive modules CBT
Rainbow SPARX (Lucassen et al., 2013; SM, KS-authorsa; Australia) Depression Questionnaires/FGs - YP (16-27) Gamification, avatars CBT
Digital programme (Authors, country) MH difficulties Stakeholder involvement (ages of CYP in years, if stated) Technological approaches Psychological & other theories
MATE (Monshat et al., 2012; Australia) General MH Interviews - YP (16-26) Interactive modules, videos, forum Mindfulness
MEMO (Whittaker et al., 2012; SM, KS-authorsa; New Zealand) Depression FGs - YP (13-17), practitioners, mHealth experts Text messages, videos/ animations CBT
Stressbusters (Robinson et al., 2011; P. Abelesa, 26.2.20; England) Depression Surveys/FGs - YP, practitioners, designers Interactive sessions, videos CBT
Think-Feel-Do (Stallard et al., 2011; PS-authora; England) Depression, anxiety FGs - CYP (11-16), designers Interactive sessions, videos CBT
CATCH-IT (Landback et al., 2009; USA) Depression Groups/questionnaires - YP, practitioners Interactive modules CBT, IPT
Reach Out! (Oliver et al., 2006; Australia) General MH Forums/mixed-methods - YP (16-25), practitioners Forum, gaming, podcasts, blogs CBT, social support
Technologies identified via personal communication (articles describing CYP involvement not available) BRAVE-ONLINE (S.Marcha, 21.2.20; Australia) Anxiety Surveys/FGs - CYP (7-18), practitioners Interactive sessions, animations, games CBT
Mightier (JK-authora; USA) Emotional regulation Observations/FGs/Interviews - CYP (6-14), parents / carers Games, biofeedback Mindfulness, constructivism
MoodGYM (H.Christensena, 20.2.20; Australia) Depression, anxiety Interviews/FGs - YP Interactive modules, workbook CBT
Pesky/Mindful gNATs (G.O’Reillya, 24.2.20; Ireland) Anxiety, depression FGs - CYP (9-17), practitioners, designers Gamification CBT, mindfulness
Smooth Sailing (M.Subotic-Kerrya, 30.9.19; Australia) Depression, anxiety Surveys/interviews, FGs - YP (13-16), counsellors, GPs, parents Stepped-care, interactive modules Psychoeducation, CBT, counsellor referral
The Journey (KS, SM-authorsa; New Zealand) Depression Workshops - YP, designers Gamification, videos/ animations CBT
Open in a new tab

DBT, Dialectical behaviour therapy; IPT, Interpersonal psychotherapy; MH, Mental health.

a

Personal communication.

Sixteen technologies were developed in Australasia (New Zealand, Australia; 53%), seven in North America (USA, Canada; 23%), five in Europe (England, Ireland, Wales; 17%) and two in Asia (Hong Kong; 7%). Nineteen articles (76%) focused primarily on the development phase of the technology (and the involvement of CYP in this), whilst six (24%) focused mainly on an evaluation or a trial of the technology (with briefer accounts of CYP involvement). Most articles that focused on the design/development were published in recent years (eleven since 2017).

The following key recurring themes related to the co-design process were identified: (a) the principles of co-design, including the participants/stakeholders and stages of involvement; (b) the potential methods and techniques of involving and engaging CYP; (c) co-designing the initial prototype, considering the diversity in the user group; and (d) the potential challenges of co-design with CYP, including its evaluation.

Principles of co-design

Creative collaboration

Co-design originated in the field of participatory design, which emphasises the importance of involving all potential users and stakeholders as active collaborators in the development of a product. This aims to ensure that technologies meet the users’ range of needs and preferences and are acceptable and helpful. Studies described how the process can involve all aspects of the technology, including content, design, accessibility, usability, data management/security, integration and implementation into users’ lives and everyday context (e.g. Hodson et al., 2019; Robinson et al., 2017; Thabrew et al., 2018; Werner-Seidler et al., 2017; Wiljer et al., 2017). A central need for co-design in the context of complex interventions is to understand the interaction of the technology within the complex psychosocial system within which it attempts to enact change (Craig et al., 2008; Hawkins et al., 2017).

As well as CYP as the primary users of the technologies, studies described how co-design might involve the following, especially if they are potential users: (a) families, carers and friends, (b) service practitioners/ experts (e.g. in education, health, social, youth services) to explore issues such as facilitators and barriers to use, (c) contentpractitioners/experts (e.g. clinicians, researchers) to help determine the evidence-based content of the technology, (d) practitioners with expertise in digital technologies, including designers, information technology (IT) developers, animators, scriptwriters and model-makers.

The process goes beyond involving CYP as a consultation or engagement exercise, but embraces a ‘democratic partnership’ with appropriate distribution of power, jointly exploring needs and creating possible solutions with CYP as ‘experts of their experiences’ (Thabrew et al., 2018). To help achieve this, Hodson et al. (2019) describe four important elements: (a) engagement with users before the project starts; (b) acknowledging the potential power imbalance between practitioners/researchers and CYP, and involving all as both ‘providers’ and ‘recipients’; (c) establishing activities (e.g. workshops) where all participants collaborate to generate ideas, guided by a facilitator, to develop and refine the product, with users always ‘signing-off’ on design proposals; (d) practitioners/researchers creating the final product according to project requirements, with possible further collaboration with users. CYP might also gain knowledge, skills and career advice from the process (Oliver et al., 2006).

Stages of involvement

It has been advocated that co-design is a dynamic and continuous process, featuring throughout the life cycle of the technology (Craig et al., 2008; Hawkins et al., 2017). This review focuses mainly on co-design during the initial development of the technology, as most studies reported only on CYP involvement during this phase - ranging from involving a small number of users at one point, to more prolonged and in-depth collaboration. There can be certain prescribed stages of user involvement, and twelve articles described an initial ‘scoping’ or ‘discovery’ phase involving CYP and other stakeholders regarding their needs and preferences (and a review of the literature and practice), before establishing the focus of the process. In the development of new interventions, co-design can involve iterative design cycles and start with ‘superficial probes’ to engage users (involving mainly researchers and designers), followed by a more intense generation of ideas (mainly involving users), and finally a narrowing of focus on the development of a prototype (all collaborating equally; Thabrew et al., 2018). Fourteen articles also described an evaluation of the initial prototype.

As a case study of the iterative co-design process of a digital technology for depression, SPARX (Shepherd et al., 2015) followed the development of an early version of gamified CBT, The Journey (by the authors KS and SM). This started with a review of best practice, identification of key therapeutic elements and learning goals, and workshops with young people (YP) and learning-technology experts. A Flash-based program was developed, and a pilot trial showed this approach was effective, although feedback suggested that YP wanted several improvements. These findings informed the development of SPARX, which included YP, clinicians, computer games practitioners, cultural advisors (Māori, Pacific people, Asian) and researchers. After initial consultations, YP were not involved again until the development of the first SPARX prototype, and they were negative about its design. The subsequent format was designed by agroup of fourteen YP, with aprocess of lively discussion with the clinician researchers. The group was selected to be representative of YP in New Zealand in terms of age (13-17 years), ethnicity, socioeconomic group and gender, and recruited through schools, youth groups and personal contacts. The IT team and researchers then worked together weekly to implement the ideas of the YP, who were consulted throughout the process. At the end of development, YP were supportive of the approach taken and suggested further refinements.

As a second case study, Figure 1 presents the overall development of another technology for depression, MoodHwb (Bevan Jones, Thapar, Rice, et al., 2018). This broadly follows the frameworks described above, whereby the initial ideas for the project were generated mainly from interviews with YP (with depressive symptoms or ‘at-risk’ of depression, because of a family history), parents/carers and professionals from health, education, social and youth services (discovery phase), and a systematic review (Bevan Jones, Thapar, Stone, et al., 2018). This informed the subsequent codesign phase involving focus groups (FGs) and workshops with these groups, a digital team and experts in psychology/psychiatry, services and design. During the early evaluation, YP and parents/carers used the prototype. Web-usage, questionnaire and interview data were analysed to determine its initial feasibility and acceptability, and to inform its refinement (Bevan Jones et al., 2020).

Figure 1. General framework for the development of the digital technology MoodHwb.

Figure 1

Open in a new tab

Methods of involvement and engagement

Collaborative activities

Children and YP with mental health difficulties might be reluctant to participate in research, because of the associated anxiety, embarrassment, stigma, motivational and other difficulties associated with mental health problems (Han et al., 2019; Hodson et al., 2019; Lucassen et al., 2013; Radovic et al., 2018; WHO, 1993). Techniques used in the studies to engage CYP included well-designed information sheets, consent forms, posters and websites outlining the aims and benefits of the study, and vouchers/expenses offered as a thank you (e.g. Bevan Jones, Thapar, Rice, et al., 2018). As with the SPARX study, other researchers attempted to recruit a diverse and representative range of participants, so as to capture the diversity in the user preferences. Eleven articles described recruiting from educational services (44%), 10 via community organisations and volunteers (40%), and five from health services (20%), with many recruiting from several sources.

The activities to involve stakeholders included the following: focus groups (used for twenty-one technologies), workshops (ten technologies), interviews (nine technologies) and surveys/questionnaires (eleven technologies). Twenty-three technologies (77%) were developed using a triangulation process (i.e. combination of several methods). In addition, specific concepts described include ‘design charrettes’, ‘design jams’ (Hodson et al., 2019), ‘design studios’ (Huggett et al., 2017), ‘crowdsourcing’ and ‘hacka-thons’ (Wiljer et al., 2017), all of which are face-to-face or virtual sessions/spaces to share and develop ideas, and which involve large (e.g. ‘charettes’) or small (e.g. ‘jams’) groups (Tables 1 and 2).

Table 2. Examples of methods of involvement.
Description Potential benefits Potential challenges Approaches to engage participants
Questionnaires/surveys (Paper, digital) Large amount of data, range of participants, increased reach, accessibility, economical, less intrusive Difficult to explore issues in-depth, poor engagement Engaging documents; digital: progress-bars, multiplatform approach, videos/animations
Interviews (Face-to-face, telephone/digital) Explore in-depth & new issues, participants can define agenda & choose setting, interaction with prototype, high credibility & face validity Intrusive, timeconsuming, reluctance to give critical feedback ‘Think-Alouds’ (participant observed/interviewed whilst using technology)
Focus groups (Face-to-face, digital/‘virtual’ Explore breadth of issues & new ideas, involve diverse group of CYP & other stakeholders, interaction with prototype, more economic/ efficient than interviews Difficult to talk to ‘strangers’ in new setting, social biases (e.g. conformity), travel to face-to-face groups Ground rules, screens & devices, materials
Interactive workshops/meetings (Face-to-face, digital/‘virtual’ As with FGs; less formal, range of interactive activities As with FGs; difficulties with recording, transcription & analysis As with FGs; ‘Wall storms’ (sticky notes on walls, processed as a group), ‘Word clouds’ (words used commonly/prominently grouped together)
‘Design studios’ (intensive development sessions) (Huggett et al., 2017)
‘Design charrettes’ (larger meetings e.g. to sketch/storyboard ideas)
‘Design jams’ (smaller sessions e.g. to develop multiple iterations of user experiences) (Hodson et al., 2019)
‘Crowdsourcing’ (open call to large group, often online e.g. to contribute project content)
‘Hackathon’ (digital event with large group e.g. proposing ideas for technologies) (Wiljer et al., 2017)
Observations/ethnographic approaches Understanding context & implementation, identify unexpected issues, detailed/‘faithful’ representation of behaviours & preferences Time-consuming, CYP may not act ‘naturally’ Appreciation & respect for environment
Open in a new tab

Three articles described using an ‘agile design’ process, which is a dynamic and flexible approach to co-design. This can involve a series of ‘sprint’ cycles that aim to discover, design, develop and test the product, through ‘scrums’ (where one individual leads and another facilitates a team, with predetermined time-frames) or ‘kanbans’ (where team members have specific tasks without fixed-length ‘sprints’; Thabrew et al., 2018).

The SPARX team used the agile design process in the development of the Quest-Te Whitianga CBT app for anxiety and depression. Its development was informed by that of SPARX, starting with the overall learning goals, updated best practice, and a wide scoping consultation exercise (Fleming et al., 2019). A target user group was defined (younger adolescents with a focus on Māori and Pacific YP), and a rapid iteration process was then used, based on ‘sprints’ and ‘scrums’. Three groups were involved: YP, software developers and the research team (including Māori and Pacific researchers). A two-weekly cycle over ten weeks led to the development of five modules, with input from all groups throughout. The app met with initial approval from YP, Māori and Pacific people and clinicians (Christie et al., 2019).

Techniques to engage users in activities

In the planning of co-design, studies reported tailoring activities according to the user group, and considering factors such as their age, abilities and health difficulties (e.g. regarding duration, access, media, materials and protocols). For prolonged iterative design, articles described running sessions in spaces that were convenient and appropriate for participants, for example with Quest-Te Whitianga, the workshops were held in the school over lunchtime (Christie et al., 2019). Han et al. (2019) reported difficulties in engaging YP to help guide the development of a suicide prevention app, but successfully organised a Web-based conferencing system, where users did not have to turn on their videos, protecting anonymity. Robinson et al. (2017) found that a closed social media page was a ‘useful and safe’ way to communicate with YP regarding another suicide prevention programme.

The practitioners/researchers participating in the activities, and the expertise and skills required, varied according to the user group, activity, technology and research phase (Table 1). Researchers identified the importance of being comfortable in engaging with CYP and the need to ‘buy-in’ to the collaborative approach (particularly the facilitators), which might be different to controlled research environments. Flexibility, patience and creativity were also needed to guide sessions where users could discuss issues openly (with respect towards others) and allow opportunities for new and interesting ideas to be explored, whilst ensuring that activities were productive and covered relevant user needs and preferences within the time allowed (e.g. Han et al., 2019; Radovic et al., 2018; Thabrew et al., 2018).

Safety and well-being considerations described in the articles include offering refreshments, creating enough space, regular breaks and giving clear ‘ground rules’. To open sessions, there were specific ‘ice breakers’, such as familiar games, and in certain cases the provision of psychoeducation or education on skills relevant to the technology development (Robinson et al., 2017). Interactive exercises included drawing, writing, storytelling, playing, storyboarding, creating videos/animations or virtual/ physical products, and ‘wall storms’ (sticky notes on walls, processed as a group; Fleming et al., 2019). Creative and communication tools included mood-boards, maps, screens and mobile devices to interact with existing technologies and prototypes. Articles outlined safety plans so that help was available in case CYP became distressed or reported difficulties (e.g. suicidal ideation) during the process (e.g. Han et al., 2019). Hodson et al. (2019) ensured that practitioners/facilitators were trained in mental health first aid.

Parents/carers were included in the development of eleven of the technologies (37%). In the scoping phase of Quest-Te Whitianga, evening sessions were held with parents and YP, so that relationships were built with the whole family. This is particularly important in some cultures, for example for Māori and Pacific YP, and allows for consideration about the context for the final delivery of the intervention. There was also a formal opening and closing of sessions by an elder involving a prayer, speech and introductions (Fleming et al., 2019).

Co-designing the prototype

Mapping the prototype

The creation and testing of the initial prototype of the technology was described as an integral part of the co-design process that helped users to engage more with the process. To inform the prototype development, data collection and analysis in the initial design phase were mainly qualitative in nature, based on data from questionnaires, transcripts, notes, images and other materials. When testing the prototypes, Web-usage and computer-screen tracker data were also analysed. ‘Think-Alouds’, where CYP were observed and interviewed whilst using the technologies, were reported as helpful as CYP were ‘immersed’ and ‘less self-conscious’ (Christie et al., 2019; Radovic et al., 2018).

Studies described how the specifications for the content and design of the technology were refined according to the level of importance given to them by the participants and the potential effect on the acceptability, feasibility and ease of use. Other considerations included the programme aims, theory/evidence, technical difficulty, time and resources required, and development costs (Christie et al., 2019; Lucassen et al., 2013; Monshat et al., 2012; Radovic et al., 2018). Whilst twenty-three of the technologies identified (77%) were new or ‘de novo’, the others were adaptations via secondary codesign processes (three from existing digital interventions and four from face-to-face/manualised approaches).

Figure 2 shows the development process of the MoodHwb prototype, which was created ‘de novo’. Initial note boards and sketches were created based on user and project requirements, and initial designs were developed in the FGs. Wireframes (skeletal frameworks) were then constructed showing the layout and functionality of each proposed screen, which evolved into the prototype. Discussions with the YP, designers and animator also focused on the illustrations, characters, scripts and animations (Bevan Jones, Thapar, Rice, et al., 2018).

Figure 2. Development of welcome screen and user-flow of MoodHwb: notes/sketches (above), wireframes (centre), early designs (below) (adapted from Bevan Jones, Thapar, Rice, et al., 2018).

Figure 2

Open in a new tab

Key frameworks recommend mapping the underlying intervention theory, which can be done in collaboration with CYP, for example through ‘logic models’. This ‘blueprint’ can be referenced and refined throughout the research cycle (Rehfuess et al., 2018). The model can show the possible user activities (e.g. completing exercises, personalising content, gaming), mechanisms of change (e.g. improved understanding, learning self-management skills), potential outcomes (e.g. improvement in symptoms/well-being) and context, including barriers and facilitators to use (e.g. availability of devices, ease of use) when considering implementation (Bevan Jones, Thapar, Rice, et al., 2018).

Potential negative effects were a particular concern in studies. Eighteen articles (72%) highlighted security and confidentiality as important considerations, in part so that CYP engaged with and trusted the technology. Elements to ensure compliance with data protection regulations included the use of usernames and passwords to log-in, locks, moderation, data encryption and secure servers. Choosing a benign brand or name, possibly unrelated to mental health difficulties, can help with privacy (Werner-Seidler et al., 2017).

Engaging with the diversity in the group

A range of digital elements were used in technologies to present content and to ensure they had options, increased agency and flexibility (Table 1). Twelve of the technologies used gamification (40%), twelve used interactive exercises/modules (40%), eight included videos/animations (27%), three incorporated social media/messaging (10%) and two used chatbots (7%). The majority were based at least in part on CBT (twenty-two technologies, 73%), whilst seventeen (57%) were based on several psychological or other theories.

Certain studies discussed creating technologies that could be personalised or could address diversity in the user group, for example regarding age/development, gender, cultural context and severity of difficulties (e.g. Lucassen et al., 2013; Werner-Seidler et al., 2017). In a scoping study, Fleming et al., (2019) concluded that younger adolescents who experienced stress or low mood were more likely to be interested in interactive and gamified digital interventions, whilst older adolescents with difficulties were more interested in clearly designed and ‘straight to the point’ products.

Technologies were developed or adapted to engage with specific cultures and subgroups (Saulsberry et al., 2013; Sobowale et al., 2013). Co-design considerations in this context included the language/ text, iconography/symbols, metaphors, colours, characters and, in some cases, the general principles or philosophy of the technology. For example, as noted earlier, the Māori models of mental health are particularly holistic, and this approach influenced the development of SPARX (Shepherd etal., 2015). SPARX has also been adapted for use in Japan, Nunavut and the Netherlands, and to help sexual minority youth (Lucassen et al., 2013).

Challenges of co-design

During the planning stage, studies described how the flexibility required in the process can lead to difficulties when navigating funding panels and ethics committees that might expect clear plans for the technology. There can be concern about the pace, cost and scale of the process, from users or services. It might be difficult to find the required funds and resources required for co-design activities (e.g. expenses, venues, materials, recordings, transcriptions) and technology development, from research, clinical or other funders. Therefore, authors recommend being clear from the outset about the justification and timescale for this rigorous approach (Hodson et al., 2019; Thabrew et al., 2018).

Some of the challenges related to the methods (e.g. recruitment and engagement) are noted earlier and in Table 2. Another potential risk is that the size or nature of the sample of CYP mean that they are not representative of the target population. In addition, it is likely there is self-selection, in that participants who volunteer are more likely to have an interest in mental health research, although they may still be representative of proposed end-users (Han et al., 2019; Monshat et al., 2012; Radovic et al., 2018; Werner-Seidler et al., 2017). It can be difficult to please all participants when developing the technology, and an attempt to do this can lead to a hybrid that is unacceptable to everyone. To help with these challenges, studies reported efforts to engage a diverse sample where appropriate, and to be clear regarding the need to balance feedback with other considerations (Bevan Jones, Thapar, Rice, et al., 2018; Thabrew et al., 2018). There might also be a difference between what CYP say they want, and what they actually use. A comparison between the input during development and the testing or acceptability stage was described as informative in this regard (Werner-Seidler et al., 2017).

Finally, key guidelines for intervention development recommend assessing the acceptability, feasibility and validity of co-design with CYP (Wight et al., 2016). However, only five articles (19%) described a process evaluation, and none described the impact of the process on the technology. The mixed-methods acceptability evaluations found that participants affirmed the value of collaboration, described the experience as ‘enjoyable’ and ‘rewarding’, stated they had gained knowledge and skills, and felt more able to talk about mental health issues and support others (e.g. Robinson et al., 2017). Negative comments included how activities were ‘exhausting’ (Hodson et al., 2019) and ‘dry’ (Huggett et al., 2017). Wiljer et al. (2017) noted how it had been a challenge to keep YP engaged as the project progressed and they had less ‘ownership’ if they did not have clearly defined roles.

Practice points and future developments

Based on the findings of the review, in Figure 3 we present a checklist of questions that could be considered when planning, documenting or analysing co-design activities. We recommend that practitioners and researchers consider the specific target user group, the technology to be developed and its context (if known), and then align the requirements of the co-design activities accordingly. These issues could be clarified through initial scoping activities.

Figure 3. Checklist of questions to consider when planning, reporting or analysing co-design activities.

Figure 3

Open in a new tab

Furthermore, researchers could consider the needs and preferences of the user group and the heterogeneity within this group, as well as the methods and resources required to recruit and engage them in a collaborative manner on all aspects of the technology, potentially at any point in the research cycle. The practitioners and researchers involved in the process need the necessary skills and expertise (e.g. related to the content, service, design or digital work), to co-develop an engaging, acceptable and helpful technology with and for CYP. A mixed-methods evaluation of the acceptability and feasibility of the process, as well as the potential impact of the process on the technology could be considered.

Given the fast pace of digital technology and culture, there are a number of possible future developments in co-design practice and research. Periodsofreviewbuilt into the research cycle are recommended to future-proof the design and content (Craig et al., 2008). Methods of co-design might adapt as technologies become more complex, flexible and personalised. There might be more ‘virtual’ groups and workshops, which can help with reach and access a more diverse group of users (Han et al., 2019). Evaluations of the impact of co-design might involve trials comparing codesigned technologies with those that did not involve CYP or used alternative models.

There are concerns about the timeframe of the pipeline from development to implementation, particularly with the need for faster translation of findings into the community. There is also the challenge of validating a moving target that becomes irrelevant if pausing for long. More flexible and quicker models in the ‘real world’ will be needed, for example using digital ecosystems with built-in architecture to support rapid re-testing of different versions using a range of measures (Rice et al., 2018; Thabrew et al., 2018), but maintaining a rigorous approach to development and evaluation, with no harm and minimal costs. All these developments might involve collaboration between academic, clinical and commercial sectors, and there might be learnings from gaming and commercial apps (Christie et al., 2019).

Discussion

We have conducted a review of the use of co-design in the development of digital mental health technologies with CYP and have supplemented this with case studies and practice points based on the findings and from several researchers with experience of co-design practice with this age group. There is a range of approaches to involve CYP and other stakeholders in the co-development of technologies throughout the research cycle. These methods need to be tailored according to the users (considering the diversity within the group), digital technology and setting. There are also potential challenges, in particular related to finding the resources required, balancing the input of all stakeholders and evaluating the impact on technologies. The review helps to inform practitioners and researchers interested in developing technologies for CYP.

The increase in articles published over recent years on the design and development of technologies with CYP, suggests there is increased interest in this field - especially since the publication of a systematic review by Orlowski et al (2015) on YP involvement in the design of technology-based interventions (although this focused on ‘youth’ and had a broader approach to participation, mental health/well-being and interventions). The earlier review concluded that YP involvement was mainly consultative in nature, whilst our review suggests that such activities might have become more collaborative. The lack of documented evaluations in our review is consistent with Orlowski et al’s findings that there was limited outcome data and evidence on the impact of participatory research on intervention effectiveness.

The strengths of our review include the systematic approach to the search and the collation of information by practitioners and researchers in this field from around the world. This is the first review, to our knowledge, that brings together co-design practices of digital mental health technologies for children as well as YP, although it builds on the previous review.

The review has limitations. We acknowledge that other technologies (e.g. for depression or anxiety) might have involved CYP, but were not captured by this search (e.g. because this was not documented clearly in peer-reviewed papers). Whilst the review focused on resources/interventions for specific mental health difficulties, co-design also plays an important role in technologies used in other areas of mental health, as well as for physical health, and in assessment, communication and data management. There are also other models of involving stakeholders, as well as participatory design and co-design (Orlowski et al., 2015), although we took an inclusive approach to screening articles regarding CYP involvement. We did not appraise the quality or effectiveness of activities, given the heterogeneity of practices and lack of guidance in this field. However, this could be a focus of future papers as the field develops. The practitioner points section includes some of the authors’ views, and there are likely to be other perspectives not represented here.

Whilst there are emerging studies involving YP and certain subgroups in co-design, younger children and those with learning disabilities and specific difficulties are under-represented and will have specific needs and preferences. More research is also required into the implementation phase of technologies, and how co-design can play a part (Craig et al., 2008). Most of the studies identified were based in ‘developed’ countries, and there is increasing interest in the use of digital technologies in LMIC (Naslund et al., 2017). If the field is to progress and have genuine lasting impact, further research and guidance are required on processes to involve CYP and their evaluation. The co-design practices might then become the new benchmark for how digital technologies of high quality are developed.

Supplementary Material

Figure S1

Key points.

  • The aim of co-design is to ensure that digital technologies meet the users’ needs and preference, so that the technologies are potentially more engaging, feasible, acceptable and effective.

  • Co-design can feature throughout the life cycle of the technology and in all stages of the research including development, evaluation and implementation.

  • There is a range of methods to involve and engage children, young people and other stakeholders in codesign.

  • Future work in this field will need to consider the changing face of technology, the diversity of the user group, the need for faster translation of findings into the community, and the evaluation of the co-design process and its impact.

Acknowledgements

R.B.J. is supported by the Welsh Government through Health and Care Research Wales (National Institute for Health Research Fellowship, NIHR-PDF-2018), and the authors thank them for their support. S.R. is supported by a Career Development Fellowship (APP115888) and M.A.J. is supported by an Investigator Grant (APP1177235) from the National Health and Medical Research Council, Australia. A.W-S. is supported by a NSW Health Fellowship. S.S. is supported by the Medical Research Council and the Chief Scientist Office of the Scottish Government Health and Social Care Directorates (MC_UU_12017_14, SPHSU14). The authors thank all CYP, parents/carers, practitioners, designers and researchers who have collaborated with them in their studies. The intellectual property for SPARX is held by UniServices. S.M. and K.S. stand to gain financially from any commercialisation of SPARX. J.K. is a co-founder of Neuromotion Labs, which developed/ commercialised Mightier. The remaining authors have declared that they have no competing or potential conflicts of interest.

Footnotes

Conflict of interest statement: See Acknowledgements for full disclosures.

References

  1. Bevan Jones R, Thapar A, Rice F, Beeching H, Cichosz R, Mars B, Simpson SA. A Web-based psychoeducational intervention for adolescent depression: Design and development of MoodHwb. JMIR Mental Health. 2018;5:e13. doi: 10.2196/mental.8894. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bevan Jones R, Thapar A, Rice F, Mars B, Agha SS, Smith DJ, Simpson SA. A digital intervention for adolescent depression (MoodHwb): Mixed-methods feasibility evaluation. JMIR Mental Health. 2020 doi: 10.2196/14536. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bevan Jones R, Thapar A, Stone Z, Thapar AK, Jones I, Smith D, Simpson S. Psychoeducational interventions in adolescent depression: A systematic review. Patient Education and Counseling. 2018;101:804–816. doi: 10.1016/j.pec.2017.10.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Christie GI, Shepherd M, Merry SN, Hopkins S, Knightly S, Stasiak K. Gamifying CBT to deliver emotional health treatment to young people on smartphones. Internet Interventions. 2019;18:100286. doi: 10.1016/j.invent.2019.100286. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655. doi: 10.1136/bmj.a1655. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Fleming T, Merry S, Stasiak K, Hopkins S, Patolo T, Ruru S, Goodyear-Smith F. The importance of user-segmentation for designing digital therapy for adolescent mental health: Findings from scoping processes. JMIR Mental Health. 2019;6:e12656. doi: 10.2196/12656. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Han J, Torok M, Gale N, Wong QJ, Werner-Seidler A, Hetrick SE, Christensen H. Use of web conferencing technology for conducting online focus groups among young people with lived experience of suicidal thoughts: mixed methods research. JMIR Mental Health. 2019;6:e14191. doi: 10.2196/14191. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Hawkins J, Madden K, Fletcher A, Midgley L, Grant A, Cox G, White J. Development of a framework for the co-production and prototyping of public health interventions. BMC Public Health. 2017;17:689. doi: 10.1186/s12889-017-4695-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Hodson E, Dadashi N, Delgado R, Chisholm C, Sgrignoli R, Swaine R. Co-design in mental health; Mellow: A self-help holistic crisis planning mobile application by youth, for youth. The Design Journal. 2019;22:1529–1542. [Google Scholar]
  10. Hollis C, Falconer CJ, Martin JL, Whittington C, Stockton S, Glazebrook C, Davies EB. Annual research review: Digital health interventions for children and young people with mental health problems-a systematic and meta-review. Journal of Child Psychology and Psychiatry. 2017;58:474–503. doi: 10.1111/jcpp.12663. [DOI] [PubMed] [Google Scholar]
  11. Huggett D, Flynn A, Jaouich A, Taylor-Gates M, Davidson S. Engaging youth in a mental health system improvement initiative in Ontario: Developing the BeSafe resource. Canadian Journal of Community Mental Health. 2017;36:121–131. [Google Scholar]
  12. Landback J, Prochaska M, Ellis J, Dmochowska K, Kuwabara SA, Gladstone T, Van Voorhees BW. From prototype to product: development of a primary care/internet-based depression prevention intervention for adolescents (CATCH-IT) Community Mental Health Journal. 2009;45:349–354. doi: 10.1007/s10597-009-9226-3. [DOI] [PubMed] [Google Scholar]
  13. Lucassen MFG, Hatcher S, Stasiak K, Fleming T, Shepherd M, Merry SN. The views of lesbian, gay and bisexual youth regarding computerised self-help for depression: An exploratory study. Advances in Mental Health. 2013;12:22–33. [Google Scholar]
  14. Mohr DC, Schueller SM, Montague E, Burns MN, Rashidi P. The behavioral intervention technology model: An integrated conceptual and technological framework for ehealth and mhealth interventions. Journal of Medical Internet Research. 2014;16:e146. doi: 10.2196/jmir.3077. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Monshat K, Vella-Brodrick D, Burns J, Herrman H. Mental health promotion in the Internet age: A consultation with Australian young people to inform the design of an online mindfulness training programme. Health Promotion International. 2012;27:177–186. doi: 10.1093/heapro/dar017. [DOI] [PubMed] [Google Scholar]
  16. Naslund JA, Aschbrenner KA, Araya R, Marsch LA, Unützer J, Patel V, Bartels SJ. Digital technology for treating and preventing mental disorders in low-income and middle-income countries: A narrative review of the literature. The Lancet Psychiatry. 2017;4:486–500. doi: 10.1016/S2215-0366(17)30096-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. National Institute for Health and Care Excellence. Depression in children and young people: Identification and management (NG134) 2019 Retrieved from www.nice.org.uk/guidance/ng134. [PubMed] [Google Scholar]
  18. Neufeld SAS, Dunn VJ, Jones PB, Croudace TJ, Goodyer IM. Reduction in adolescent depression after contact with mental health services: A longitudinal cohort study in the UK. The Lancet Psychiatry. 2017;4:120–127. doi: 10.1016/S2215-0366(17)30002-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Oliver KG, Collin P, Burns J, Nicholas J. Building resilience in young people through meaningful participation. Australian e-Journal for the Advancement of Mental Health. 2006;5:34–40. [Google Scholar]
  20. Orlowski SK, Lawn S, Venning A, Winsall M, Jones GM, Wyld K, Bidargaddi N. Participatory research as one piece of the puzzle: A systematic review of consumer involvement in design of technology-based youth mental health and well-being interventions. JMIR Human Factors. 2015;2:e12. doi: 10.2196/humanfactors.4361. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Radovic A, DeMand AL, Gmelin T, Stein BD, Miller E. SOVA: Design of a stakeholder informed social media website for depressed adolescents and their parents. Journal of Technology in Human Services. 2018;35:169–182. doi: 10.1080/15228835.2017.1347552. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Rehfuess EA, Booth A, Brereton L, Burns J, Gerhardus A, Mozygemba K, Rohwer A. Towards a taxonomy of logic models in systematic reviews and health technology assessments: A priori, staged, and iterative approaches. Research Synthesis Methods. 2018;9:13–24. doi: 10.1002/jrsm.1254. [DOI] [PubMed] [Google Scholar]
  23. Rice S, Gleeson J, Davey C, Hetrick S, Parker A, Lederman R, Alvarez-Jimenez M. Moderated online social therapy for depression relapse prevention in young people: Pilot study of a ‘next generation’ online intervention. Early Intervention in Psychiatry. 2018;12:613–625. doi: 10.1111/eip.12354. [DOI] [PubMed] [Google Scholar]
  24. Robinson A, Yule W, Verduyn C, Smith P, Proudfoot J, Abeles P. Stressbusters: The development of a computerised CBT programme for adolescent depression. ACAMH Occasional Papers. 2011;30:45–50. [Google Scholar]
  25. Robinson J, Bailey E, Hetrick S, Paix S, O’Donnell M, Cox G, Ftanou M, Skehan J. Developing social media-based suicide prevention messages in partnership with young people: Exploratory study. JMIR Mental Health. 2017;4:e40. doi: 10.2196/mental.7847. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Saulsberry A, Corden ME, Taylor-Crawford K, Crawford TJ, Johnson M, Froemel J, Van Voorhees BW. Chicago Urban Resiliency Building (CURB): An internet-based depression-prevention intervention for urban African-American and Latino adolescents. Journal of Child and Family Studies. 2013;22:150–160. [Google Scholar]
  27. Shepherd M, Fleming T, Lucassen M, Stasiak K, Lambie I, Merry SN. The design and relevance of a computerized gamified depression therapy program for indigenous Māori adolescents. JMIR Serious Games. 2015;3:e1. doi: 10.2196/games.3804. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Shum AK, Lai ES, Leung WG, Cheng MN, Wong HK, So SW, Yip PS. A digital game and schoolbased intervention for students in Hong Kong: Quasi-experimental design. Journal of Medical Internet Research. 2019;21:e12003. doi: 10.2196/12003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Sobowale K, Zhou AN, Van Voorhees BW, Stewart S, Tsang A, Ip P, Chim D. Adaptation of an internet-based depression prevention intervention for Chinese adolescents: From “CATCH-IT” to “Grasp the opportunity”. International Journal of Adolescent Medicine and Health. 2013;25:127–137. doi: 10.1515/ijamh-2013-0020. [DOI] [PubMed] [Google Scholar]
  30. Stallard P, Porter J, Grist R. A Smartphone App (Bluelce) for young people who self-harm: Open phase 1 prepost trial. JMIR mHealth and uHealth. 2018;6:e32. doi: 10.2196/mhealth.8917. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Stallard P, Richardson T, Velleman S, Attwood M. Computerized CBT (Think, Feel, Do) for depression and anxiety in children and adolescents: Outcomes and feedback from a pilot randomized controlled trial. Behavioural and Cognitive Psychotherapy. 2011;39:273–284. doi: 10.1017/S135246581000086X. [DOI] [PubMed] [Google Scholar]
  32. Teesson M, Newton NC, Slade T, Chapman C, Allsop S, Hides L, Andrews G. The CLIMATE schools combined study: A cluster randomised controlled trial of a universal Internet-based prevention program for youth substance misuse, depression and anxiety. BMC Psychiatry. 2014;14:32. doi: 10.1186/1471-244X-14-32. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Thabrew H, Fleming T, Hetrick S, Merry S. Codesign of eHealth interventions with children and young people. Frontiers in Psychiatry. 2018;9:481. doi: 10.3389/fpsyt.2018.00481. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Werner-Seidler A, O’Dea B, Shand F, Johnston L, Frayne A, Fogarty AS, Christensen H. A smartphone app for adolescents with sleep disturbance: development of the Sleep Ninja. JMIR Mental Health. 2017;4:e28. doi: 10.2196/mental.7614. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Whittaker R, Merry S, Stasiak K, McDowell H, Doherty I, Shepherd M, Rodgers A. MEMO-a mobile phone depression prevention intervention for adolescents: Development process and postprogram findings on acceptability from a randomized controlled trial. Journal of Medical Internet Research. 2012;14:e13. doi: 10.2196/jmir.1857. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Wight D, Wimbush E, Jepson R, Doi L. Six steps in quality intervention development (6SQuID) Journal of Epidemiology and Community Health. 2016;70:520–525. doi: 10.1136/jech-2015-205952. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Wiljer D, Johnson A, McDiarmid E, Abi-Jaoude A, Ferguson G, Hollenberg E, Law M. Thought Spot: Co-creating mental health solutions with post-secondary students. Studies in Health Technology and Informatics. 2017;234:370–375. [PubMed] [Google Scholar]
  38. World Health Organization. The ICD-10 classification of mental and behavioural disorders: Diagnostic criteria for research. WHO; Geneva, Switzerland: 1993. [Google Scholar]
  39. Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: Application to digital health-related behavior change interventions. Journal of Medical Internet Research. 2015;17:e30. doi: 10.2196/jmir.4055. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Figure S1
EMS137683-supplement-Figure_S1.docx (68.1KB, docx)

RESOURCES