Abstract
Mental health problems are a major public health concern worldwide. Approximately 50% of the population will experience mental problems during their life. Traditional treatment is based on psychopharmacotherapy or psychotherapy, with face-to-face interaction between the patient and the therapist. New technologies such as Internet-delivered treatments are seen as an opportunity to offer more scalable and cost-efficient treatments in the field of mental health. Despite the growing interest and new evidence supporting the effect of Internet-delivered treatments is it remarkably little research on how the technology and the usability of Internet-delivered treatment programs affects the treatment. In this paper, we propose a set of evaluation criteria for evaluating the usability and the responsive design of Internet-delivered treatment systems. By our knowledge we are the first to include usability and universal design principles in the evaluation of Internet-delivered treatment systems. Our findings indicate that despite the good treatment results and proven clinical effects, the systems in general have several issues regarding usability, universal design and outdated technology. Based on our findings we propose that there should be established guidelines for testing the usability and technology of Internet-delivered treatment systems.
Keywords: ICBT, Usability, Universal design, WCAG 2.0
Highlights
-
•
The usability and technology were evaluated for 6 different Scandinavian Internet delivered mental health treatment systems.
-
•
We propose a set of evaluation criteria based on usability, universal design principles, security and internet standards.
-
•
The systems could be improved with regards to usability and universal design. Newer systems comply better than older ones.
-
•
The security mechanisms of the systems were satisfactory.
1. Introduction
Mental health problems are a major public health concern worldwide. Treatment is mainly delivered as psychopharmacotherapy or psychotherapy, with face-to-face interaction between the patient and the therapist. However, new technologies such as Internet-delivered treatments are seen as an opportunity to offer more scalable and cost efficient treatments in the field of mental health (Andersson and Cuijpers, 2018; Andersson et al., 2014; Titov et al., 2018a; Ebert et al., 2018). Internet-delivered psychological treatments often make use of the same therapeutic principles as evidence–based face-to-face psychological treatments. The difference is that the therapeutic content is provided via the Internet on computers, tablets and/or smart phones. The structure of the Internet-delivered treatments is similar to that of regular face-to-face treatments; content and assignments are dispatched weekly with the aim that patients will practice and apply the techniques in their daily lives. Most Internet-delivered treatments provide text-based information and are highly structured with systematically presented psycho-education, assignments (homework), symptom assessment through questionnaires and other resources (e.g. activity monitoring, electronic (thought) diaries) and are often supplemented with audio, video and animations. The treatments can be provided with some level of guidance of a clinician or coach via (asynchronous) email or a secure messaging system, or (synchronous) online chat, video-consultations or telephone calls. Internet-delivered treatments can also be entirely self-guided with no or automated feedback on (homework) assignments, symptom level etc. A considerable amount of research has shown that Internet-delivered CBT (ICBT) can be effective means in the treatment of common mental disorders (Andersson et al., 2014; Titov et al., 2018a; Ebert et al., 2018). When compared with traditional face-to-face treatment, clinician guided Internet interventions can even be as effective when treating depression and anxiety (Andersson et al., 2014). One important finding that emerges from earlier meta-analytic studies of Internet-delivered treatments is the positive correlation between time spent on the program and clinical effects (Farrer et al., 2014; Baumeister et al., 2014; Karyotaki et al., 2018). Reflecting the weight of this encouraging evidence regarding Internet-delivered treatments, several health care providers internationally now offer guided Internet-delivered treatment as part of routine care in i.e. Europe, North-America and Australia (Titov et al., 2018b). In Scandinavia, Sweden, Denmark, and Norway all have clinics which provide guided Internet-delivered treatment for depression as part of routine care (Folker et al., 2018). However, nowadays Internet-delivered treatment systems are often developed with limited resources, which often results in issues concerning usability, personalized treatment, interactive treatment, and adaptable treatment, both from the patient and the health service point of view. In order to increase the impact of Internet-delivered treatments in routine care, one should investigate how the technology and the usability of the system correspond with the treatment outcome. It should be systematically studied what treatment effect the following factors have: is the system in line with the user requirements, is the system adaptable to what patients want and to what patients need (e.g. based on the patients educational level, motivation and functional level). By meeting the needs of the patients, the motivation for and the adherence to the treatment might be changed (increased), this is important since adherence is linked to positive outcome in ICBT (Titov et al., 2018a). Simpson et al. (2012) published a study indicating that adherence was related to improved outcome for Obsessive compulsive disorder. However, we are not aware of any systematic research studying the relation between usability, engagement and adherence of Internet-delivered treatment. In this paper, we take a step in this direction by identifying a set of evaluation criteria and performing a systematic review of Internet-delivered treatment systems with emphasis on usability aspects based on the WCAG 2.0 framework.
More specifically, we study the usability of the system and how the patient is interacting with the therapist and the health care system, we also study the responsiveness of the system i.e. how well the system adapts to different platforms. For each system, we also give some brief background information about the clinical usage, including the methodology, and the inclusion criteria for the depression treatment. Five systems in Norway and Sweden for adults with depression, the most common mental health disorder, will be examined.
2. Method
2.1. Procedure
The assessment of the Scandinavian systems was conducted in the spring 2018 and was done by one ICT researcher taking the role as a patient. The researcher conducted a full intervention in one day in each Internet-delivered treatment system. The researcher could experience how good the usability and the accessibility were by taking the role as a patient on a PC, and on an android mobile phone to evaluate if the system uses responsive web design. We used two test tools to test whether the Internet-delivered treatment systems meet the success criteria from WCAG 2.0. The first tool, W3C checker1 was used to test whether the system meets the programmatic successes criteria, e.g. if there is a text alternative for a picture. The tool specifies which criteria are not met and where the source code error exists. The second tool, Color Contrast Analyser2 check whether the contrast satisfies the criteria according to WCAG 2.0. Some success criteria need to be verified manually, e.g. the reading level.
2.2. Assessment
2.2.1. Clinical usage
2.2.1.1. Target group
The inclusion criteria for the treatments are (with some local differences) being adults, diagnosed with a current major depression, stable medication and reading and writing the language of the treatment program. Exclusion criteria (with some local differences) are ongoing suicidal ideation, other severe disorders with an immediate need of treatment, no computer or Internet access.
2.2.1.2. Recruitment of patients
The ICBT treatment programs in the Scandinavia Internet clinics are through referrals from a General Practitioner (GP) and self-referrals from the general population.
2.2.1.3. Treatment model
All programs in the study are mainly based on the model of CBT.
2.2.1.4. Therapist support
Programs with and without asynchronous therapist guidance via the system is included.
2.2.2. Usability
Usability assess the quality of the user's experience when interacting with a product or system e.g. a user interface on a website, including challenges regarding universal design and user-friendliness according to the standard Web Content Accessibility Guidelines 2.0 (WCAG 2.0) (Caldwell et al., 2008). WCAG 2.0 is a standard for evaluating usability of software systems that takes into account new technologies, different user agents (any software that retrieves and presents web content for users) and universal design in sufficient degree (Caldwell et al., 2008). The standard consists of four general principles and 12 guidelines, including 61 testable requirements called success criteria on the following sale: A (lowest), AA and AAA (highest). To be qualified for a level, a system needs to fulfill all the success criteria for that level. The four main principles are as follows: 1) Perceivable: The users must be able to perceive the information being presented; 2) Operable: The users must be able to operate the interface; 3) Understandable: The users must be able to understand the information as well as the operation of the user interface; 4) Robust – The users must be able to access the content as technologies advance.
2.2.3. Interaction
To evaluate the user interaction of the Internet-delivered treatment systems we have built on Kraut et al. (Kraut et al., 2002) terminologies for affordance in communication media (interaction through the technology). The goal of evaluating the interaction with the system is to map the functionality they provide and evaluate how user-friendly they are. The four chosen terminologies to describe the interaction possibilities through the systems are: 1) Communication, communication mechanisms, functionality and Roles (Table 1).
Table 1.
The evaluation criteria for interaction.
| Criteria | Description |
|---|---|
| Communication | In which degree is there a direct interaction between the patient and therapist? Is the communication asynchronous or synchronous, automated feedback? |
| Communication mechanism | What kind of communication mechanisms are used? i) Video ii) Telephone iii) Email iv) Chat/SMS |
| Functionality | What kind of functiionality is supported? i) Chat ii) Exercises iii) Forum iv) Questionnaires v) Diary vi) Video lectures vii) Audio lecture viii) Animation |
| Roles | Which roles do the systems support? i) Patient and therapist ii) Group therapy (many patient and one therapist) |
The criteria used for evaluating responsive web design and compliance to web standards covers navigation, content, image, video, typography and touch controls (Table 2).
Table 2.
Criteria for responsive web design (Bohyun, 2013; Marcotte, 2011; Institute of Electrical and Electronics Engineers (IEEE), 2012).
| Criteria | Description |
|---|---|
| Navigation | Make sure the maximum display area is available for the main content. Check if they use a drop-down bar for small phones. Allow columns to reduce in width up to a point for tablets, but they should then collapse under each other, allowing the user on smart phones (small) and on smaller tablets to scroll through the content. |
| Content | Ensure any content that is not the primary point of the page is not surrounded by auxiliary content. The website has configured the viewport (the user's visible area) i.e. adapt the width of the page to the screen width of the device. |
| Image | The images should be resized to fit with the resolution of the screen without losing the fidelity of the images. |
| Video | Supporting newer HTML5 formats for web video like Web-M, Theora Ogg and H.264 instead of Flash player. |
| Typography | Typefaces should be carefully considered since clarity is important, especially when being read devices. |
| Touch controls | Touch control placement: options in the lower half of the screen will generally be easier to control with a single hand than in the upper portion of screen. It should be easy to click on links, buttons and other clickable items. The size of control target sizes should be minimum of 44px in width or height at 72 dpi. |
3. Results
In this study we have investigated five Internet-delivered treatment systems and reviewed how they comply with universal design principles and responsive design. 1) eMeistring provides guided Internet-delivered treatment in secondary outpatient mental health clinics in Norway. The treatment has documented effects from routine care (Nordgreen et al., 2019; Nordgreen et al., 2018; Nordgreen et al., 2017). 2) Assistert Selvhjelp is an Internet-delivered treatment system targeting mild or moderate depression and anxiety in primary care where patients are self-referred, today in use by more than 30 Norwegian municipalities. 3) MoodGym is an Internet-delivered treatment system designed for patient to prevent and manage symptoms of depression and anxiety first introduced in Australia in 2001, today with more than 1 million users worldwide. The treatment has shown to have small but significant effect in community settings reported in a randomized controlled trial (Christensen and Griffiths, 2001). 4) Psyktools (renamed to YouWell after the study) is a Norwegian Internet-delivered treatment system designed for therapist and patients. The tool helps the therapist to manage and conduct digital self-management program for patients with mental health problems. The system is still under development and not applied for treatment yet. 5) Internetpsykiatri is a system to create, administer, and evaluate therapist-supported Internet-delivered psychological treatment in the Stockholm area (Sweden). The system has continuously been updated and used in research and regular care since 2003. The effects for the interventions for adults with i.e. social anxiety disorder (Furmark et al., 2009), depression, insomnia (Kaldo et al., 2015) and Irritable Bowel syndrome (Ljótsson et al., 2010) has proven to be large in randomized controlled trials and open effectiveness trials (Hedman et al., 2014).
A detailed summary of the main findings from “MoodGym 2.0” are presented below and in Table 3, the other systems are presented in Table 4, Table 5, Table 6, Table 7, Table 8.
Table 3.
Evaluation of usability for MoodGym 2.0.
| Success criteria | Level | Result | Evaluation |
|---|---|---|---|
| 1.1.1 Non-text content | A | ✗ | Missing the text for the image. |
| 1.2.1 Audio-only and video-only (prerecorded) | A | ✓ | |
| 1.2.2. Captions (prerecorded) | A | ||
| 1.2.3 Audio description or media alternative (prerecorded) | A | ||
| 1.2.4 Captions (live) | AA | ||
| 1.2.5 Audio description (prerecorded) | AA | ||
| 1.2.6 Sign language (prerecorded) | AAA | ||
| 1.2.7 Extended audio description (prerecorded) | AAA | ||
| 1.2.8 Media alternative (prerecorded) | AAA | ||
| 1.2.9 Audio-only (live) | AAA | ||
| 1.3.1 Info and relationships | A | ✗ | The tables are not coded like they look. |
| 1.3.2 Meaningful sequence | A | ✓ | |
| 1.3.3 Sensory characteristics | A | ✗ | Using symbols only to remove rows in the tables. |
| 1.4.1 Use of color | A | ✗ | Using color only to highlight important information in a text. |
| 1.4.2 Audio control | A | ||
| 1.4.3 Contrast (minimum) | AA | ✗ | The measured contrast is 3,1:1. |
| 1.4.4 Resize text | AA | ✗ | The text is unclear when zooming to 200% in certain text box. |
| 1.4.5 Images of text | AA | ✓ | |
| 1.4.6 Contrast (enhanced) | AAA | ✗ | The measured contrast is 3,1:1. |
| 1.4.7 Low or no background audio | AAA | ✓ | |
| 1.4.8 Visual presentation | AAA | ✗ | Cannot choose background color. |
| 1.4.9 Images of text (no exception) | AAA | ✓ | |
| 2.1.1 Keyboard | A | ✗ | Cannot expand the text field by using keyboard only. |
| 2.1.2 No keyboard trap | A | ✓ | |
| 2.1.3 Keyboard (no exception) | AAA | ✗ | Cannot expand the text field by using keyboard only. |
| 2.2.1 Timing adjustable | A | ||
| 2.2.2 Pause, stop, hide | A | ✗ | Cannot pause, stop or hide the animation. |
| 2.2.3 No timing | AAA | ✓ | |
| 2.2.4 Interruptions | AAA | ||
| 2.2.5 Re-authenticating | AAA | ||
| 2.3.1 Three flashes or below threshold | A | ✓ | |
| 2.3.2 Three flashes | AAA | ✓ | |
| 2.4.1 Bypass blocks | A | ✓ | |
| 2.4.2 Page titled | A | ✓ | |
| 2.4.3 Focus order | A | ✓ | |
| 2.4.4 Link purpose (in context) | A | ✗ | The system has “link” designed like links which it is not. |
| 2.4.5 Multiple ways | AA | ✓ | |
| 2.4.6 Headings and labels | AA | ✓ | |
| 2.4.7 Focus visible | AA | ✓ | |
| 2.4.8 Location | AAA | ✓ | |
| 2.4.9 Link purpose (link only) | AAA | ✗ | The link does not describe the purpose correctly. |
| 2.4.10 Section headings | AAA | ✓ | |
| 3.1.1 Language of page | A | ✓ | |
| 3.1.2 Language of parts | AA | ||
| 3.1.3 Unusual words | AAA | ✗ | There is no mechanism for explaining unusual words like jargon words. |
| 3.1.4 Abbreviations | AAA | ✗ | The expanded form of abbreviations was not provided for every abbreviations, for example i.e. |
| 3.1.5 Reading level | AAA | ||
| 3.1.6 Pronunciation | AAA | ✗ | There is no mechanism for supporting this purpose. |
| 3.2.1 On focus | A | ✓ | |
| 3.2.2 On input | A | ✓ | |
| 3.2.3 Consistent navigation | AA | ✓ | |
| 3.2.4 Consistent identification | AA | ✓ | |
| 3.2.5 Change on request | AAA | ✓ | |
| 3.3.1 Error identification | A | ✗ | In few web forms there not provided a text description of the error. |
| 3.3.2 Labels or instructions | A | ||
| 3.3.3 Error suggestion | AA | ✗ | The web form has not provided a suggestion to fix the error. |
| 3.3.4 Error prevention (legal, financial, data) | AA | ✗ | The user did not have the chance to check or control the data before sending web form. |
| 3.3.5 Help | AAA | ✗ | The website had not help documentation and specific instructions for the web forms. |
| 3.3.6 Error prevention (all) | AAA | ✗ | The user did not have the chance to check or control the data before sending web form. |
| 4.1.1 Parsing | A | ✓ | |
| 4.1.2 Name, role, value | A | ✓ |
Table 4.
Evaluation of usability for eMeistring.
| Success criteria | Level | Result | Evaluation |
|---|---|---|---|
| 1.1.1 Non-text content | A | ✗ | Missing the text for the image (empty attribute alt). |
| 1.2.1 Audio-only and video-only (prerecorded) | A | ||
| 1.2.2. Captions (prerecorded) | A | ||
| 1.2.3 Audio description or media alternative (prerecorded) | A | ||
| 1.2.4 Captions (live) | AA | ||
| 1.2.5 Audio description (prerecorded) | AA | ||
| 1.2.6 Sign language (prerecorded) | AAA | ||
| 1.2.7 Extended audio description (prerecorded) | AAA | ||
| 1.2.8 Media alternative (prerecorded) | AAA | ||
| 1.2.9 Audio-only (live) | AAA | ||
| 1.3.1 Info and relationships | A | ✓ | |
| 1.3.2 Meaningful sequence | A | ✗ | The text will not be read in right order by using assertive technology. |
| 1.3.3 Sensory characteristics | A | ✗ | Using a sensory characteristics instruction: “You will find the tasks in the left menu”. |
| 1.4.1 Use of color | A | ✗ | Using only green (to add row) and red (to delete row).. |
| 1.4.2 Audio control | A | ||
| 1.4.3 Contrast (minimum) | AA | ✗ | The measured contrast is 2,8:1. |
| 1.4.4 Resize text | AA | ✓ | |
| 1.4.5 Images of text | AA | ✗ | An image of “Example” instead of text. |
| 1.4.6 Contrast (enhanced) | AAA | ✗ | The measured contrast is 2,8:1. |
| 1.4.7 Low or no background audio | AAA | ||
| 1.4.8 Visual presentation | AAA | ✗ | User can not choose background color. |
| 1.4.9 Images of text (no exception) | AAA | ✗ | An image of “Example” instead of text. |
| 2.1.1 Keyboard | A | ✓ | |
| 2.1.2 No keyboard trap | A | ✓ | |
| 2.1.3 Keyboard (no exception) | AAA | ✓ | |
| 2.2.1 Timing adjustable | A | ||
| 2.2.2 Pause, stop, hide | A | ✓ | |
| 2.2.3 No timing | AAA | ✓ | |
| 2.2.4 Interruptions | AAA | ||
| 2.2.5 Re-authenticating | AAA | ||
| 2.3.1 Three flashes or below threshold | A | ✓ | |
| 2.3.2 Three flashes | AAA | ✓ | |
| 2.4.1 Bypass blocks | A | ✓ | |
| 2.4.2 Page titled | A | ✗ | The page title is not relevant to current page |
| 2.4.3 Focus order | A | ✓ | |
| 2.4.4 Link purpose (in context) | A | ✗ | The link does not describe the purpose - “click here”. |
| 2.4.5 Multiple ways | AA | ✗ | Only one way to navigate to a specific side. |
| 2.4.6 Headings and labels | AA | ✓ | |
| 2.4.7 Focus visible | AA | ✓ | |
| 2.4.8 Location | AAA | ✗ | The system do not shows the location of current page. |
| 2.4.9 Link purpose (link only) | AAA | ✗ | The link does not describe the purpose - “click here”. |
| 2.4.10 Section headings | AAA | ✗ | |
| 3.1.1 Language of page | A | ✗ | The language of the text is not provided in the source code. |
| 3.1.2 Language of parts | AA | ||
| 3.1.3 Unusual words | AAA | ✗ | There is no mechanism for explaining unusual words like jargon words. |
| 3.1.4 Abbreviations | AAA | ✗ | The expanded form of abbreviations was not provided for every abbreviations: e.g. |
| 3.1.5 Reading level | AAA | ||
| 3.1.6 Pronunciation | AAA | ✗ | Not supported for this purpose. |
| 3.2.1 On focus | A | ✓ | |
| 3.2.2 On input | A | ✗ | The data is stored while the user is filling the form. |
| 3.2.3 Consistent navigation | AA | ✓ | |
| 3.2.4 Consistent identification | AA | ✓ | |
| 3.2.5 Change on request | AAA | ✓ | |
| 3.3.1 Error identification | A | ✓ | |
| 3.3.2 Labels or instructions | A | ✗ | None instruction is provided. |
| 3.3.3 Error suggestion | AA | ✗ | None suggestions are provided when the error is automatic found. |
| 3.3.4 Error prevention (legal, financial, data) | AA | ✓ | |
| 3.3.5 Help | AAA | ✗ | No help documentation is provided. |
| 3.3.6 Error prevention (all) | AAA | ✓ | |
| 4.1.1 Parsing | A | ✗ | The system has logic error: By using check box you can be men and woman. |
| 4.1.2 Name, role, value | A | ✓ |
Table 5.
Evaluation of usability for Assistert Selvhjelp.
| Success criteria | Level | Result | Evaluation |
|---|---|---|---|
| 1.1.1 Non-text content | A | ✗ | Missing the text for the image (empty attribute alt). |
| 1.2.1 Audio-only and video-only (prerecorded) | A | ||
| 1.2.2. Captions (prerecorded) | A | ✗ | Missing captions on video lectures. |
| 1.2.3 Audio description or media alternative (prerecorded) | A | ✗ | The alternatives are not provided for any videos. |
| 1.2.4 Captions (live) | AA | ||
| 1.2.5 Audio description (prerecorded) | AA | ✗ | Audio descriptions are not provided for any videos. |
| 1.2.6 Sign language (prerecorded) | AAA | ✗ | Sign language is not supported for any videos. |
| 1.2.7 Extended audio description (prerecorded) | AAA | ✗ | Audio descriptions are not provided for any videos. |
| 1.2.8 Media alternative (prerecorded) | AAA | ✗ | A text alternative that serves the equivalent purpose is not provided. |
| 1.2.9 Audio-only (live) | AAA | ||
| 1.3.1 Info and relationships | A | ✗ | “Select element is missing an associated label”. |
| 1.3.2 Meaningful sequence | A | ✓ | |
| 1.3.3 Sensory characteristics | A | ✗ | Instructions: “See her answers on the next page”. |
| 1.4.1 Use of color | A | ✓ | |
| 1.4.2 Audio control | A | ||
| 1.4.3 Contrast (minimum) | AA | ✗ | The measured contrast is 3,0:1. |
| 1.4.4 Resize text: AA 3 | |||
| 1.4.5 Images of text | AA | ✗ | Using image of text to explain human body reactions for the mental disorder. |
| 1.4.6 Contrast (enhanced) | AAA | ✗ | The measured contrast is 3,0:1. |
| 1.4.7 Low or no background audio | AAA | ||
| 1.4.8 Visual presentation | AAA | ✗ | Impossible to choose background color for the website. |
| 1.4.9 Images of text (no exception) | AAA | ✗ | Using image of text to explain human body reactions for the mental disorder. |
| 2.1.1 Keyboard | A | ✗ | Cannot close dialogue box with keyboard. |
| 2.1.2 No keyboard trap | A | ✗ | Cannot close dialogue box with keyboard which leads to a keyboard trap. |
| 2.1.3 Keyboard (no exception) | AAA | ✗ | Cannot close dialogue box with keyboard which leads to a keyboard trap. |
| 2.2.1 Timing adjustable | A | ||
| 2.2.2 Pause, stop, hide | A | ||
| 2.2.3 No timing | AAA | ✓ | |
| 2.2.4 Interruptions | AAA | ||
| 2.2.5 Re-authenticating | AAA | ||
| 2.3.1 Three flashes or below threshold | A | ✓ | |
| 2.3.2 Three flashes | AAA | ||
| 2.4.1 Bypass blocks | A | ✓ | |
| 2.4.2 Page titled | A | ✗ | Web pages do not have titles that describe clearly the purpose. |
| 2.4.3 Focus order | A | ✓ | |
| 2.4.4 Link purpose (in context) | A | ✗ | The link does not describe the purpose and missing underline. |
| 2.4.5 Multiple ways | AA | ✓ | |
| 2.4.6 Headings and labels | AA | ✓ | |
| 2.4.7 Focus visible | AA | ✗ | Focus indicator is not visible by using the keyboard. |
| 2.4.8 Location | AAA | ✗ | The user does not know where web page is located in the website. |
| 2.4.9 Link purpose (link only) | AAA | ✗ | The link does not describe the purpose correctly: “click here to download the form”. |
| 2.4.10 Section headings | AAA | ✓ | |
| 3.1.1 Language of page | A | ✓ | |
| 3.1.2 Language of parts | AA | ||
| 3.1.3 Unusual words | AAA | ✗ | There is no mechanism for explaining unusual words like jargon words. |
| 3.1.4 Abbreviations | AAA | ✗ | The expanded form of abbreviations was not provided for every abbreviations e.g. PTSD. |
| 3.1.5 Reading level | AAA | ||
| 3.1.6 Pronunciation | AAA | ✗ | There is no mechanism for supporting this purpose. |
| 3.2.1 On focus | A | ✓ | |
| 3.2.2 On input | A | ✓ | |
| 3.2.3 Consistent navigation | AA | ✓ | |
| 3.2.4 Consistent identification | AA | ✓ | |
| 3.2.5 Change on request | AAA | ✗ | Using pop-up forms to evaluate treatment at the end of the treatment. |
| 3.3.1 Error identification | A | ✓ | |
| 3.3.2 Labels or instructions | A | ✓ | |
| 3.3.3 Error suggestion | AA | ✓ | |
| 3.3.4 Error prevention (legal, financial, data) | AA | ✗ | The system does not have a mechanism for reviewing data, confirming or correcting information. |
| 3.3.5 Help | AAA | ✗ | There system does not provide a detail descriptions and instructions for all the web forms. |
| 3.3.6 Error prevention (all) | AAA | ✗ | The system does not have a mechanism for reviewing data, confirming or correcting information. |
| 4.1.1 Parsing | A | ✓ | |
| 4.1.2 Name, role, value | A | ✓ |
Table 6.
Evaluation of usability for MoodGym.
| Success criteria | Level | Result | Evaluation |
|---|---|---|---|
| 1.1.1 Non-text content | A | ✗ | Missing the text for the image (empty attribute alt). |
| 1.2.1 Audio-only and Video-only (prerecorded) | A | ||
| 1.2.2. Captions (prerecorded) | A | ||
| 1.2.3 Audio description or media alternative (prerecorded) | A | ||
| 1.2.4 Captions (live) | AA | ||
| 1.2.5 Audio description (prerecorded) | AA | ||
| 1.2.6 Sign language (prerecorded) | AAA | ||
| 1.2.7 Extended audio description (prerecorded) | AAA | ||
| 1.2.8 Media alternative (prerecorded) | AAA | ||
| 1.2.9 Audio-only (live) | AAA | ||
| 1.3.1 Info and relationships | A | ✗ | Field “password” missing associated label. |
| 1.3.2 Meaningful sequence | A | ✓ | |
| 1.3.3 Sensory characteristics | A | ✗ | Instruction: “Please click on the yellow next button to left”. |
| 1.4.1 Use of color | A | ✗ | Using color only to highlight important information. |
| 1.4.2 Audio control | A | ||
| 1.4.3 Contrast (minimum) | AA | ✗ | The measured contrast is 3,1:1. |
| 1.4.4 Resize text | AA | ✗ | The text is unclear when zooming to 200%. |
| 1.4.5 Images of text | AA | ✓ | |
| 1.4.6 Contrast (enhanced) | AAA | ✗ | The measured contrast is 3,1:1. |
| 1.4.7 Low or no background audio | AAA | ||
| 1.4.8 Visual presentation | AAA | ✗ | Cannot choose background color. |
| 1.4.9 Images of text (no exception) | AAA | ✓ | |
| 2.1.1 Keyboard | A | ✓ | |
| 2.1.2 No keyboard trap | A | ✓ | |
| 2.1.3 Keyboard (no exception) | AAA | ✓ | |
| 2.2.1 Timing adjustable | A | ||
| 2.2.2 Pause, stop, hide | A | ✗ | Cannot pause, stop or hide the animation. |
| 2.2.3 No timing | AAA | ✓ | |
| 2.2.4 Interruptions | AAA | ||
| 2.2.5 Re-authenticating | AAA | ||
| 2.3.1 Three flashes or below threshold | A | ✓ | |
| 2.3.2 Three flashes | AAA | ✓ | |
| 2.4.1 Bypass blocks | A | ✗ | There is not provided a menu or shortcuts to skip to the main content. |
| 2.4.2 Page titled | A | ✓ | |
| 2.4.3 Focus order | A | ✗ | The website has 319 questions on a single questionnaire, should split it up. |
| 2.4.4 Link purpose (in context) | A | ✗ | The purpose of the link is not described: “click here”. |
| 2.4.5 Multiple ways | AA | ✗ | There is no menu at the beginning of the website; buttons of “next” and “previous”. |
| 2.4.6 Headings and labels | AA | ✓ | |
| 2.4.7 Focus visible | AA | ✓ | |
| 2.4.8 Location | AAA | ✗ | Information about the user's location within a set of web pages is unavailable. |
| 2.4.9 Link purpose (link only) | AAA | ✗ | The purpose of the link is not described: “click here”. |
| 2.4.10 Section headings | AAA | ✗ | Sections are not used to organize the content. |
| 3.1.1 Language of page | A | ✗ | The software code has an invalid language code. |
| 3.1.2 Language of parts | AA | ||
| 3.1.3 Unusual words | AAA | ✗ | There is no mechanism for explaining unusual words. |
| 3.1.4 Abbreviations | AAA | ✗ | The expanded form of abbreviations was not provided for every abbreviations, like “e.g.”. |
| 3.1.5 Reading level | AAA | ||
| 3.1.6 Pronunciation | AAA | ✗ | There is no mechanism for supporting this purpose. |
| 3.2.1 On focus | A | ✓ | |
| 3.2.2 On input | A | ✓ | |
| 3.2.3 Consistent navigation | AA | ✓ | |
| 3.2.4 Consistent identification | AA | ✓ | |
| 3.2.5 Change on request | AAA | ✓ | |
| 3.3.1 Error identification | A | ✗ | The web form did not show where the error was detected. |
| 3.3.2 Labels or instructions | A | ✗ | The text label was empty. |
| 3.3.3 Error suggestion | AA | ✗ | The web form has not provided a suggestion for fix the error. |
| 3.3.4 Error prevention (legal, financial, data) | AA | ✗ | The user did not have the chance to check or control the data before sending web form. |
| 3.3.5 Help | AAA | ✗ | The website had not help documentation and specific instructions for the web forms. |
| 3.3.6 Error prevention (all) | AAA | ✗ | The user did not have the chance to check or control the data before sending web form. |
| 4.1.1 Parsing | A | ✓ | |
| 4.1.2 Name, role, value | A | ✓ |
Table 7.
Evaluation of usability for Psyktools (Youwell).
| Success criteria | Level | Result | Evaluation |
|---|---|---|---|
| 1.1.1 Non-text content | A | ✗ | Missing the text for the image. |
| 1.2.1 Audio-only and video-only (prerecorded) | A | ||
| 1.2.2. Captions (prerecorded) | A | ✗ | Missing captions on video lectures. |
| 1.2.3 Audio description or media alternative (prerecorded) | A | ✗ | The alternatives are not provided for any videos. |
| 1.2.4 Captions (live) | AA | ||
| 1.2.5 Audio description (prerecorded) | AA | ✗ | Audio descriptions are not provided for any videos. |
| 1.2.6 Sign language (prerecorded) | AAA | ✗ | Sign language is not supported for any videos. |
| 1.2.7 Extended audio description (prerecorded) | AAA | ✗ | Audio descriptions are not provided for any videos. |
| 1.2.8 Media alternative (prerecorded) | AAA | ✗ | A text alternative that serves the equivalent purpose is not provided. |
| 1.2.9 Audio-only (live) | AAA | ||
| 1.3.1 Info and relationships | A | ✓ | |
| 1.3.2 Meaningful sequence | A | ✓ | |
| 1.3.3 Sensory characteristics | A | ✓ | |
| 1.4.1 Use of color | A | ✓ | |
| 1.4.2 Audio control | A | ||
| 1.4.3 Contrast (minimum) | AA | ✗ | The measured contrast is 2,1:1. |
| 1.4.4 Resize text | AA | ✓ | |
| 1.4.5 Images of text | AA | ✓ | |
| 1.4.6 Contrast (enhanced) | AAA | ✗ | The measured contrast is 2,1:1. |
| 1.4.7 Low or no background audio | AAA | ||
| 1.4.8 Visual presentation | AAA | ✗ | Impossible to choose foreground and background color. |
| 1.4.9 Images of text (no exception) | AAA | ✓ | |
| 2.1.1 Keyboard | A | ✓ | |
| 2.1.2 No keyboard trap | A | ✓ | |
| 2.1.3 Keyboard (no exception) | AAA | ✓ | |
| 2.2.1 Timing adjustable | A | ||
| 2.2.2 Pause, stop, hide | A | ||
| 2.2.3 No timing | AAA | ✓ | |
| 2.2.4 Interruptions | AAA | ||
| 2.2.5 Re-authenticating | AAA | ||
| 2.3.1 Three flashes or below threshold | A | ✓ | |
| 2.3.2 Three flashes | AAA | ||
| 2.4.1 Bypass blocks | A | ✓ | |
| 2.4.2 Page titled | A | ✓ | |
| 2.4.3 Focus order | A | ✓ | |
| 2.4.4 Link purpose (in context) | A | ✓ | |
| 2.4.5 Multiple ways | AA | ✗ | Only one way to navigate to a specific side. |
| 2.4.6 Headings and labels | AA | ✓ | |
| 2.4.7 Focus visible | AA | ✗ | Focus indicator is not visible by using the keyboard. |
| 2.4.8 Location | AAA | ✗ | The user does not know where web page is located in the website. |
| 2.4.9 Link purpose (link only) | AAA | ✓ | |
| 2.4.10 Section headings | AAA | ✗ | Sections are not always used to organize the content |
| 3.1.1 Language of page | A | ✗ | The software code does not provide the language of the page. |
| 3.1.2 Language of parts | AA | ||
| 3.1.3 Unusual words | AAA | ✗ | There is no mechanism for explaining unusual words, like “temporallappen”. |
| 3.1.4 Abbreviations | AAA | ✗ | The expanded form of abbreviations was not provided for every abbreviations, like “e.g.” |
| 3.1.5 Reading level | AAA | ||
| 3.1.6 Pronunciation | AAA | ✗ | There is no mechanism for supporting this purpose. |
| 3.2.1 On focus | A | ✓ | |
| 3.2.2 On input | A | ✓ | |
| 3.2.3 Consistent navigation | AA | ✓ | |
| 3.2.4 Consistent identification | AA | ✓ | |
| 3.2.5 Change on request | AAA | ✓ | |
| 3.3.1 Error identification | A | ✗ | There is no mechanism for validating the input from the user. |
| 3.3.2 Labels or instructions | A | ✓ | |
| 3.3.3 Error suggestion | AA | ||
| 3.3.4 Error prevention (legal, financial, data) | AA | ✗ | The user did not have the chance to check or control the data before sending web form. |
| 3.3.5 Help | AAA | ✗ | The website had not help documentation and specific instructions for the web forms. |
| 3.3.6 Error prevention (all) | AAA | ✗ | The user did not have the chance to check or control the data before sending web form. |
| 4.1.1 Parsing | A | ✓ | |
| 4.1.2 Name, role, value | A | ✓ |
Table 8.
Evaluation of usability for Internetpsykiatri.
| Success criteria | Level | Result | Evaluation |
|---|---|---|---|
| 1.1.1 Non-text content | A | ✗ | Missing the text for the image. |
| 1.2.1 Audio-only and video-only (prerecorded) | A | ||
| 1.2.2. Captions (prerecorded) | A | ||
| 1.2.3 Audio description or media alternative (prerecorded) | A | ||
| 1.2.4 Captions (live) | AA | ||
| 1.2.5 Audio description (prerecorded) | AA | ||
| 1.2.6 Sign language (prerecorded) | AAA | ||
| 1.2.7 Extended audio description (prerecorded) | AAA | ||
| 1.2.8 Media alternative (prerecorded) | AAA | ||
| 1.2.9 Audio-only (live) | AAA | ||
| 1.3.1 Info and relationships | A | ✓ | |
| 1.3.2 Meaningful sequence | A | ✗ | |
| 1.3.3 Sensory characteristics | A | ✗ | Instruction: “Print out at the top right of the screen”. |
| 1.4.1 Use of color | A | ✓ | |
| 1.4.2 Audio control | A | ||
| 1.4.3 Contrast (minimum) | AA | ✗ | The measured contrast is 2,9:1. |
| 1.4.4 Resize text | AA | ✓ | |
| 1.4.5 Images of text | AA | ✗ | Tables are pasted as a picture. |
| 1.4.6 Contrast (enhanced) | AAA | ✗ | The measured contrast is 2,9:1. |
| 1.4.7 Low or no background audio | AAA | ||
| 1.4.8 Visual presentation | AAA | ✗ | Cannot choose background color or foreground color. |
| 1.4.9 Images of text (no exception) | AAA | ✗ | Tables are pasted as a picture. |
| 2.1.1 Keyboard | A | ✗ | Cannot expand the text field by using keyboard only. |
| 2.1.2 No keyboard trap | A | ✓ | |
| 2.1.3 Keyboard (no exception) | AAA | ✗ | Cannot expand the text field by using keyboard only. |
| 2.2.1 Timing adjustable | A | ||
| 2.2.2 Pause, stop, hide | A | ||
| 2.2.3 No timing | AAA | ✓ | |
| 2.2.4 Interruptions | AAA | ||
| 2.2.5 Re-authenticating | AAA | ||
| 2.3.1 Three flashes or below threshold | A | ✓ | |
| 2.3.2 Three flashes | AAA | ||
| 2.4.1 Bypass blocks | A | ✓ | |
| 2.4.2 Page titled | A | ✗ | Using the same page title on every single page. |
| 2.4.3 Focus order | A | ✓ | |
| 2.4.4 Link purpose (in context) | A | ✗ | Purpose of link is not provided. |
| 2.4.5 Multiple ways | AA | ✓ | |
| 2.4.6 Headings and labels | AA | ✓ | |
| 2.4.7 Focus visible | AA | ✓ | |
| 2.4.8 Location | AAA | ✓ | |
| 2.4.9 Link purpose (link only) | AAA | ✗ | Purpose of link is not provided. |
| 2.4.10 Section headings | AAA | ✓ | |
| 3.1.1 Language of page | A | ✗ | According to ISO-639-1: Two letter language code is missing. |
| 3.1.2 Language of parts | AA | ||
| 3.1.3 Unusual words | AAA | ✗ | There is no mechanism for explaining unusual words, like “Behavioral activation”. |
| 3.1.4 Abbreviations | AAA | ✗ | The expanded form of abbreviations was not provided for every abbreviations, like “e.g.” |
| 3.1.5 Reading level | AAA | ||
| 3.1.6 Pronunciation | AAA | ✗ | There is no mechanism for supporting this purpose. |
| 3.2.1 On focus | A | ✓ | |
| 3.2.2 On input | A | ✓ | |
| 3.2.3 Consistent navigation | AA | ✓ | |
| 3.2.4 Consistent identification | AA | ✓ | |
| 3.2.5 Change on request | AAA | ✓ | |
| 3.3.1 Error identification | A | ✗ | Several webpages have not a mechanism for validating the input from the user. |
| 3.3.2 Labels or instructions | A | ✗ | The activity form does not explain how to calculate the sleep effect. |
| 3.3.3 Error suggestion | AA | ✗ | The forms do not indicate where the error is. |
| 3.3.4 Error prevention (legal, financial, data) | AA | ✗ | The user did not have the chance to check or control the data before sending web form. |
| 3.3.5 Help | AAA | ✗ | The website had not help documentation and specific instructions for the web forms. |
| 3.3.6 Error prevention (all) | AAA | ✗ | The user did not have the chance to check or control the data before sending web form. |
| 4.1.1 Parsing | A | ✓ | |
| 4.1.2 Name, role, value | A | ✓ |
3.1. MoodGym (2.0)
MoodGym is an Internet-delivered treatment system designed for patient to prevent and manage symptoms of depression and anxiety. Moodgym 2.0 was released in 2017 and is a new version of MoodGym, using more modern technologies and is more user-friendly than the previous one (Table 3). The system offers a digital self-help interactive program first introduced in Australia in 2001, today it has more than 1 million users worldwide.
3.1.1. Clinical usage
MoodGym is a treatment system targeting adolescents and adults with symptoms of depression. The treatment is based on Internet-delivered cognitive behavioral therapy and is implemented as a stand-alone intervention. There are no opportunity for communication between the patient and the therapist through the system, but therapist guidance may be delivered via telephone or face-to-face consultations. The patients are self-referred. The treatment has shown to have small but significant effect in community settings reported in a randomized controlled trial (Christensen and Griffiths, 2001). However, it has also been reported that MoodGym have no additional effect to a “treatment as usual” follow up by a General Practitioner.
3.1.2. Usability
The system partially satisfies the usability criteria. There are several improvements related to the usability of the system, especially considering the criteria about presentation of the content. In practice, this means that the system does not strictly follow the principles of WCAG 2.0 (Table 3). The system has only fulfilled few of the criteria that belongs to the first (perceivable) principle of WCAG 2.0. The system was using sensor characteristics to provide the information about how to remove rows in the table, by using only symbols. The use of symbols only makes it difficult for people with visual disability to locate and read the symbols. The system does not make the content accessible for people with reduced vision by using two different colors to highlight important content and using white text on grey background. The second principle of WCAG 2.0 was partially fulfilled by the system. It is visible where you are navigating on the website and you may reach all the functionalities by using keyboard only. The user must have the possibilities to pause, stop or hide animations that start automatically. By not having this opportunity the user may be distracted, which leads to poor user experience. In the system you may find blue underlined text that looks like links but does not behave like links. This is called false affordance in interaction design (Gaver, 1991). The system shows the user's location within a set of web pages for example: Home – Module 1 – Introduction. This will help users to keep track of their treatment progression.
The third principle of WCAG 2.0 has not been followed by the system with few exceptions. Every single form provides instructions about how to fill out the form and is providing error identification when an error is occurring by text description. However, the forms do not provide any error recovery suggestions while filling out the forms. A process for confirming the answers from questionnaires is required by the criteria, but is not supported by the system. The system has fulfilled every criterion that belongs to the fourth and last principle, robustness of WCAG 2.0.
3.1.3. Responsive web design
The system has a good solution for responsive web design, but still need some improvement to meet the criteria.
The system is not browser and device independent, hence it do not meet one of the seven criteria for responsive web design. MoodGym 2.0 is using a viewport that make sure that the content is not wider than the screen size of the device. This means that the images are scaled up or down relative to the screen size. The newest HTML5 technology is used to play video and animations, which support browser independence. Furthermore, the majority of the content is readable on small devices, but one may find it difficult to read the text on clickable buttons in the menu.
We also evaluated the systems for how they provide user interaction and their security and privacy mechanisms. Importantly, there was found only a few issues related to the security criteria that may concern the systems. Details about these evaluations are left out from the paper due to space limitations.
4. Discussion
The main result from the study shows that the systems mostly satisfy the evaluation criteria for interaction and security, but less for usability and responsive design. The systems only partially fulfill the universal design requirement from WCAG 2.0 (Table 9).
Table 9.
Evaluation of Internet-delivered treatment systems according to WCAG 2.0)
| Systems | Levels |
|||||
|---|---|---|---|---|---|---|
| A |
AA |
AAA |
||||
| Number | Percentage | Number | Percentage | Number | Percentage | |
| eMeistring | 10/20 | 50.0% | 6/10 | 60.0% | 5/15 | 33.3% |
| Assistert Selvhjelp | 12/21 | 57.1% | 6/11 | 54.5% | 2/17 | 11.8% |
| MoodGYM | 9/20 | 45.0% | 5/10 | 50.0% | 5/15 | 33.3% |
| MoodGYM 2.0 | 13/21 | 61.9% | 6/10 | 60.0% | 6/16 | 37.5% |
| Psyktools | 16/21 | 76.2% | 5/10 | 50.0% | 5/17 | 29.4% |
| Internetpsykiatri | 11/19 | 57.9% | 6/10 | 60.0% | 4/14 | 28.6% |
Despite the fact that Internet-delivered treatments are nowadays disseminated into regular mental health care, few of these systems have been released with consideration of their quality with regard to usability. It is well known that Internet and WEB technologies have been rapidly evolving during the last decade, but most Internet-delivered treatment systems in use today were developed with older technologies and have not been updated in accordance with the evolution of the technology. This was especially affecting the criteria on responsive web design. We observed several errors concerning browser and device independence, e.g. images and buttons were not adapted when the size of the screen was changed. Our findings indicate that the latest developed systems (MoodGym 2.0 and Psyktools) are using technologies that follow today's standard, however Psyktools was the only system that fully satisfies the responsive web design criteria.
On the positive side, our study indicates that the evaluated Internet-delivered mental health treatment systems have adequate solutions concerning security and privacy of patient data, which is of great importance to establish trust between the health care providers and the patients. All of the systems use the encrypted HTTPS communication protocol, hence they ensure secure data transfer between the patients and the systems. The authentication mechanism could be improved for some of the systems, but two of the systems (Internetpsykiatri and eMeistring) uses a robust two factor authentication mechanism making it difficult for intruders to login to the system. The scoring on the usability criteria was not satisfactory for any of the systems in the evaluation. All of the systems have issues regarding the universal design guideline WCAG 2.0. The following errors were observed: the navigation structure of the system is not adapted to the patients’ needs, use of color and contrast is making it difficult for people with visual disability such as color blindness, images and text was not scaling well when the resolution and size was changed. However, we see a tendency that the newer treatment systems score better on usability, but they still need to improve to meet the criteria. Especially were the results on the WCAG 2.0 evaluation below expectation, with an average score on 58% on level A, 56% on level AA and only 29% on level AAA.
A limitation of the present study was that we were not able to assess the architecture of the present systems. In an early stage of the study we tried to evaluate the architecture of the systems, but unfortunately, we did not have access to this part of the systems. Therefore, it could be interesting for further research to investigate if the systems are following the standards for architecture and interoperability like HL7-FHIR (HL7, 2019), LOINC (Regenstrief Institute, 2018), openEHR (openEHR Foundation, 2018) and ICD-11 (World Health Organization, 2018). One other interesting direction of further research will be to systematically study the possibilities Internet-delivered treatment systems has for integration with other health care system like Electronic health records and medical charts. One other limitation of the study is that we only reviewed the patient interface of the treatment systems. One can argue that if the therapist interface to the program is not good enough, the therapist might not be motivated to use the system, could make mistakes or could miss important information about the patient. We would recommend that a similar study for the usability of the therapist interfaces should be performed.
4.1. Future research
As described in the previous section are today's Internet-delivered treatment systems not following the state-of-the-art knowledge of usability and Internet technology. We propose that it should be conducted more research on the relationship between usability, technology and outcome of Internet-delivered treatment. We also advice to involve end-users in the design and continuous evaluation of treatment systems. Usability should preferably be considered from the start of system development, however few Internet-delivered treatment systems have undergone usability evaluation prior to dissemination. We propose that it should be developed benchmark tests for systematic evaluation of Internet-delivered treatment systems based on usability heuristics, universal design principles (WCAG 2.0) and security and privacy principle, to enhance technology acceptance by end-users and to guide system modification.
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This publication is part of the INTROducing Mental health through Adaptive Technology (INTROMAT) project, funded by the Norwegian Research Council (agreement 259293).
Footnotes
Contributor Information
Aravinthan Yogarajah, Email: aravinthan_92@hotmail.com.
Yngve Lamo, Email: yngve.lamo@hvl.no.
References
- Andersson G., Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn. Behav. Ther. 2018;38(4):196–205. doi: 10.1080/16506070903318960. (2009) [DOI] [PubMed] [Google Scholar]
- Andersson G., Cuijpers P., Carlbring P., Riper H., Hedman E. Guided internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry. 2014;13(3):288–295. doi: 10.1002/wps.20151. (2014) [DOI] [PMC free article] [PubMed] [Google Scholar]
- Baumeister H., Reichler L., Munzinger M., Lin J. The impact of guidance on internet-based mental health interventions — a systematic review. Internet Interv. 2014;1 doi: 10.1016/j.invent.2014.08.003. [DOI] [Google Scholar]
- Bohyun Kim. Responsive web design, discoverability, and mobile challenge. Library Technology Reports. 2013;49(6):29–39. [Google Scholar]
- B. Caldwell, M. Cooper, L.G. Reid, and G. Vanderheiden. Web content accessibility guide-lines (WCAG) 2.0. Available: https://www.w3.org/TR/2008/REC-WCAG20-20081211/, 2008. accessed 21-February-2018.
- Christensen H., Griffiths K. Moodgym: a web-based intervention for the treatment and prevention of depression. Aust. N. Z. J. Psychiatry. 2001;35(8):A4. [Google Scholar]
- Ebert D.D., Daele T., Nordgreen T., Karekla M., Compare T.A., Zarbo C., Brugnera A., Oeverland S., Trebbi G., Jensen K.L., Kaehlke F., on behalf of the EFPA E-Health Taskforce Internet- and mobile-based psychological interventions: applications, efficacy, and potential for improving mental health a report of the EFPA E-health taskforce. Eur. Psychol. 2018;23(2):167–187. doi: 10.1027/1016-9040/a000318. [DOI] [Google Scholar]
- Farrer L.M., Griffiths K.M., Christensen H., Mackinnon A.J., Batterham P.J. Predictors of adherence and outcome in internet-based cognitive behavior therapy delivered in a telephone counseling setting. Cogn. Ther. Res. 2014;38(3):358–367. [Google Scholar]
- Folker A.P., Mathiasen K., Lauridsen S.M., Stenderup E., Dozeman E., Folker M.P. Implementing internet-delivered cognitive behavior therapy for common mental health disorders: a comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interv. 2018;11:60–70. doi: 10.1016/j.invent.2018.02.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Furmark T., Carlbring P., Hedman E., Sonnenstein A., Clevberger P., Bohman B., Eriksson A., Hållén A., Frykman M., Holmström A. Guided and unguided self-help for social anxiety disorder: randomised controlled trial. Br. J. Psychiatry. 2009;195(5):440–447. doi: 10.1192/bjp.bp.108.060996. [DOI] [PubMed] [Google Scholar]
- Gaver W.W. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM; 1991. Technology affordances; pp. 79–84. [Google Scholar]
- Hedman E., Ljótsson B., Kaldo V., Hesser H., El Alaoui S., Kraepelien M., Andersson E., Rück C., Svanborg C., Andersson G. Effectiveness of internet-based cognitive behaviour therapy for depression in routine psychiatric care. J. Affect. Disord. 2014;155:49–58. doi: 10.1016/j.jad.2013.10.023. [DOI] [PubMed] [Google Scholar]
- HL7 HL7 FHIR. 2019. https://www.hl7.org/fhir/ Available.
- Institute of Electrical and Electronics Engineers (IEEE) Best practices in responsive web design. 2012. https://www.ieee.org/about/webteam/resources/responsive_design/best_practices.html Available.
- Kaldo V., Jernelöv S., Blom K., Ljótsson B., Brodin M., Jörgensen M., Kraepelien M., Rück C., Lindefors N. Guided internet cognitive behavioral therapy for insomnia compared to a control treatment–a randomized trial. Behav. Res. Ther. 2015;71:90–100. doi: 10.1016/j.brat.2015.06.001. [DOI] [PubMed] [Google Scholar]
- Karyotaki E., Ebert D.D., Donkin L., Riper H., Twisk J., Burger S.…Cuijpers P. Do guided internet-based interventions result in clinically relevant changes for patients with depression? An individual participant data meta-analysis. Clinical Psychology Review. 2018;63:80–92. doi: 10.1016/j.cpr.2018.06.007. [DOI] [PubMed] [Google Scholar]
- Kraut R.E., Fussell S.R., Brennan S.E., Siegel J. Understanding effects of proximity on collaboration: implications for technologies to support remote collaborative work. Distributed Work. 2002:137–162. [Google Scholar]
- Ljótsson B., Falk L., Vesterlund A.W., Hedman E., Lindfors P., Rück C., Hursti T., Andréewitch S., Jansson L., Lindefors N. Internet-delivered exposure and mindfulness based therapy for irritable bowel syndrome–a randomized controlled trial. Behav. Res. Ther. 2010;48(6):531–539. doi: 10.1016/j.brat.2010.03.003. [DOI] [PubMed] [Google Scholar]
- Marcotte E. 2011. Responsive Web Design, a Book Apart. [Google Scholar]
- Nordgreen T., Gjestad R., Andersson G., Carlbring P., Havik O.E. The implementation of guided internet-based cognitive behaviour therapy for panic disorder in a routine-care setting: effectiveness and implementation efforts. Cogn. Behav. Ther. 2017 doi: 10.1080/16506073.2017.1348389. [DOI] [PubMed] [Google Scholar]
- Nordgreen T., Gjestad R., Andersson G., Carlbring P., Havik O.E. The effectiveness of guided internet-based cognitive behavioral therapy for social anxiety disorder in a routine care setting. Internet Interv. 2018;13:24–29. doi: 10.1016/j.invent.2018.05.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Nordgreen T., Blom K., Andersson G., Carlbring P., Havik O.E. Effectiveness of guided internet-delivered treatment for major depression in routine mental healthcare-an open study. Internet Interv. 2019:100274. doi: 10.1016/j.invent.2019.100274. [DOI] [PMC free article] [PubMed] [Google Scholar]
- openEHR Foundation openEHR. 2018. http://www.openehr.org/releases/BASE/latest/docs/architecture_overview/architecture_overview.html#_purpose Available.
- Regenstrief Institute Logical observation identifiers names and codes. 2018. https://loinc.org/ Available.
- Simpson H.B., Marcus S.M., Zuckoff A., Franklin M., Foa E.B. Patient adherence to cognitive-behavioral therapy predicts long-term outcome in obsessive-compulsive disorder. J. Clin. Psychiatry. 2012;73(9):1265. doi: 10.4088/JCP.12l07879. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Titov N., Blake D., Nielssen O., Staples L., Hadjistavropoulos H., Nugent M., Adlam K., Nordgreen T., Bruvik K.H., Hovland A., Repål A., Mathiasen K., Kraepelien M., Blom K., Svanborg C., Lindefors N., Kaldo V. ICBT in routine care: a descriptive analysis of successful clinics in five countries. Internet Interv. 2018;13:108–115. doi: 10.1016/j.invent.2018.07.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Titov N., Dear B., Nielssen O., Staples L., Hadjistavropoulos H., Nugent M., Adlam K., Nordgreen T., Bruvik K.H., Hovland A. Icbt in routine care: a descriptive analysis of successful clinics in five countries. Internet Interv. 2018;13:108–115. doi: 10.1016/j.invent.2018.07.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- World Health Organization ICD-11 for mortality and morbidity statistics. 2018. https://icd.who.int/browse11/l-m/en Available.