User feedback and usability testing of an online training and support program for dementia carers

Abstract

Background

eHealth interventions have been explored to provide convenient support and training to informal dementia carers. Design and usability issues may however hinder user acceptance of Web-based interventions. iSupport is an online program developed by World Health Organization (WHO) to support informal dementia carers.

Purpose

This study gathers user-feedback and evaluates the usability of the European-Portuguese version of iSupport.

Methods

A mixed-methods study was conducted. Two focus groups discussions (n = 15) and 15 usability test sessions were conducted with carers and health/social support professionals to collect data on user satisfaction and requirements on the program's contents and Web interface. The usability testing protocol comprised pre/post-test surveys, a structured interview and observation of task performance in the Web platform. Content analysis of text data was performed by two independent coders.

Results

Success rates were superior to 80% for most tasks. An average score of 89.5 on the System Usability Scale demonstrated an excellent perception of the program's usability. The program was perceived as trustworthy, and participants were globally satisfied with its aesthetics and easiness of use. Feedback and personalization were valued functional requirements, while no consensus was found for mood self-assessment or professional/peer interaction features. Lessons were considered culturally fit, thematically pertinent, and comprehensive. Suggestions for improvement were wide-ranging from program's layout to accessibility, usability, functional requirements, content format and lessons' themes.

Conclusions

This study derives user requirements for an evidence-based program for dementia carers. Both the study findings and usability assessment methodology can be imported to the development of similar programs worldwide.

Highlights

• •iSupport-Portugal was appraised as trustworthy and easy to use.
• •An average score of 89.5 on SUS shows a good perception of iSupport's usability.
• •Feedback and personalization are functional requirements valued by users.
• •Improvements on program aesthetics, usability, accessibility and contents are needed.
• •iSupport is a feasible mean of providing accessible training for dementia carers.

1. Introduction

The unparalleled phenomenon of population ageing is having a significant impact on the advent of the dementia epidemic (Prince et al., 2013). Worldwide, 50 million people are estimated to live with dementia (World Health Organization, 2019a). This group of disorders is an important cause of dependency among older adults (Sousa et al., 2009). The provision of consistent care to dependent persons with dementia is often guaranteed by informal carers, who provide unpaid and ongoing support with basic or instrumental activities of daily living (Roth et al., 2015). Dementia carers are frequently exposed to multiple sources of stress, and are at a greater risk of experiencing anxiety and mood disorders when compared to the general population (World Health Organisation, 2015).

eHealth interventions have been explored for their potential to bridge gaps in care provision, either as adjunct or alternative solutions to usual care. eHealth typically includes health services and information provided or enhanced via the Internet and related technologies (Eysenbach, 2001). These interventions have been recognized by its ubiquity, accessibility, convenience, affordability and potential for scalability (Egan and Pot, 2016). Such attributes may offer opportunities to expand training and support interventions for dementia carers, which is a strategic priority affirmed in ageing and dementia action plans (World Health Organization and Alzheimer's Disease International, 2012; World Health Organization, 2017). Research on the acceptance of online training and support interventions for these carers have been encouraging (Chiu et al., 2009; Cristancho-Lacroix et al., 2014; Marziali and Garcia, 2011; Pot et al., 2015). Systematic reviews and meta-analysis also reported beneficial effects of such interventions on carers' perceived stress and burden (Boots et al., 2014; Parra-Vidales et al., 2017; Hopwood et al., 2018), anxiety and depression symptoms (Boots et al., 2014; Parra-Vidales et al., 2017; Hopwood et al., 2018; Yinan et al., 2019; Egan et al., 2018), and self-efficacy (Boots et al., 2014; Parra-Vidales et al., 2017).

Although eHealth solutions can facilitate the delivery of training and support to dementia carers, there are significant and diverse challenges for its successful implementation. Challenges might be technology-related (e.g., design, usability), organizational (e.g., lack of awareness; gap between research and solutions), socioeconomic (e.g., lack of funding) or ethical (e.g., privacy, dehumanization of care) (Hassan, 2020). High dropout rates were documented for online interventions (31% on average and reaching up 83% in (Melville et al., 2010)). In a recent review, higher attrition was reported for technology driven psychoeducation-skill building interventions aimed at dementia carers, as compared to those interventions delivered face-to-face (average attrition rate of 23% and 8.7%, respectively) (Kishita et al., 2018). One of the essential aspects for the participants' satisfaction with eHealth interventions, as well as for its uptake and retention in, is whether the design of such interventions is user-centered (Ahern, 2007; Holthe et al., 2018). Indeed, it was hypothesized that usability and interface issues are among the factors influencing attrition in eHealth interventions (Eysenbach, 2005). Empirical studies on the relationship of usability with intervention attrition or adherence are scant. However, usability measures were found to be associated with indicators of adherence (Baumel and Yom-Tov, 2018); and it was suggested that embedding persuasive design features in an interface (e.g., reminders, praise, dialog support) may promote higher adherence to internet interventions (Kelders et al., 2012). Usability studies are instrumental to uncover design problems in eHealth interventions, explore user experiences, and are useful for evaluating a system/program before an effectiveness trial. Usability refers to whether a system can be used in a specified context to achieve specified goals with effectiveness, efficiency and satisfaction (International Organization for Standardization, 2010). It is a multidimensional concept comprising learnability, memorability, efficiency, errors while using the system, and user satisfaction (Fisk et al., 2004; Nielsen, 2012). Usability is an important dimension for determining whether a system/program is useful (Nielsen, 2012).

This study sough to gather user feedback and evaluate the usability of an online knowledge and skills training program aimed at dementia carers: ‘iSupport-Portugal’. The program consists in a culturally adapted version (Teles et al., 2020a) of ‘iSupport for dementia’ authored by the World Health Organization (Pot et al., 2019). By appraising the usability and analysing user feedback on the European-Portuguese version of iSupport, this study offered insights on the feasibility of a one-of-a-kind resource for dementia carers in Portugal. Study findings and methods can inform the development of similar programs worldwide.

2. Overview of iSupport-Portugal

iSupport is an online program aimed at providing education, skills training and social support to informal carers of persons with dementia (Pot et al., 2019). It resorts to problem-solving and cognitive behavioural therapy techniques, including psycho-education, behavioural activation, cognitive reframing, relaxation and antecedent-behaviour-consequence (ABC) analysis (Pot et al., 2019). The program comprises twenty-three lessons distributed over five modules, which cover well-established themes on caregiving and carers' self-care (Fig. 1).

Fig. 1.

Overview of modules and lessons on iSupport.

Adapted from (World Health Organization, 2019b) (generic version of iSupport).

iSupport was originally designed to be self-managed, self-paced and without therapist, peer, or other personal input. Via a Web interface, registered users can access the program anytime/anywhere. A welcome page, which users visit before logging in, presents an overview of the program and prompts the carer to register. The registration page is fed with basic personal data about the carer and the care recipient (e.g., gender, kinship, nickname), which is used by the system to personalize the text within lessons (e.g., ‘Talk to your mother’). Each user can self-personalize the training plan to their needs by adding/deleting lessons into ‘My plan’. Five default lessons are recommended to all first users (Module 1: What is dementia; Module 2: The journey together; Module 3: Thinking differently; Module 4: An enjoyable day; Module 5: Introduction). The lessons comprise text information, illustrative images, caregiving scenarios and interactive skills training exercises. Immediate feedback is provided on carers' answers to exercises (Fig. 2, left). In ‘My printouts’ section, lessons can be printed/saved in PDF; the printouts are personalized with carers' answers to exercises. Users can self-assess their mood status using both a numerical scale and free text entry, and self-monitor their mood over time through a graphical or list display of their history (Fig. 2, right). iSupport features such as free access, personalization, interactivity, and self-monitoring are thought to favour the uptake of online health interventions by its end users (Hassan, 2020; Rogers et al., 2017; Svartvatten et al., 2015; Batterham and Calear, 2017).

Fig. 2.

Mock-up for the drag-and-drop exercises (left screen) and mood assessment tool (right screen) of iSupport European-Portuguese version.

3. Related work

iSupport was designed in a generic version which has undergone translation and/or adaptation in several countries (e.g., Australia (Xiau, 2020; Xiao et al., 2020); Brasil (Oliveira et al., 2020); India (Baruaha et al., 2021; Baruah et al., 2020); the Netherlands (Pinto-Bruno et al., 2019); Portugal (Teles et al., 2020a); Switzerland (Fiordelli and Albanese, 2020)). iSupport-Portugal is among the first culturally adapted versions of the program (Teles et al., 2020a). A five-step methodological approach was used to adapt iSupport to Portugal comprising: 1) needs assessment (Teles et al., 2021; Teles et al., 2020b); 2) content translation of the program by an authorized translator, and technical accuracy check of the translation by health and social support professionals; 3) cultural adaptation (semantic and conceptual equivalence of expressions, adaptation to cultural habits, traditions, local resources, and practices); 4) independent appraisal of contents by a panel of experts; and 5) fidelity check by WHO (Teles et al., 2020a).

The needs assessment studies conducted in Portugal, showed that dementia carers were experiencing high psychological needs (poor perceived mental health); appraised negatively the availability of psychosocial support services; and were facing numerous contextual barriers to access face-to-face interventions, such as incompatibilities of interventions schedules with full time jobs or full time caregiving responsibilities (Teles et al., 2021). In examining the potential of internet interventions in general, and of iSupport in particular, to bridge such gaps, the attitudes of digitally literate Portuguese carers towards online interventions were prospectively appraised and found to be positive (Teles et al., 2020c). Carers' training needs were assessed, and the most prominent needs categories matched the themes approached in iSupport (Teles et al., 2020b). In analysing carers' preferences for design features of online interventions, it was concluded that many of the most valued features were embedded in iSupport (e.g., personalization) (Teles et al., 2020b). In that study, the second most valued feature of an online intervention was its easiness of use (after plain language) (Teles et al., 2020b), highlighting the significance of systematically testing iSupport's usability. Currently, there are no published reports on the usability of iSupport,1 thus this paper offers a novel examination of the program.

4. Materials and methods

A mixed-methods study was conducted. Focus groups discussions and usability test sessions were performed. Focus groups are a useful strategy to understand people's beliefs, opinions and attitudes about a given topic, and especially the development of reasoning in the process of discussing with others (Britten et al., 1995). Usability tests allow observing the interaction of users with a system (International Organization for Standardization, 2010) and may include both qualitative and quantitative data collection techniques. Both carers and professionals participated in focus groups and usability tests: the first as prospective target-users of iSupport, and the second as prospective ‘prescribers’ of the program and knowledgeable persons on the needs of family carers. The study was approved by the Ethics Committee for Health of the São João University Hospital Center/Faculty of Medicine, University of Porto (reference 208/18).

4.1. Focus groups (FG)

4.1.1. Participants

Two focus groups were conducted: one with informal carers of persons with dementia and one with health and social support professionals experienced in working with family carers and/or people with dementia. Carers were recruited by convenience in the Northern Portuguese region from a Memory Cafe project. Carers received information on iSupport and were invited to return a form with their sociodemographic and context of care data, if willing to participate in the focus groups. Seventeen carers returned the form. Carers who met the following criteria were contacted: 1) Portuguese adults; 2) who were providing unpaid care; to 3) a person diagnosed with dementia; 4) who were frequent internet users (at least once a week); and 5) had an email address. Based on inclusion criteria, six carers were excluded: one professional carer; and five former carers. The remaining carers were divided into the ‘main interest group’ (n = 8)2 and surrogate participants (n = 3). Carers in the first group were selected based on their characteristics to include, as balanced as possible, both male and female participants; spouses and children of the person with dementia; and to be diverse with respect to education, duration of caregiving and time spent caring (weekly). The main interest group was contacted first, with seven carers accepting to participate; one surrogate carer was recruited later. Health and social support professional were recruited by convenience (n = 8) based on the following inclusion criteria: 1) Portuguese adults; 2) with a background in any of the following disciplines – psychology, gerontology, social work, education (social), medicine, nursing, occupational or physical therapy; and 3) who have ever provided health and/or psychosocial support to people with dementia and/or carers.

4.1.2. Data collection tools

A focus group guide was developed to gather participants' perspectives on eight domains of iSupport-Portugal: 1) overall usage experience; 2) look and feel appeal; 3) usability; 4) functional requirements; 5) lessons themes; 6) language clarity and sensitiveness; 7) scientific precision of contents and agreement with recommendations (professionals); and 8) motivations to use and/or recommend the program. A voting question about the name of the program was added to the discussion. A semi-structured format with open-ended questions was adopted (Table 1). Analogous guides were used with carers and professionals, except for topic 6 (for professionals only).

Table 1.

Semi-structured focus group guide.

Branch questions	Additional inquiries (if needed)
1) From your usage experience, what was your overall impression of iSupport-Portugal?	If any, which are the most positive aspects of your experience? If any, which are the most negative aspects of your experience?
2) What do you think about the visual appearance of iSupport-Portugal?	To what extent do you consider the program to be visually appealing? What do you think about the chosen colours and images? To what extent do you consider the text to be easily readable? What do you think about the visual organization of contents?
3) What is your opinion about the easiness of use of iSupport-Portugal?	To what extent do you consider that most carers would be able to use the program without help? Did you find any error or problem when using the program?
4) What is your opinion about the language used in the program?	To what extent do you consider the language to be plain and accessible to most carers? Did you find any negative, offensive or inadequate term or expression? What is your opinion about the adequacy of the texts to our cultural reality?
5) What are your thoughts about the themes approached in the program's lessons?	What do you think about the usefulness of the themes approached in iSupport-Portugal? What kind of information would you like to get from the program, which is currently missing? What kind of information, which is currently in the program, do you think to be unnecessary or inadequate?
6) To what extend do you consider the explanations and recommendations provided in iSupport to be scientifically precise and adequate? (Professionals)
	Did you find any incorrect or imprecise explanation in iSupport-Portugal? Did you find any recommendation in the program which is questionable/non-consensual? What kind of recommendation would you like to find in the program, which is currently missing?
7) Besides the lessons, the program allows to personalize your plan, produce printouts of each lesson and assess your mood. What do you think about such features?	Which advantages, if any, do you see in such functionalities? Which aspects, if any, would you change in each of those functionalities?
8) What would motivate you to use and keep using iSupport-Portugal? (Carers) What would motivate you to recommend iSupport-Portugal to (other) carers? (carers and professionals).	What would encourage you to start using the program? What would keep you visiting the program over time? What would discourage you to start using the program? What would make you stop using the program? Would you recommend the program to other carers? Which carers do you think would benefit the most from this program?
Voting question: name of the program	Would you prefer the name “iSupport” to be maintained in the Portuguese version or would you rather change it/translate it?

Participants' sociodemographic and context of care or professional practice data were collected. Attitudes towards technology were measured with the positive (6 items) and negative (3 items) attitudes subscales of the Media and Technology Usage and Attitudes Scale (Rosen et al., 2013). Each item is rated from 1 (strongly agree) to 5 (strongly disagree), with lower scores in the positive attitudes subscale and higher scores in the negative attitudes subscale representing more positive attitudes towards technology.

4.1.3. Procedures

Two weeks before the focus group session, participants were provided with the URL of iSupport-Portugal, granted with a unique access code, along with instructions to register into the program. Participants were encouraged to explore the program freely; however, to minimize burden, half of participants in each group were asked to focus more on modules 1 and 5 (11 lessons), while the other half on modules 2, 3 and 4 (12 lessons). A template with the ID/title of each webpage within the program was provided for participants to write down any comments (optional), and they were invited to deliver any written notes to the researchers. The focus groups were carried in January and February 2020. Both sessions were moderated by a researcher experienced in dementia caregiving and trained in qualitative research methods. An observer took written notes of the discussion. The durations of the focus groups were of 135 and 114 min with carers and professionals, respectively (with breaks). The consent form, a non-disclosure term of the identity/information from other participants and a data collection form (see Data collection tools) were filled by participants at the beginning of the sessions. Prior to the discussion, the moderator briefed the participants about iSupport, session aims, and ground rules. The moderator opened the discussion with the branch questions, and further enquiries were used if necessary (Table 1). Audio recordings were transcribed verbatim.

4.1.4. FG analysis

A thematic content analysis (Bardin, 2011) of focus groups contents was performed using the software NVivo, version 11. The analysis followed a horizontal scheme, and the categories/sub-categories were defined in a data-driven bottom-up approach. The contents were independently codified by two researchers (ST and AF). The categories/sub-categories were first derived by researcher A (ST) and next piloted by researcher B (AF), and both researchers coded the full content. The inter-rater agreement was examined, differences on coding were discussed, and disagreements were solved by consensus to reach the final coding tree. The Cohen's Kappa coefficient was computed to determine the initial inter-rater reliability and guide the refinement of the coding. Kappa values between 0.01 and 0.20 show none to a slight agreement; 0.21 to 0.40 indicate fair agreement, 0.41 to 0.60 moderate agreement, 0.61 to 0.80 substantial agreement and 0.81 to 1.00 almost perfect agreement (Landis and Koch, 1977). Results are presented by displaying absolute frequencies for the number of references coded in each (sub)category and the number of participants contributing to each (sub)category. Word frequencies are presented in a word cloud and participants' quotations illustrate contents within subcategories. For publishing, excerpts were translated from European-Portuguese into English.

4.2. Usability tests

4.2.1. Participants

Usability test sessions were conducted with a convenience sample of informal dementia carers (n = 10) as well as with health and social support professionals (n = 5). Insights on the usability of an interface can be gained with a small testing group of its prospective end-users: it is accepted that 15 users would suffice to detect the great majority of usability problems (Bastien, 2010). Inclusion criteria for both groups of participants were the same from focus groups. However, accounting for the interruption of Memory Cafe sessions due to COVID-19 pandemic, carers were recruited through an advertising campaign posted on social media groups of dementia carers. Carers volunteering to participate (19 volunteers) received the study information, consented to participate, and filled in a form with their sociodemographic and context of care data. Five volunteers were excluded as they were professional carers, and ten participants were selected among the remaining by gathering heterogeneous participants across demographics and ICT use variables.

4.2.2. Testing protocol

The usability of iSupport-Portugal was analysed by resorting to the collection of both objective and subjective measures. First, prior to the testing session, the participants filled a form with sociodemographic and context of care or professional practice information. Questions on use and perspectives towards ICT were also asked including: 1) frequency of internet use (daily; at least once a week; at least once a month; less than once a month); 2) devices used to access the internet; and 3) attitudes towards technology (positive and negative subscales of the Media and Technology Usage and Attitudes Scale (Rosen et al., 2013)). For the testing sessions, the following data collection strategies were used, by this order:

• 1)5 seconds test: participants were invited to enter iSupport's landing page and given 5 s to view the design, after which they were asked to a) use 3 adjectives/short sentences to describe their first impressions; and b) rate, on a 5-point scale from “totally disagree” to “totally agree”, their agreement with the sentences “The appearance of this entry page is trust inspiring”; “The appearance of this entry page makes me want to explore the program further” and “From the entry page I can tell that the program targets informal dementia carers”.
• 2)performance of six tasks (Nielsen, 1993) on the interface including a) registration; b) login; c) add a lesson to ‘My plan’; d) complete a lesson; e) print a lesson; and f) self-assess mood status. Participants were instructed to ‘think aloud’ while performing each task (Nielsen, 1993). The moderator registered whether tasks were successfully completed by using a 3-point classification system: 0 = not completed; 1 = completed with difficulty or acceptable prompts; and 2 = easily completed. Tasks were concluded when one of three conditions occurred: a) the participant completed the task; b) the participant declared to have completed the task, even if it was not completed; or c) the participant gave up.
• 3)participants' subjective appraisal of each task about a) easiness (on a 5-point scale from “very difficult” to “very easy”); and b) efficiency/quickness (on a 5-point scale from “very unsatisfied” to “very satisfied”).
• 4)rating of the program usability, resorting to the Portuguese version of the System Usability Scale (SUS) (Brooke, 1996; Martins et al., 2015) composed by 10 items answered in a 5-point Likert scale ranging from “strongly disagree” to “strongly agree”. The final score of SUS can range from 0 to 100, with higher values representing better usability and a cut off of 68 for defining a usability score above or below the average (Brooke, 1996).
• 5)post-test debrief, including two open questions on the most prominent strengths and weaknesses of the program from participants' interaction experience.

One third of usability tests were carried in person and the remaining were conducted by videoconference due to the COVID-19 pandemic outbreak. The testing protocol was the same in both situations. Participants' verbalizations, non-verbal reactions, and navigational choices (e.g., clicks, mouse movements) were registered in written notes, audio capture, and video screen capture. The audio recordings were transcribed verbatim. Each test session lasted 50 min on average.

4.2.3. Data analysis

Descriptive statistics (absolute and relative frequencies or central tendency and dispersion measures) are used to describe quantitative usability indicators, including task completion and scale ratings. A thematic content analysis was performed for text data, following the same procedures adopted for the focus groups data.

5. Results

For text data, results from focus groups and usability tests were reported in an integrated manner, and quality control measures for the coding process were described (see Section 5.5). Findings from quantitative data were reported separately (see 5.2, 5.3, 5.4).

5.1. Characterisation of study participants

5.1.1. Sociodemographic, context of care and professional practice data

Fifteen participants contributed to the focus groups discussions: 7 informal dementia carers and 8 professionals (Table 2). Carers were fairly distributed by gender (57.1% female), and age diversity was achieved (age range: 35–81 years; M 59.3). All carers were highly educated (education range: 12–22 years of schooling; M 16.4). Most people in care were female (71.4%), with a mean age of 73.1; both a case of early onset of dementia and an older old were represented (care recipients' age range: 57–81). Both spouses (57.1%) and children (42.9%) participated in the discussion. The caregiving duration varied from 3 to 11 years (M 6.1) and the number of hours spent caring per week from 48 to 168 (Mdn 50.0). Most participants shared care provision with either professional or informal carers. The participants were fairly distributed across (perceived) levels of care recipients' dependence as mild/moderate (42.9%) or total/severe (57.1%). Professionals participating in the focus groups were all female, with a mean age of 35.2 years and 20.6 years of formal education on average. With a background in Gerontology, Psychology or Education (social), participants accumulated 12.3 years on average of professional experience and 7.4 years on average of experience working with persons with dementia and/or their carers. More than half reported to have experience in designing, implementing and/or evaluating intervention programs for dementia carers, mostly of the psychoeducational type and carried in group. Three participants had experience with ICT-mediated interventions.

Table 2.

Sociodemographic, context of care, and professional practice variables of carers (IC) and professionals (P).

Variables	Focus groups (N = 15)		Usability tests (N = 15)
	IC (n = 7)	P (n = 8)	IC (n = 10)	P (n = 5)
Sociodemographics
Gender (female), n (%)	4 (57.1)	8 (100)	7 (70.0)	5 (100)
Age (years), M (SD)	59.3 (15.7)	35.2 (5.2)	50.8 (8.5)	38 (13.5)
Schooling (years), M (SD)	16.4 (3.1)	20.6 (1.7)	15.9 (5.0)	18.8 (3.8)
Context of care
CR gender (female), n (%)	5 (71.4)	–	8 (80.0)	–
CR age (years), M (SD)	73.1 (8.0)	–	77.7 (9.7)	–
Relationship with the CR, n (%)
Spouses	4 (57.1)	–	2 (20.0)	–
Children/grandchildren	3 (42.9)	–	8 (80.0)	–
CR dependence level, n (%)
Total/severe	4 (57.1)	–	3 (30.0)	–
Moderate/mild	3 (42.9)	–	7 (70.0)	–
Caregiving duration (years), M (SD)	6.1 (3.4)	–	4.1 (2.9)	–
Hours caring (per week), Mdn (IQR)	50 (120.0)	–	30 (108.5)	–
Shared caregiving (yes), n (%)	7 (100)	–	6 (60.0)	–
Professional practice
Background, n (%)
Gerontology	–	3 (37.5)	–	2 (40.0)
Psychology	–	3 (37.5)	–	1 (20.0)
Education (social)	–	2 (25.0)	–	1 (20.0)
Geriatrics (undergraduate)	–	–	–	1 (20.0)
Professional experience (years), M (SD)	–	12.3 (4.3)	–	11.2 (2.4)
Experience with IC and/or CR (years), M (SD)	–	7.4 (6.5)	–	5.7 (3.7)
Experience with intervention programs, n (%)
Design/planning	–	5 (62.5)	–	4 (80.0)
Implementation	–	6 (75.0)	–	4 (80.0)
Evaluation	–	6 (75.0)	–	3 (60.0)
Type of program, n (%)
Psychoeducation	–	6 (75.0)	–	4 (80.0)
Support groups	–	2 (25.0)	–	2 (40.0)
Individual	–	4 (50.0)	–	3 (60.0)
Group	–	6 (75.0)	–	4 (80.0)
Experience with ICT interventions (YES), n (%)	–	3 (37.5)	–	3 (60.0)

IC = informal carer; CR = care recipient; ICT = information and communications technology; M = mean; SD = standard deviation; n = number.

Fifteen participants were enrolled in usability tests: 10 carers and 5 professionals (Table 2). Carers were mostly female (70.0%), with a mean age of 50.8 years and an age range of 43 to 70 years. While most carers were highly educated (M 15.9 years of schooling), two less educated users were recruited (9 and 6 years of schooling). Most care recipients were female (80.0%), with a mean age of 77.7 years; cases of early onset of dementia and oldest old were represented (care recipients' age range: 53–95). Children/grandchildren were more represented (80.0%) than spouses (20.0%). The caregiving duration varied from 6 months to 10 years (M 4.1). The number of weekly hours spent caring per week varied from 11 to 168 (Mdn 30.0), and care responsibilities were mostly shared (60.0%). Most carers perceived the care recipients as mildly/moderately dependent (70.0%). Professionals enrolled in usability tests were all female, with a mean age of 38.0 years and 18.8 years of schooling on average. Professionals had 11.2 years of professional experience on average, 5.7 years with people with dementia and/or their carers, and had a background in Gerontology, Psychology, Education or Geriatrics (undergraduate). Most reported to have experience with designing, implementing and/or evaluating intervention programs with informal dementia carers, mostly of the psychoeducational type and delivered in group, and 60.0% had experience with ICT-based interventions. None of the carers participating in the focus groups or usability tests had prior experiences with online support programs.

5.1.2. ICT use and perspectives towards ICT

All participants in this study were self-reported daily internet users. Professionals, as well as carers participating in focus groups were savvy and autonomous internet users. Two carers participating in the usability tests (20.0%) reported to sometimes require support from others to consult webpages. Smartphones and laptops were the most used devices to access the internet either by professionals (100% for both devices; n = 13) and carers participating in both focus groups and usability tests (88.2%; n = 15; 82.4%, n = 14, for smartphone and laptop respectively).

Overall, participants showed positive attitudes towards technology: low scores at the positive attitudes subscale (6 items), in which lower scores represent more positive attitudes, were obtained, with a mean rate of 1.5 (SD 0.5) and of 1.7 (SD 0.4) for carers participating in focus groups and usability tests, respectively; and a mean rate of 2 (SD 0.9) and of 1.8 (SD 0.2) for professionals participating in focus groups and usability tests, respectively. Overall neutral scores were shown for the negative attitudes subscale, in which higher scores represent more positive attitudes, with mean scores of 3.3 (SD 1.3) and of 2.8 (SD 1.0) for carers participating in focus groups and usability tests, respectively. Professionals from focus groups and usability tests obtained an average rate of 2.9 (SD 0.6) and of 2.8 (0.8), respectively.

5.2. Should I stay, or should I go? First impressions of the interface

The trustworthiness of the entry page was overall well rated by participants (M 4.5; SD 0.5), while better appraised by carers (M 4.7; SD 0.5) than by professionals (M 4.2; SD 0.5). The featured logos of the organizations promoting the program and a plain layout are described as trust promoting design elements: “It looks completely reliable to me. I have no doubt!” (CUT33); “It is trust inspiring as it has this clean, plain look and the logos of these well-known organizations” (PUT5).

Participants' willingness to explore the program was also high (M 4.5; SD 0.6), whereas again higher for carers (M 4.6; SD 0.7) as compared to professionals (M 4.2; SD 0.5): “It is pleasurable to look at, it's not dark, conveys a positive message of joy, love and support, so I feel I want to see more” (CUT7). Some professionals (n = 4) and two carers found the program less sophisticated than expected/desired in terms of its design. However, the thematic contents approached in iSupport may contribute more than its appearance to participants' willingness to use it: “If I look at it and think: ‘Wow’! No, I do not. The program has a simple look. However, the carers are in need for help so the look will not discourage me to use it, my willingness to explore has to do with its contents” (CUT4).

Overall, participants found the program to communicate well about to whom it is targeted at, although this dimension was appraised less favourable than the previous ones (M 4.1; SD 1.2); and more negatively assessed by professionals (M 3.8; SD 1.3) than by carers (M 4.3; SD 1.2): “Just by the name of the program, the photos and the description I see right away that this is targeted at someone caring for a person with dementia” (CUT2). Five participants (2 carers; 3 professionals) found it hard to understand immediately to whom the program is directed at: “You can see that the page is related with dementia but not with caregiving” (PUT3).

Adjectives used by participants to describe their first impressions about the program (Fig. 3; translated to English) conveyed a positive experience with the program's:

• 1)Visuals - “appealing” (n = 9, 60.0%), “nice” (n = 5, 33.3%), “beautiful” and “unflashy” (each: n = 4, 26.7%). One adjective – “simple” - was used to convey either a positive (n = 5) or negative (n = 2; meaning basic/unsophisticated) impression of the program.
• 2)Emotional tone - “trustworthy” (n = 10, 66.7%), “warm” and “joyful” (each: n = 4, 26.7%);
• 3)Layout and navigation - “organized” and “easy” (each: n = 5, 33.3%), “friendly” and “clean” (each: n = 4, 26.7%); and
• 4)Language and contents - “clear” (n = 12, 80%); “useful” (n = 4, 26.7%); and “accessible” (n = 2, 13.3%).

Fig. 3.

Word cloud showing the 15 most frequent adjectives to describe the first impressions of the program. The analysis assumed stemmed words.

5.3. Task performance

Full or partial successes were achieved for all tasks (Table 3). Full success rates were superior to 80% for most tasks, except for the registration one (73.3%). Partial successes in the registration task resulted from attempts to register on the login area; failures in defining a criteria-compliant password; and defining the password in the access code field. Prompts were also given to more than one user to complete the tasks of: 1) adding a lesson to ‘My plan’; or 2) rating mood status. A single interaction problem emerged for each of those tasks: trying to add lessons to ‘my plan’ without accessing the lesson; and not understanding how to rate mood. Errors in registration, mood rating, and login were committed by less ICT-skilled users, while errors in other tasks were also committed by skilled users. All users were able to follow and complete a lesson. All tasks were rated favourably by participants concerning easiness of completion and efficiency. There were some inconsistencies between task completion and perceptions on easiness to complete a task, e.g., easiness was better rated for a task with 73.3% of full successes than for another with 100% of full success. Task easiness was rated 3 or above even when users were only able to complete the task with prompts.

Table 3.

Task completion rates, satisfaction rates and verbalizations while performing tasks. N = 15.

Task	Full success n (%)	Partial success n (%)	Type of errorsa	Easiness M (SD)	Efficiency M (SD)	User verbalizations
1. Registration	11 (73.3)	4 (26.7)	Filling the login area Filling both registration and login areas Unable to enter a criteria-compliant password Entering the password in the access code field	4.8 (0.6)	4.9 (0.4)	“Oh, yes, I was trying to fill both areas but the one bellow is to login” (CUT3) “These password rules are not very visible” (PUT5)
2. Login	14 (93.3)	1 (6.7)	Invalid password due to extra space entered	4.9 (0.3)	4.9 (0.3)	“It is a normal login, easy, but I would prefer to have it on the top of the page” (PUT3)
3. Add lesson to ‘My plan’	13 (86.7)	2 (13.3)	Try to add a lesson via the menu ‘my plan’	4.9 (0.4)	4.8 (0.4)	“Well, I'm in the ‘my plan’ page but I don't see here the lesson you ask me to add” (CUT10)
4. Complete a lesson	15 (100)	–	–	4.7 (0.8)	4.6 (0.8)	“It is straightforward, one just need to click ‘next’ to follow the session (…) but the button to go back should also have text or I'll use the browser arrow” (PUT2)
5. Print a lesson	14 (93.3)	1 (6.7)	Try to print directly from a lesson	4.7 (0.5)	4.7 (0.5)	“I'm clicking on the lesson to print…no…oh, I must click on the menu…one should be able to print directly on the lesson” (PUT2)
6. Mood self-assessment	13 (86.7)	2 (13.3)	Not able to rate the mood Not able to see the score chosen Not saving the score chosen	4.6 (0.6)	4.8 (0.4)	“This scale…should I choose any point? Why do we have a question mark here?” (CUT7) [about ‘slide’ to rate mood; question mark is replaced by a number]

^aAn error is here defined as any interaction with the program that does not lead to reaching the goal unless the user is prompted by the researcher, which results in a partial success.

5.4. Usability ratings

The average participant's ratings on the usability of iSupport-Portugal, after normalization of raw SUS scores into percentile rankings, was of 89.5 (SD 8.0), suggesting an excellent perception of the program's usability (Bangor et al., 2009). Carers revealed a slightly more positive perception of iSupport's usability (M 90.3; SD 8.7) than professionals (M 88.0; SD 7.2). The best appraised characteristic of the program was its uncomplicated use (M 4.9, SD 0.4, inverted score on item 8 of SUS “I found the system very cumbersome to use”). By disaggregating scores from carers and professionals, both rated the best the above-mentioned feature (M 4.8, SD 0.4 vs. M 5.0, SD 0, respectively), but carers rated equally well the complexity of the system (M 4.8, SD 0.4, inverted score on item 2 “I found the system unnecessarily complex”). However, for professionals, this item was the second worst rated (M 4.2, SD 0.8). The aspects receiving less favourable, yet positive, evaluations were the system consistency, i.e., on how the program operates, with similar actions leading to similar results (M 4.4, SD 0.9, inverted score on item 6 “I thought there was too much inconsistency in this system”) and time that most people would need to learn how to use the program (item 7; M 4.4, SD 0.7). Carers appraised worse the learning demands to use the system (items 7 and 10 inverted, M 4.4, SD 0.7 vs. M 4.4, SD 0.7, respectively); while professionals were less satisfied with system's consistency (M 4, SD 1.2, inverted score on item 6).

5.5. User feedback

Both transcribed verbalizations and written notes (n = 4, all professionals) from participants in focus groups and usability tests were analysed. The content analysis revealed similar codes for carers and professionals, thus contents from both groups are presented in an integrated manner.

5.5.1. Quality control of the coding

A coding tree was first proposed by researcher A (ST) and used by researcher B (AF) to analyse all information sources. A very good/substantial initial inter-rater agreement was found for the main categories (Kappa values from 0.66 to 0.92); thus, those were kept. The high agreement was most likely facilitated by the fact that the coding tree was not derived from scratch by the two researchers. For subcategories, the inter-rater agreement ranged from fair to substantial (Kappa values from 0.42 to 0.97), thus requiring a refinement for the final coding tree. Sub-themes under ‘Usability & Functional Requirements’ accounted for more disagreements. All disagreements were solved by consensus between the two researchers and consultancy with a third (PL) was requested, when needed.

5.5.2. Themes

The final coding tree reached after consensus rounds, resulted in a structure with five main categories and 26 subcategories (Table 4). For a detailed definition of each category see Appendix A.

Table 4.

Categories, subcategories and absolute frequencies of references coded in each (aggregated and disaggregated).

Category	Subcategory	References (N)
Style & Aesthetics		115
	Layout	53
	Affective Tone	21
	Use of Colours	16
	Use of Images	13
	Overall Appeal	12
Usability & Functional Requirements		296
	Easiness of Use	127
	Feedback	45
	System & Self-personalization	34
	Self-monitoring	28
	Peer & Professional Interaction	28
	Accessibility	15
	Technical Issues	14
	Efficiency	5
Program Content		178
	Lessons' Themes, Structure & Materials	80
	Language Clarity	45
	Language Sensitivity	29
	Cultural Adequacy	17
	Content Format	7
Usefulness & Endorsement (of iSupport)		84
	Usefulness	38
	Overall satisfaction	21
	Endorsement	16
	Willingness to use	9
User Profile		21
	Digital Skills & ICT use	10
	Caregiving Context	5
	Secondary Users	4
	Education	2

The coding process resulted in the categorisation of 694 text units. The ‘Usability & Functional Requirements’ category reunites the larger number of references coded (n = 296) and the greater number of participants contributing to a category (29 out of 30; Fig. 4). Results per category are next described.

Fig. 4.

Number of cases (participants) coded at each main category.

5.5.2.1. Theme 1 - Style & Aesthetics

Most excerpts within this category consist of participants' considerations about the layout of iSupport's web platform (n = 53). Most statements (n = 30)4 suggested that participants were overall satisfied with the organization of the visual elements of the website: “I would say that the platform is well organized, one can easily understand where things are” (CUT10). Improvements on the program's layout were suggested with respect to: 1) information hierarchy in the presentation of cases, exercises or modules/lessons (n = 10); 2) visibility of design elements in a given webpage (n = 9); 3) appeal of design elements (n = 2); and 4) intuitiveness of elements' location (n = 2).

Participants reported the experience of positive emotions triggered by design elements of the program (n = 21), including positivism (n = 13); affection (n = 5); trust (n = 2) and hope (n = 1): e.g., “it sends me a message of joy, affection and support” (CUT7). The colour scheme of the program was highlighted as attractive (n = 16). Participants' statements were however less favourable to choices on images and illustrations: while the use of real pictures was supported (n = 5), illustrations were mostly perceived as childish (n = 3) or meaningless (n = 5), especially by professionals. Generic statements on the overall appeal of the program were mostly favourable (n = 9) – “It seems to me that the platform is sufficiently appealing to make a person want to explore it further” (CUT7); however, two participants would like to find a more attractive interface. See Appendix A for more quotations on Theme 1.

5.5.2.2. Theme 2 - Usability & Functional Requirements

Overall, participants found the program efficient (n = 5); as well as “intuitive” (PU1) and “easy to use” (CUT10) (n = 127). Still, several usability issues were identified (n = 43), which include: 1) visual/labelling inconsistencies (e.g., a ‘next’ button using both an icon and label, while a ‘previous’ button uses only an icon); 2) not labelled/unfamiliar icons (e.g., the ‘star’ icon for sessions on ‘my plan’); 3) non-clickable elements looking like clickable (e.g., iSupport's logo); 4) lack of ‘exit’ options to leave unwanted states (e.g., no ‘Home’ button/icon, clickable logo or breadcrumb trail to reach the homepage); 5) visibility of irrelevant/competing information (e.g., showing the login area together with the registry page, misleading the user to fill the first); 6) provision of multiple/competing feedback (e.g., display of both smiles and numbers to rate mood); 7) lack of system prompts to use important features (e.g., no prompt to use mood assessment tools); and 8) lack of alternative action pathways covering different user preferences (e.g., printing option only in the menu, while some users would prefer it on lessons' page).

Participants' considerations pertaining to the accessibility of iSupport were favourable with respect to readability (font size and style; n = 10). However, one participant noticed that there is no option on the website to enlarge/custom text sizes. Three participants requested for text alternatives for non-text content to improve understanding of icons and buttons: “together with the ‘printer’ icon or while hoovering the icon, one should see the text ‘print’, so we know what it does” (CUT8); and two users would like to find text to speech options to improve content accessibility by users with disabilities.

With respect to functional requirements, all participants revealed to expect and value feedback provided by the system (n = 45). Both professionals and carers highlighted their satisfaction with the program's functionality of providing immediate feedback on skills training exercises (n = 26): “it is really good and helpful because we don't feel, you know, unsecure.” (CFG5). On the other hand, participants would like to find feedback on mood self-assessments (n = 19): “Maybe it could provide feedback according to certain landmarks and make an analysis like: ‘this week you have kept the same mood status’, and then guide the person ‘do this, visit this lesson, ask for help there’” (CUT8). The system's functionality of personalizing text contents with information inputted on the users' profile was positively appraised by participants (n = 9) – “It is important because it seems that it was made for me” (CFG3) -; and most participants identified advantages on self-personalizing the plan of lessons to their interests and needs (n = 22), or on deciding whether to use or not mood assessment tools (n = 3). With regards to program's self-monitoring features, in particular to mood appraisal and monitoring, participants' considerations were both favourable (n = 17) and unfavourable (n = 11). Negative views related to the belief that carers would have no time, no motivation, or no emotional awareness to self-monitor their mood, or that doing it in a self-managed program could produce iatrogenic effects. A preference or need for finding in iSupport a mean to interact with health/social support professionals (n = 12) and/or other carers (n = 10) emerged in participants' discourses: “I like the program except for the fact that we can't talk with other carers” (CFG2); “I would prefer to interact with a professional… in this program we make a self-analysis, but it would be good to have someone helping us” (CUT4). Five carers, however, have identified risks in interacting with professionals within iSupport program, which include 1) getting a generic/standardized type of support; and 2) receiving mixed information from professionals in iSupport and from other community services. Minor technical issues were reported (n = 14), mostly for the mobile version of the website. See Appendix A for more quotations on Theme 2.

5.5.2.3. Theme 3 – Program Content

In overall, iSupport lessons were described as comprehensive and the approached themes were considered necessary for carers (n = 80): “I think it covers 95% of issues [on dementia and caregiving] and the other 5% I don't even know if they are so relevant” (CFG1). However, the participants identified relevant issues which are not covered or are less detailed in iSupport, including: disease progression and genetics (n = 3); overeating (n = 8) and tube feeding (n = 3); anticipatory grief (n = 3); decision making on institutionalizing the care recipient (n = 7); community resources/services and financial support measures (n = 7); legislation and regulations affecting carers and care recipients (n = 4). Considerations on the format used to present lesson contents showed that participants expected and preferred multimedia contents (e.g., audio files, video clips) in addition to written contents (n = 7): “I personally like to read but I'm convinced that is easier or more appealing for most people to listen to the lessons. A person could be driving or cooking at the same time; to read one need to be focused” (CUT7).

All considerations made about the cultural adequacy of the program (n = 17) were favourable: “the program, its examples, are perfectly fit to our culture” (CFG5). There was no consensus about keeping the original name of the program or translating it. With respect to the comprehensibility of the language in iSupport most statements described it as clear, plain and free of jargon (n = 24): “The language is very accessible and very clear. I can tell you I have visited the program late at night, I was tired and yet it was not boring as it is free of technical terms” (CFG3). However, participants identified the need for minor adjustments (n = 21) to improve language clarity, including: 1) not using different terms with the same meaning; or 2) not using ambiguous terms. Improvements were also suggested with regards to language sensitivity, even though participants' appraisals of this feature were overall positive (n = 29). Suggestions consisted in: 1) softening terms or expressions in the text such as ‘incorrect’ [answer] or ‘learn’; and 2) improve gender-inclusiveness in a small set of sentences. See Appendix A for more quotations on Theme 3.

5.5.2.4. Theme 4 - Usefulness & Endorsement

All participants' statements on the overall satisfaction with iSupport were positive (n = 21): “I liked and, in my opinion, the other carers will be satisfied as well” (CFG1). Similarly, all remarks on iSupport's usefulness were favourable (n = 38). The program was described as addressing carers' needs; and diverse positive outcomes were anticipated or experienced, which included: 1) being more informed about the disease and caregiving; 2) increased self-reflexion; 3) being more able to relax; 4) feeling more confident about caregiving; and 5) increased well-being.

Both carers and professionals would recommend the program to other carers (n = 16) – “I would recommend it, and I think it should be integrated in the national health services” (CUT8). Two professionals would recommend the program only to digitally skilled and more educated carers. Statements from carers on their anticipated use of the program showed willingness to use it (n = 9) – “I'm willing to learn everything and explore it [the program] (…) I will use it several times and take notes” (CUT3), even though two carers anticipated time constraints for visiting iSupport's lessons. See Appendix A for more quotations on Theme 4.

5.5.2.5. Theme 5 - User Profile

With regards to the profile of potential users of iSupport, most considerations concerned individuals' digital skills and internet use habits (n = 10), with the shared perspective that iSupport is not suited for novice internet users (n = 8); or for carers who do not use the internet as an information source on caregiving (n = 2). Two participants stressed that iSupport may attract more educated carers: “I think it [iSupport] will attract a profile of highly educated carers” (PFG1). To overcome such issues, four participants suggested that access to iSupport could be intermediated by secondary users, either family members or health/social support professionals. Considerations about the caregiving context of iSupport prospective users (n = 5) included the perspective that the program is more suited for secondary (n = 1), employed carers (n = 1), caring for someone in the initial stage of the disease (n = 3): “the program is targeted at carers of people at the initial stage (…) for those caring for someone in more advanced stages this will be distracting” (PFG8). See Appendix A for more quotations on Theme 5.

6. Discussion

The purpose of this study was to gather feedback and assess the usability of iSupport-Portugal, an online training and support program for informal dementia carers. A mixed-methods design was followed. Overall, our findings showed that both carers and professionals appraised iSupport-Portugal positively with regards to usability and usefulness. A first highlight is the trustworthiness that all participants reported to feel about the program. Trust in online platforms is a complex matter as it depends on multiple factors, either pertaining to user characteristics and design/technology-related features (Teles et al., 2018; Bart et al., 2005). Factors promoting trust on health programs include a clear layout, interactive features, authority of the owner, ease of use and quality content; while age, gender and perceived health status are individual influential variables (Sbaffi and Rowley, 2017). Some of these - including the layout, authority of the owner and contents - were stressed by participants in this study as trust promoting. These results are encouraging accounting the sense of trust as an important determinant for using a Web service/page (Sbaffi and Rowley, 2017).

Despite overall positive user feedback, we have concluded for a need of implementing numerous improvements on the program concerning: 1) style and aesthetics (Theme 1 from the content analysis); 2) usability and functional requirements (Theme 2); and 3) program contents (Theme 3). Style and aesthetics aspects requiring further attention include information hierarchy; visibility and location of design elements; use of illustrations (not favoured by users); and use of design elements to better communicate about the program's purpose and target group. Such visual aspects are closely related with usability: using polished visual cues with expected placements to help keeping the navigation easy. Results from the usability testing of iSupport (n = 15) showed high success rates on task performance (>80.0% for most tasks) along with an excellent perception of the program usability (SUS = 89.5). Still, failures occurred for the critical task of registering on the program and were observed in less savvy internet users, meaning that there is room to improve the program's design ‘for all’. The European Institute for Design and Disability defines design for all as a “design for human diversity, social inclusion and equality” (EIDD, 2004). Applied to eHealth interventions, this approach is aimed at guaranteeing that everyone has equal opportunities to use them; however, it is a significant creative challenge for researchers, designers, and developers. This challenge must be embraced for iSupport soon, accounting for relevant issues found on the program accessibility as well as usage by less ICT-skilled participants (e.g., labelling, text magnifying or text to speech options). Including contents of different formats (video, audio) in a program almost exclusively text-based would be an important step not only to improve accessibility but also to match different learning styles and preferences as well as to make content more easily consumable. Usability issues identified in this study were diverse and to a certain extent contradict some of Nielsen's ‘heuristics’ – i.e., recognized interface design and usability principles - including visibility of the system's status; user control and freedom; as well as consistency and standards (Nielsen and Molich, 1990). Similar contradictions were reported in other studies on eHealth/mHealth solutions (Cristancho-Lacroix et al., 2014; Hassan, 2020; Brown and Nam Kim, 2018). A high success on tasks together with positive participants' appraisals about the program's ease of use, suggest that most usability issues are not severe, while must be addressed for a more satisfying user experience and efficient navigation. Some usability issues, however, may be challenging to solve in the short term and there are also open discussions with regards to functionally. Related to those, is whether and how mood self-assessment tools or mechanisms to interact with professionals and/or peers should be included in the program. Using more behaviour changing techniques and methods to communicate with participants in internet interventions seems to enhance its effectiveness (Webb et al., 2010). However, participants in this study were not in agreement with respect to including such dimensions in iSupport, with iatrogenic effects being anticipated (e.g., receiving generic/inadequate feedback from professionals; not being able to deal with insights on emotions). Concerning interaction with professionals in online interventions, participants in this study were perhaps more hesitant than what is usually reported in the literature (Klein and Cook, 2010). We have concluded in previous research that professional mediation in online interventions is less preferred by informal dementia carers caring for longer (Teles et al., 2020b). In this study, the average duration of caregiving was of 6.1 (focus groups) and 4.1 (usability tests) years. iSupport was designed as a self-managed program and some carers may prefer it like this as they may have more autonomy, feel uncomfortable in talking with others, or do not trust professionals they do not know; however, as participants were not consensual on the inclusion of a module to interact with professionals and/or peers in iSupport, one may desire to cover different preferences in planning a service logic for the program. This would imply integrating virtual and ‘human’ support, entailing a far more complex service infrastructure (e.g., 24/7 available professionals; funding). Accounting that iSupport aimed, in the first place, to avoid such complexity to minimize barriers in offering support, such trade-offs need to be carefully considered in the future. Contrary to a lack of consensus on the functionalities of mood self-monitoring or professional interaction, system feedback and personalization were functional requirements strongly supported by participants. This finding is consistent with other studies (Hassan, 2020; Batterham and Calear, 2017; Teles et al., 2020b). With regards to self-personalization, we observed that this feature seems relevant for accepting an online intervention, as it was seen by carers as an advantage in relation to face-to-face ones. Feedback and personalization features must then be reinforced in iSupport.

With respect to the prospective user profile of iSupport, participants expressed concerns about carers' digital literacy and, concerning sociodemographics, about carers' education rather than on age (a frequently discussed factor for adoption of ICT interventions). Education is indeed a strong determinant of internet use (Goldfarb and Prince, 2008), and accounting for low education levels of informal Portuguese carers (Barbosa and Matos, 2014) iSupport is more likely to be used by a segment of dementia carers. We anticipate, based on this and previous research, that this segment will include more educated, secondary, and professionally active carers (Teles et al., 2020b). Still, as internet usage trends evolve rapidly, one should think sustainably and universally in adapting iSupport. This means, for instance, keeping an accessible language even acknowledging the tradeoff that more educated carers might find it oversimplified. As suggested by participants, a way of reaching less educated and digitally illiterate users is to expand iSupport use by mediators such as health/social support professionals, a use case deserving future attention. An interesting study finding is that participants found iSupport to be more suitable for carers supporting persons in the initial stage of the disease. Many of the thematic contents considered to be missing in iSupport are more likely to be relevant at later disease stages (e.g., tube feeding). iSupport being targeted at those caring for someone at the initial disease stage is not stated as an aim or condition by the program authors (WHO), perhaps because many issues on dementia care are transversal to all disease stages and the negative effects of caring may perambulate in all. Accounting however for the problem of late access to psychosocial services by dementia carers (Stephan et al., 2018), it is not uncommon for those carers to seek for help only later as the disease progresses. As such, adjusting iSupport to the needs of those carers must be discussed soon.

Findings from this study must be seen in the context of study limitations. While a diverse group of participants was included with respect to sex, age, and caregiving context, only two carers had nine or less years of schooling. Education is a well-known determinant of internet usage (Goldfarb and Prince, 2008), thus the fact that participants are more educated relates the most with the nature of the intervention than with methodological issues. Still, as shown in usability tests, although not straightforward, less educated users may face more challenges in using the program. In a design-for-all logics, challenges faced by these users should be a point of reference to appraise the website usability. Secondly, carers in this study volunteered to participate, which – despite further selection according to inclusion criteria and diversity – may have resulted in a self-selection of users presenting positive attitudes towards technology. Participants may fit a profile of early adopters, who are more prone to use innovations than their counterparts. Another study limitation concerns the inclusion of primary (carers) and secondary (professionals) end-users only. Further research must ensure that the complete eHealth stakeholder ecosystem is represented by including also tertiary end-users, i.e., organizations that may have a role in paying or enabling eHealth interventions. We were also not able to offer insights on particular kinds of non-functional requirements for iSupport – including safety, security or reliability (Nuseibeh and Easterbrook, 2003) – as well as on specific usability attributes, such as memorability (re-establishment of proficiency after a period not using the program). Finally, while findings from this study with respect to participants' willingness to use the program and anticipated benefits were encouraging, further research endeavours are needed. Future research includes collecting usage data (e.g., number of visits), and conducting an effectiveness study (see (Teles et al., 2020d) for the RCT protocol).

7. Conclusions

The deployment of accessible, acceptable, and effective training and support interventions for informal dementia carers is a strategic priority on dementia. iSupport might be a valuable alternative or adjunct care approach for dementia carers, even more in the current pandemic scenario. In the context of eHealth interventions, a usable interface may reduce barriers to use, enhance user experience and engagement, and minimize dropout. Results from this study were encouraging in suggesting that iSupport is a feasible means of providing accessible information and training for digitally literate informal dementia carers. The fact that several usability issues were uncovered, showed that usability evaluations of eHealth interventions are highly recommended as those support researchers and developers in determining its adequacy and usefulness and may be cost saving. While this study was designed to improve the European-Portuguese version of iSupport, the methods followed to assess usability and gather user requirements can be reproduced as part of a user-centered design in any eHealth intervention. This paper populates the scarce literature on the usability of eHealth interventions, and lessons learnt may offer relevant information to other country specific versions of iSupport under development and to other same-purpose programs.

The following are the supplementary data related to this article.

Appendix A

Framework for the content analysis of the data: coding tree, definition of categories and subcategories and examples of quotations per subcategory.

Funding

This work was supported by national funds through the Portuguese Foundation for Science and Technology, I.P. (FCT), in the scope of the project UIDB/04255/2020 [Este trabalho é financiado por fundos nacionais através da FCT Fundação para a Ciência e a Tecnologia, I.P., no âmbito do projeto «UIDB/04255/2020»]. Two authors are individually funded by the Portuguese Foundation for Science and Technology, I.P. (FCT) [Grant number SFRH/PD/BD/135496/2018 (Apoio financeiro da Fundação para a Ciência e Tecnologia no âmbito do Programa Operacional Regional do Norte, comparticipado pelo Fundo Social Europeu e por fundos nacionais do Ministério da Ciência, Tecnologia e Ensino Superior, através da atribuição de uma bolsa de doutoramento individual); Grant number IF/00693/2015].

Ethical considerations

This study was approved by the Ethics Committee for Health of the São João University Hospital Center/Faculty of Medicine of the University of Porto (reference 208/18).

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.