Abstract
Background
Thorough holistic development of eHealth can contribute to a good fit among the technology, its users, and the context. However, despite the availability of frameworks, not much is known about specific research activities for different aims, phases, and settings. This results in researchers having to reinvent the wheel. Consequently, there is a need to synthesize existing knowledge on research activities for participatory eHealth development processes.
Objective
The 3 main goals of this review are to create an overview of the development strategies used in studies based on the CeHRes (Center for eHealth Research) Roadmap, create an overview of the goals for which these methods can be used, and provide insight into the lessons learned about these methods.
Methods
We included eHealth development studies that were based on the phases and/or principles of the CeHRes Roadmap. This framework was selected because of its focus on participatory, iterative eHealth design in context and to limit the scope of this review. Data were extracted about the type of strategy used, rationale for using the strategy, research questions, and reported information on lessons learned. The most frequently mentioned lessons learned were summarized using a narrative, inductive approach.
Results
In the included 160 papers, a distinction was made between overarching development methods (n=10) and products (n=7). Methods are used to gather new data, whereas products can be used to synthesize previously collected data and support the collection of new data. The identified methods were focus groups, interviews, questionnaires, usability tests, literature studies, desk research, log data analyses, card sorting, Delphi studies, and experience sampling. The identified products were prototypes, requirements, stakeholder maps, values, behavior change strategies, personas, and business models. Examples of how these methods and products were applied in the development process and information about lessons learned were provided.
Conclusions
This study shows that there is a plethora of methods and products that can be used at different points in the development process and in different settings. To do justice to the complexity of eHealth development, it seems that multiple strategies should be combined. In addition, we found no evidence for an optimal single step-by-step approach to develop eHealth. Rather, researchers need to select the most suitable research methods for their research objectives, the context in which data are collected, and the characteristics of the participants. This study serves as a first step toward creating a toolkit to support researchers in applying the CeHRes Roadmap to practice. In this way, they can shape the most suitable and efficient eHealth development process.
Keywords: eHealth, community-based participatory research, human-centered design, CeHRes Roadmap, internet-based intervention, technological innovations
Introduction
Background
Over the past years, many different types of eHealth technologies have been developed, implemented, and studied in practice. These eHealth technologies, such as web-based interventions or mobile apps, are used to support health, well-being, and health care using technology [1]. Although the e in eHealth illustrates the importance of technology, eHealth encompasses much more than merely adding information and communication technology (ICT). It characterizes a novel way of thinking and working, and it changes the way health care is organized [2,3]. eHealth can offer many benefits such as increased access to care, increased efficiency and quality of care, and more ownership and self-management among patients [1]. However, in practice, many of these potential benefits are not achieved. A reason for this is low uptake; many eHealth technologies are not used as often as would be expected [4]. This can be partly explained by a suboptimal fit among the characteristics of a technology, the needs and skills of the users, and the context in which the technology is used [5]. If the content of an eHealth technology does not fit with the structures of an organization and the characteristics of end users, chances of its being successfully used are low [6]. To illustrate, if a web-based intervention requires a lot of reading, it will probably not fit well within an organization that mostly treats patients with low literacy skills. This interrelationship highlights the holistic nature of eHealth, in which technology, people, and context are intertwined. For that purpose, a user-centered, iterative, and multi-method development process in which all stakeholders are actively involved is recommended [7-10]. By means of a thorough development process in which multiple research activities are combined, eHealth that provides added value for its users and context can be realized [11].
Models for eHealth Development
There are multiple frameworks and models that can be used to guide human-centered, iterative development processes of eHealth. Well-known examples are the CeHRes (Center for eHealth Research) Roadmap [2,11]; the person-based approach [7]; the Accelerated Creation-to-Sustainment model [12], Intervention Mapping [9], the Persuasive System Design model [13], and the agile science approach [14]. Although these abstract models offer valuable guidelines and principles, they are not, and should not be, viewed as step-by-step prescriptions of eHealth development [14,15]. Rather, they should be viewed as a framework that researchers and developers use to shape their own development process and select the most appropriate research activities. However, not much is known about which research activities are most suitable for eHealth development within specific types of contexts and participants [16]. Consequently, there might be an availability bias in eHealth development: researchers might mostly use research activities that they are experienced with or those that are often described in literature [17]. However, other less-known research activities might have been a better fit with their research questions and context. To increase knowledge on how to apply development models in practice, existing eHealth frameworks could be supplemented with practical toolkits. Such toolkits could support the operationalization of the more abstract frameworks into specific research activities [15]. They can be based on experiences and lessons learned from earlier research. In this way, they could provide an overview of the kinds of development activities that can be used in the different phases of a specific eHealth development framework and offer guidelines on when and how to use these activities.
Objective
In this study, we aim to create the foundation for a toolkit for a specific eHealth development framework. We provide an overview of research activities for the development of eHealth technology in context. Including studies on all eHealth development projects conducted would be a very time consuming and nearly impossible task. Therefore, this paper focuses only on studies that are based on the CeHRes Roadmap. The CeHRes Roadmap is a much-used structured framework for the development of eHealth technologies [5,16]. The results of this review can support researchers to apply the CeHRes Roadmap in practice by supporting them to select not the most obvious but the most suitable research activities. The CeHRes Roadmap (Figure 1) is based on five principles: (1) eHealth development is a participatory development process; (2) eHealth development creates new infrastructures for improving health care, health, and well-being; (3) eHealth development is intertwined with implementation; (4) eHealth development is coupled with persuasive design; and (5) eHealth development requires continuous evaluation cycles [2,11]. The CeHRes Roadmap consists of 5 intertwined phases (the contextual inquiry, value specification, design, operationalization, and summative evaluation phase) that are connected by formative evaluation cycles [11]. The first 3 phases are focused on the development of eHealth. As the CeHRes Roadmap is very comprehensive, many of these principles are also important in other (aforementioned) eHealth development models. Consequently, although this review does not cover all eHealth development models, the identified research activities and lessons learned would also be suitable for application to other eHealth development models. To provide an overview of research activities used in studies guided by the CeHRes Roadmap, this narrative scoping review focuses on the following research questions:
Figure 1.
Which research activities have been used in the development process of eHealth technologies that were based on the principles of the CeHRes Roadmap?
With which goal(s) and in which phase(s) have these research activities been used in the development process of eHealth technologies?
What are the experiences with, and lessons learned from, the use of the research activities?
Methods
Inclusion and Exclusion Criteria
As the main goal of this study is to provide a focused overview of development activities used in the context of the CeHRes Roadmap and because no quality assessment of the to-be-included studies is required to reach this goal, a narrative scoping review was performed [19]. Records were included if they presented (part of) a development process of an eHealth technology that was based on the principles and/or phases of the CeHRes Roadmap. This refers to the use of ≥1 of the first 3 phases of the CeHRes Roadmap or explicit application of ≥1 of its principles. Consequently, studies were included if they referred to the original 2011 paper in which the CeHRes Roadmap was first introduced in either the introduction or methods section. This had to have been done in such a way that it became clear that ≥1 of the CeHRes Roadmap’s phases or principles was used to inspire the design of the study. Studies that only referred to the CeHRes Roadmap in their discussion section and/or did not contain activities for eHealth development were thus excluded. Studies that only focused on implementation and summative evaluation were excluded. Records were also excluded if they did not present any data but merely discussed abstract guidelines or models for eHealth development. Furthermore, records not written in English, Dutch, German, or Portuguese were excluded. Finally, because of the broad scope and exploratory focus of this study, only study designs from peer-reviewed journals or books were included. Student reports, preprints, and poster abstracts were excluded because they were not peer reviewed.
Literature Search
To provide a complete overview of the studies that explicitly used or were inspired by principles of the CeHRes Roadmap, a straightforward search strategy was applied. Studies that referred to either the 2011 paper in which the CeHRes Roadmap and its principles were introduced or the new book chapter about the CeHRes Roadmap were identified in Scopus, Google Scholar, and Web of Science [2,11]. To include studies that were based on the principles of the CeHRes Roadmap but had been written before the 2011 paper, a snowball sampling strategy in which records that were coauthored by the founder of the CeHRes Roadmap (JEWC van Gemert-Pijnen) were searched in the same 3 databases. All searches were performed up until June 2021.
After removing duplicates in Covidence (Veritas Health Innovation Ltd), 3 researchers (HK, JK, and MCDS) screened the titles and abstracts using the aforementioned inclusion and exclusion criteria. As it might be possible that the development process was not fully explained in the title or abstract, the criteria were applied broadly to prevent the unjust exclusion of relevant articles. In case of doubt, a record was included to prevent overlooking relevant publications. Next, records were included for full-text screening if at least one of the authors decided to include an abstract. Full texts were assessed by 1 researcher (HK, MCDS, or JK) and, in case of doubt, discussed with one of the other researchers.
Data Extraction and Analysis
The data extraction process was performed by 3 researchers (HK, MCDS, and JK) and based on a table developed in an earlier study, which was used to present, and reflect on, eHealth development strategies [15]. All relevant information from the included records was copied into the data extraction table. The narrative data extraction form was divided into 3 main categories with accompanying subcategories and is presented in Multimedia Appendix 1. First, information on the overall goal and type of study design of the entire paper was included. Second, information was extracted for each development activity that was reported in the record. As in participatory or human-centered eHealth development processes, nonparticipatory activities such as literature reviews can also be valuable, no distinction was made between activities in which users were and were not actively involved [11]. In other words, nonparticipatory activities might also be valuable or even necessary for participatory development processes. For each activity, the following information was reported in the form: research goal, target group and participants, description of research activity used, rationale for research activity, main results that were obtained by means of the activity, and phase of the CeHRes Roadmap that the research activity was used in. If the phase of the CeHRes Roadmap was not explicitly mentioned in the record, this information was deduced by the authors using the goals and methods as reported in a recent publication on the CeHRes Roadmap [11]. Third, all lessons learned about the application of the method that were reported in the records were copied into the data extraction form.
To analyze the data and answer the research questions, multiple steps were taken. To answer the first research question, an overview of research activities used in all studies was created. As activities were often named in slightly different ways, researchers formulated overarching categories for development activities by means of discussions until consensus was reached. In addition, a definition for each research activity was formulated. This definition was created by means of the information provided by the authors of the included records. If necessary, the definition was subsequently fine-tuned. This was done using other relevant literature—mainly a book that was edited by the research group of this paper’s authors [5]—and discussion among the authors of this paper. To answer the second research question, all information on the goal of a research activity, its main results, and the phase of the CeHRes Roadmap in which it was used was combined into 1 document. Researchers used this information to summarize the ways in which an activity was used and identify examples to illustrate the goals that can be achieved with the research activity. Again, if necessary, discussion among the authors took place until consensus was reached. Third, to answer the final research question on lessons learned, all extracted fragments with information about lessons learned were provided per activity; 2 researchers (HK and JK) went through these fragments separately and individually summarized the most important lessons learned. After discussion, an overview of all lessons learned was created. On the basis of this overview, the 3 most prevalent and applicable lessons learned were selected by the researchers and presented in a narrative way.
Results
Search Results
As can be seen in Figure 2, the initial literature search yielded 1713 unique records. After title and abstract screening by 3 researchers (HK, JK, and MCDS) using the aforementioned inclusion and exclusion criteria, of the 1713 records, 377 (22.01%) remained. After full-text screening of these 377 records, 160 (42.4%) were included. The main reason for excluding full texts was a lack of specific focus on the development of an eHealth technology.
Figure 2.
Search strategy and results. CeHRes: Center for eHealth Research.
Study Designs and Technologies
An overview was generated of the designs of all included studies (n=160). Most development studies used a multi-method approach, in which various qualitative and quantitative activities were combined (90/160, 56.3%). Other designs that were identified were qualitative cross-sectional (21/160, 13.1%), quantitative cross-sectional (18/160, 11.3%), literature studies (17/160, 10.6%), qualitative longitudinal (12/160, 7.5%), and quantitative longitudinal (2/160, 1.3%).
The included studies focused on a broad range of eHealth technologies. The main goal and described methods per study are provided in Multimedia Appendix 2 [8,15,20-176]. Most of the studies focused on web-based interventions (74/160, 46.3%) such as a web-based module for treatment of depression or a decision support tool for nurses. Mobile apps were the focus of development in 21.3% (34/160) of the studies. Apps were used, among other things, to support patients with breast cancer in doing arm and shoulder exercises and to support citizens in dealing with tick bites. In addition, virtual reality (VR) was studied in 1.9% (3/160) of the papers, which mostly focused on role-playing in treatment of patients with psychiatric disorders. Furthermore, in 16.9% (27/160) of the studies, there was no clear description of the technology, either because it was too early in the development process or because the authors did not include a description.
Overview of Methods and Products Per Phase of the CeHRes Roadmap
During the data extraction process, a distinction had to be made between research methods and products. Methods such as interviews were used to collect new data, and products such as prototypes were based on, or summarize, previously collected data and can be used as tools to collect new data. In Table 1, an overview of all methods and products that were identified is provided, including references to the accompanying studies (Multimedia Appendix 2). In addition, for each method or product, the number of studies that used it in the contextual inquiry, value specification, or design phase is provided. As in some studies multiple methods or products were used in different phases or because methods that were relevant for >1 phase were used, the sum of the columns is not the same as the number of studies. In 30% (48/160) of the studies, the phases of the CeHRes Roadmap were mentioned explicitly; in the other studies, the CeHRes Roadmap was mostly used to inspire the design. Two authors (HK and JK) categorized the methods and products in these papers based on the definitions of the phases of the CeHRes Roadmap [11].
Table 1.
An overview of the identified methods and products, the accompanying references, and the phases of the CeHRes (Center for eHealth Research) Roadmap in which they were categorized in the included studies (N=160).
| Method or producta | CeHRes Roadmap phases | References | ||||||||
|
|
Contextual inquiry, n (%) | Value specification, n (%) | Design, n (%) |
|
||||||
| Methods | ||||||||||
|
|
Focus group (n=71) | 45 (63) | 62 (87) | 29 (41) | [8,15,20-95,177] | |||||
|
|
Interview (n=70) | 41 (59) | 32 (46) | 21 (30) | [15,21,24,26,28-31,33,34,40,44,45,47,49,50,52,54-56, 58,61,67,69,70,74,76,78,79,81,83,84,88,89,91,96-129,178] | |||||
|
|
Questionnaire (n=51) | 28 (55) | 16 (31) | 19 (37) | [15,24,26,29,32-34,46,49,56,60,67,69,71,76,80,84,89,96-98,101,104,107, 109,115,125-127,129-151] | |||||
|
|
Usability test (n=51) | 0 (0) | 1 (2) | 67 (131) | [8,20,21,24,25,37-39,41-43,45,48-50,54,55,63,64,66,67,71,72, 75,77,80,81,84,92,98,102,104,106,107,109,110,112,113,117,118,127,131,152-156] | |||||
|
|
Literature study (n=43) | 36 (84) | 2 (5) | 4 (9) | [8,20,21,27,28,38,43,46,53,55,58,69,72,75,80,81,84,92,93,99,104, 112-114,129,153,157-173] | |||||
|
|
Desk research (n=15) | 14 (93) | 3 (20) | 0 (0) | [15,31,33,34,43,51,52,56,59,61,88,91,98,112,135] | |||||
|
|
Log data analysis (n=10) | 1 (10) | 0 (0) | 8 (80) | [68,86,104,108,111,133,140,174-176] | |||||
|
|
Card sorting (n=9) | 1 (11) | 1 (11) | 8 (89) | [8,25,34,36,70,77,98,102,125] | |||||
|
|
Delphi study (n=3) | 1 (33) | 2 (67) | 0 (0) | [72,84,101] | |||||
|
|
Experience sampling (n=1) | 1 (100) | 0 (0) | 0 (0) | [130] | |||||
| Products | ||||||||||
|
|
Prototype (n=32) | 0 (0) | 8 (25) | 29 (91) | [15,25,31,36,40,46,48,51,52,55,63,66,69,72,74,75,77,78,81,83-85, 88,92,94,96,102,107,110,113,115,121,124,127,129] | |||||
|
|
Requirements (n=11) | 0 (0) | 10 (91) | 0 (0) | [20,30,31,38,39,70,91,92,114,115,129] | |||||
|
|
Stakeholder map (n=10) | 4 (40) | 4 (40) | 0 (0) | [15,27,30,35,46,56,59,129,135,153] | |||||
|
|
Values (n=7) | 0 (0) | 7 (100) | 0 (0) | [15,79,81,97,109,129,152] | |||||
|
|
Behavior change strategies (n=5) | 0 (0) | 1 (20) | 4 (80) | [69,79,84,91,109] | |||||
|
|
Personas (n=5) | —b | 1 (20) | — | [67,70,73,74,81] | |||||
|
|
Business model (n=4) | 0 (0) | 4 (100) | 0 (0) | [27,62,84,99] | |||||
aThe sum of the times a method was used in the contextual inquiry, value specification, or design phase is higher than the number of included studies per method because in multiple studies, one method was used more than once in the development process.
bNo relevant records were identified for the category.
Definition, Applicability, and Lessons Learned Per Method
Overview
In the following sections, the definition that was generated by means of the included studies is provided for each method. In addition, different examples of how the method was used are given. Finally, the 3 most relevant lessons learned that were mentioned in the included papers are summarized.
Focus Group
Definition
Focus groups refer to meetings where qualitative data are collected by involving a relatively small number of stakeholders in a group discussion. This discussion is focused on a particular topic or set of issues, ranging from relatively unstructured workshops and generative design sessions to highly structured meetings.
Examples of Applications
Of the 160 included studies, 71 (44.4%) were focus group studies, most of which had some sort of predetermined structure. The extent to which data were systematically analyzed differed among the studies. In some, extensive coding schemes were created, whereas in others, the most important findings were summarized. Furthermore, some focus groups included a single type of stakeholder, for example, only patients, whereas others included a combination of multiple stakeholders such as therapists, patients, and technology developers. In addition, some studies included novel participants in each focus group, whereas others used recurring coresearchers [90]. As can be seen in Table 1, focus groups were used throughout all phases of the development process. This shows that focus groups can be used to reach a broad range of goals. Examples of these goals include the following: to identify points of improvements of the current situation, such as self-management of patients with chronic obstructive pulmonary disease (COPD) [32] or care for cerebral palsy [33]; to discuss the possibilities of a specific technology, such as the values of people with obesity regarding a to-be-developed behavior change intervention [79]; to gain insight into cognitions such as attitudes toward measures to reduce antimicrobial resistance [8]; to identify or validate values or requirements with potential end users such as health care providers [23,93]; or to collect input for the improvement of a prototype, for example, for a portal for infection control [27].
Lessons Learned
First, authors of multiple studies indicated that focus groups are a good way to gain more insight into the specific needs, wishes, and opinions of individuals regarding eHealth. To achieve this, focus groups can be conducted with a group of similar or very different stakeholders. Participants can bounce ideas off of each other and can directly respond to each other and can provide insight into the prevailing consensus or the range of different opinions or perspectives regarding eHealth [35-37,43,54,57,58]. However, researchers should take potential power imbalances or potentially sensitive conflicting values into account when inviting participants of a focus group. A second lesson refers to the iterative nature of eHealth development. Focus groups can be used in a sequential way: multiple focus groups can be conducted in a row and the goal and content of each focus group can be based on the outcomes of the previous focus groups. However, this iterative approach was said to be quite time consuming for researchers and participants [33,54,64,82,84-86,90,92]. Third, to ensure that valuable information for eHealth development is gathered, the content and form of focus groups need to be adapted, based on the topic and target group [39-41,51,55,61,87]. To illustrate, in-person focus groups are not suitable for every topic. Web-based alternatives might be considered when, for example, sensitive topics such as sexual health are discussed. Furthermore, different types of participants might require different types of focus groups. For example, focus groups with people with an intellectual disability or with older adults require a setup with more concrete examples of eHealth and might benefit from icebreakers and room for informal conversations [87]. In contrast, focus groups with therapists or researchers can cover more abstract topics [15].
Interview
Definition
In interviews, individuals are asked questions in a structured, semistructured, or unstructured way to obtain answers from a broad range of possible stakeholders, guided by an interview scheme.
Examples of Applications
Of the 160 included studies, 70 (43.8%) featured interviews that took place at multiple points in the development process. Interviews can be conducted from the start of a development process to not only analyze a problem, but also evaluate prototypes. Consequently, interviews can have a broad range of goals. Examples include identifying points of improvement for a current situation such as treatment of forensic psychiatric patients [96]; analyzing target or risk groups in, for example, tick bites [100]; identifying points of improvement for prototypes or existing (eHealth) interventions according to end users or design experts [83,84]; identifying potential barriers and facilitators for implementation later in the process, such as high costs and required skills training [78,96]; describing a current behavior and its determinants [91]; collecting experiences of participants after letting them try out an app in real life [83]; and generating or validating values and requirements [15,28,98,103].
Lessons Learned
First of all, in multiple papers, authors mentioned the importance of individual, in-depth interviews to incorporate the perspective of vulnerable, complex target groups such as people with dementia or severe mental illness in eHealth development [15,21,61,70,96,97,109,111,121,123,127]. This is especially important because perspectives of these underserved target groups are often overlooked in eHealth development. However, despite the benefits, including these types of target groups was found to be challenging, mostly because participating in relatively long, in-depth interviews requires a fairly high level of cognitive abilities such as attention and memory. Consequently, researchers should account for the characteristics of their target groups by, for instance, keeping the interviews as short as possible [78,128]. Another option is the use of concrete examples of eHealth technologies to account for response or recollection biases [24,52,97,121,123]. Second, although interviews can yield valuable results, a limitation is that they can offer a 1-sided picture of stakeholders’ needs and wishes regarding eHealth. Selection bias can result in a sample that is overly positive or negative [78,128]. To overcome these issues, which are related to generalizability, multiple authors recommended that interviews be combined with other methods in a multi-method or mixed methods approach [24,40,52,54,69,78,96,97,100,102,108,109,116,118,120,124,126]. This could be done by combining interviews with a small sample size with a questionnaire with a larger sample size. Although small sample sizes were not necessarily considered problematic in eHealth development, combining methods in an iterative way was suggested as a way to overcome issues with generalizability [81,127]. Third, interviews were used quite often and were generally viewed as a useful method that can be used at any point in the eHealth development process [15,21,45,54,96-98,100,105,106,109].
Questionnaire
Definition
A questionnaire can be either quantitative or qualitative; it consists of a series of open- or closed-ended questions for the purpose of gathering information from—often—a relatively large sample of respondents and can be distributed on the web or on paper.
Examples of Applications
Of the 160 included studies, 51 (31.9%) featured questionnaires that were applied in all phases of the development process, which means that they can be used for a broad range of goals. Examples include gathering information for stakeholder identification and analyses [135]; identifying points of improvements in conceptions and knowledge of stakeholders on infection outbreaks and antimicrobial resistance [134,178]; mapping attitudes toward technologies such as embodied conversational agents [132]; identifying needs and wishes (values) regarding a to-be-developed technology [97]; or evaluating low-fidelity prototypes, for example, scenarios on multiple possible VR interventions [15]. In questionnaires, either new questions can be generated by researchers, based on previous research, or existing questionnaires can be used, for example, the Personal Involvement Inventory, the eHealth Literacy Scale, or System Usability Scale [15,24,84,127].
Lessons Learned
First, in many studies, authors reflected on possible biases that might arise when using questionnaires to develop eHealth [24,69,80,97,109,115,126,134,138,139,142,146,149]. Among other things, vulnerable target groups with low literacy skills and no internet access were often hard to include in questionnaires [96]. Furthermore, some researchers used students or services such as Amazon Mechanical Turk to generate large samples; however, this raises questions about the generalizability of the results [136]. This means that results of questionnaires have to be interpreted with care and should not serve as the sole input for an eHealth technology. Second, multiple authors indicated that a questionnaire, especially one on the web, is a suitable method to quickly and efficiently collect data from different types of stakeholders and to check for differences among groups in, for example, opinions or knowledge [60,67,69,98,126,131,147,150]. A pitfall of this approach is that collecting rich in-depth information about, for example, an existing problem or a prototype is challenging. The main reason is that participants often do not provide elaborate answers to open-ended questions; in addition, it is not possible to ask probing questions [15,139,141,146]. The third lesson learned is that questionnaires need to be combined with other types of data such as interviews or focus groups to meaningfully contribute to the development process [24,26,33,60,71,96,97,126,150]. As eHealth development requires a complete picture of the current situation and needs and wishes of the stakeholders, triangulation of methods should be used. For example, products generated earlier, such as values, can be cross-referenced or interviews can be used to provide more context to the outcomes of a questionnaire.
Usability Testing Methods
Definition
Usability testing is an umbrella term that can refer to a broad range of methods such as a think-aloud method with scenarios, cognitive walkthrough, heuristic evaluation, or eye tracking. These methods are used to conduct formative evaluations of prototypes by testing them with participants such as potential users or experts. Usability refers to the extent to which a user can use a product effectively and without effort, immediately learning its use. Usability tests can be used to identify usability problems, flaws, and points of improvement or gather overall opinions.
Examples of Applications
Of the 160 included studies, 51 (31.9%) reported on usability tests. As these tests require a prototype that should be based on earlier research, they are often not conducted at the beginning of the development process when the scope and content of the to-be-developed eHealth technology still have to be determined. An exception is when an existing technology is evaluated in the contextual inquiry to collect input for redesign. Possible goals of usability testing are to identify points of improvements of a low- or high-fidelity prototype according to experts and/or end users (see the Prototype section under Definition, Applicability, and Lessons Learned per Product) [77,81,127,152]; to evaluate a technology’s potential to improve problems in a specific organization [77]; to analyze the way a high-fidelity prototype is used by prospective end users [24,84]; to assess whether the prototype fits the current work practice of end users such as nurses [25]; or to generate new, or further specify, values or requirements [131].
Lessons Learned
First, multiple authors indicated that usability tests should be conducted with a broad range of stakeholders: not only end users such as patients, but also caregivers, managers, technology developers, and experts on content and design [21,43,54,64,77,112,127,131,152,155]. Different types of participants can provide different kinds of feedback on an eHealth technology. It was suggested that experts on, for example, usability or persuasive design can be included by means of cognitive walkthroughs or heuristic evaluations, whereas users can be involved through think-aloud procedures, guided by scenarios [127]. Second, authors stated that ideally, multiple methods should be combined to paint a full picture of a prototype’s usability [24,25,64,77,80,118,152]; for example, qualitative methods such as think-aloud procedures and interviews can be combined. Qualitative approaches can also be combined with quantitative data collected by means of, for example, log data analyses, questionnaire data, or eye tracking [127]. A way to do this is by using an iterative approach based on the user-centered design framework [81,127]. Third, in multiple studies, authors indicated that values or requirements of the to-be-developed eHealth technology can be used to guide usability tests. These can be used to structure data collection by, for example, verifying whether the requirements are present in the technology, or to analyze the data by means of deductive coding using the values to ensure that everything is well aligned [8,21,42,54,113,131].
Literature Study
Definition
Although there are many ways to conduct a literature study, for example, rapid, systematic, or scoping reviews, they all aim to create an overview of a certain topic using scientific literature, often in a systematic manner.
Examples of Applications
Of the 160 included articles, 43 (26.9%) were literature studies, most of which were conducted at the start of a development process to create an overview or get acquainted with a specific topic. The included studies in this review ranged from relatively unstructured, quick literature scans to elaborate systematic reviews in line with PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. An example of a literature study is a systematic review, which is often used to provide a complete, exhaustive summary of the current literature on a specific topic in a highly structured way. Another example that is often used in eHealth research is a scoping review, which offers insight into the status quo of scientific literature in a certain broad field of study by means of a systematic search of literature without paying too much attention to the quality of the studies. Possible goals of literature studies include the following: to gain insight into specific problems such as antimicrobial resistance or into broad domains such as technology in forensic psychiatry [158], to gain insight into development-related questions such as suitable persuasive features or behavior change techniques (BCTs) [159], or to develop recommendations for the design of specific interventions such as embodied conversational agents [172].
Lessons Learned
First, in several eHealth development studies, literature reviews were conducted quite rapidly and a systematic approach was not described. However, multiple authors indicated that it is always important to use a systematic approach when planning and executing any sort of literature review to ensure completeness of the results [158,159,166,167,173]. Second, multiple authors pointed out that often a lot of time has passed between initial data collection and publication of studies—which is especially relevant for the rapidly changing domain of eHealth. Consequently, not all state-of-the-art knowledge on technologies is published at the point of conducting the review [99,158,163,167,172]. Therefore, researchers can combine scientific literature with gray literature such as policy documents, company reports, or communication about work in progress to provide a more comprehensive overview of the current state of affairs. Third, because many literature studies in the contextual inquiry have a broad setup, it is often not possible or desirable to include a quality appraisal or only include experimental studies. If this is done, much valuable information on, for example, potential applications of a technology might be missed. In any case, it is important to reflect on the choices that were made in the reviewing process when interpreting the results [158-160].
Desk Research
Definition
Desk research refers to the nonsystematic collection of often nonscientific material such as presentations of an intervention, management reports, project documents, or activities or tasks of stakeholders. This material supports the development team in learning as much as possible about a specific topic.
Examples of Applications
Of the 160 included studies, 15 (9.4%) were desk research studies, which ranged from very unstructured studies, for example, talking to people or checking correspondence, to relatively structured studies, for example, a systematic analysis of the content of nonscientific documents or protocols. Desk research can be used throughout all phases of the development process when questions arise that do not require thorough, replicable research. In the included studies, desk research was used most often at the beginning of a development process to gain insight into specific fields or problems. Examples of applications include the following: to identify stakeholders, to gather information on protocols or guidelines [34,98], to create an overview of nonpublished projects on a technology in a specific sector [15], to observe existing face-to-face workshops to gain insight into their content [66], to gather correspondence for the analysis of existing communication processes [33], or to search for similar (eHealth) interventions [43].
Lessons Learned
First of all, authors stated that desk research should not be viewed as a synonym for randomly collecting information. Desk research should be used as a method that requires a clear research question and, if possible, a systematic search strategy that is clearly connected to the goal of the eHealth development process [15]. To illustrate, in the study by Breeman et al [81], desk research that was used to evaluate existing eHealth apps was structured by means of a newly developed evaluation tool for apps. Second, there is not necessarily a limit to the kinds of materials that can be included in desk research, depending on the research question. Some examples are apps that are available in an app store, scientific literature, protocols on cognitive behavioral therapy, written communications, policy documents, and presentations at meetings or conferences [33,98]. Third, it was suggested that desk research is a suitable method to look outside of a project’s scope to learn from the use of an eHealth technology in other domains. For example, researchers in mental health care investigated how VR is used in other settings such as hospitals or advertising [15].
Log Data Analysis
Definition
Log data are objective registrations of events that can be recorded on an individual basis, such as logging in to a website, entering a room (measured by sensors), sending a message, or performing a certain action in an app. These often large data sets can be analyzed to gain insight into behavior patterns.
Examples of Applications
Although log data are often used for the evaluation of the use of an already implemented eHealth technology, of the 160 included studies, 10 (6.3%) were log data studies that showed that log data can also provide valuable input for the development process. Log data can be analyzed in different ways, ranging from descriptive statistics to more complex analyses such as Markov modeling. In development, log data can, for example, be used to gain insight into use and points of improvement of (high-fidelity) prototypes [86,175] or to provide insight into dos and don’ts for the design of similar to-be-developed systems by analyzing existing similar websites [176].
Lessons Learned
First, authors stated that log data can be used in many ways to gain insight into online or offline behavior of prospective end users, not just eHealth use. Analyses can be performed with real-life log data on, for example, the use of an existing eHealth technology or with log data on the use of interaction-enabled prototypes in laboratory settings. Log data can also be collected on behavior in real-life settings that is not related to the use of an intervention but is useful for the contextual inquiry. An example is logging the number of times that doors within a nursing ward were opened [111,175]. Second, when log data on a prototype or newly developed eHealth technology are collected, researchers have to discuss with designers what data they want and how they should be collected beforehand. A well–thought out activity log protocol that describes which features should be logged should guide this debate [175]. Third, although very complicated analyses can be conducted with log data, straightforward descriptive statistics can often also be very valuable for researchers during the eHealth development process. This makes log data analysis a less complicated method than is often assumed [175].
Card Sorting
Definition
Card sorting is a method that can help design or evaluate the information architecture and structure of a technology. It can be used to structure units of information provided on cards. A distinction can be made between closed- and open-ended card sorts. In closed-ended card sorts, the main categories are provided by the researchers, whereas in open-ended card sorts, participants create their own main categories.
Examples of Applications
Of the 160 included studies, 9 (5.6%) involved card sorting. Card sorting is often used a bit further along the development process because it is often focused on the content and structure of an eHealth technology. Card sorting can be used to create new information structures or validate existing ones, for example, the structure of menus on websites with infection control guidelines [77,98], or to create a logical content structure that is in line with the users’ mental model, for example, for apps or web-based tools for health care workers [34].
Lessons Learned
Multiple authors indicated that card sorting is a fast, efficient, and cheap method to gain insight into the structure of a to-be-developed eHealth technology, especially when conducted on the web [25,70,77,98]. Web-based card sorting using programs such as Optimal Workshop software was said to be less prone to error, less labor intensive, and more efficient because larger numbers of participants can be included with less effort. Second, card sorting offers a concrete way to support people in expressing their thoughts and needs regarding an eHealth technology. This helps researchers gain more insight into how information should be structured. Third, to paint a complete picture of prospective end users’ requirements for the eHealth technology, card sorting should be combined with other methods such as interviews or focus groups [25,77,98,102]. In addition, the categories or structures that result from card sorting should be validated by using methods such as usability testing or interviews [25].
Delphi Study
Definition
A Delphi study offers a systematic way to determine consensus among various stakeholders, mostly experts. It solicits opinions from groups in an iterative process of answering questions, usually in multiple rounds.
Examples of Applications
Delphi studies can be used at multiple points in the eHealth development process when consensus among stakeholders—often experts—is needed. They can, for example, be used to gain insight into preferences for the eHealth technology or to reach consensus about values or requirements. Of the 160 included studies, 3 (1.9%) were Delphi studies, which used this method to identify expert recommendations for parents to reduce the risk of depression or anxiety in their children to include in the eHealth technology [72], to reach consensus on the most optimal way to integrate an eHealth technology in standard treatment [101], and to gain insight into expert opinions on relevant self-management behaviors for reducing the impact of COPD [84].
Lessons Learned
First of all, authors explained that Delphi studies have to consist of multiple rounds of data collection to reach consensus. This can be done using web-based questionnaires or focus groups in which participants have to participate at least two times [72]. Other types of data can be used as a starting point, such as scientific literature or outcomes from interview studies. Second, it can be challenging to recruit participants for a Delphi study because, for example, there are not many experts in a specific or new field, which can be especially relevant when studying novel applications of an eHealth technology [101]. This implies that researchers have to carefully think about whom to involve, for example, through snowball sampling, before starting with the study. Third, reaching consensus can take time and can be complex. This means that Delphi studies can require a lot of time from researchers and participants, which is not always available in eHealth development processes [101].
Experience Sampling
Definition
Experience sampling—sometimes also referred to as ecological momentary assessment (EMA)—is a structured diary method. It can be used to gather relevant, subjective experiences such as physical symptoms, mood, and behavior in daily life using multiple measures throughout the day using pen and paper or technology such as apps or wearables.
Examples of Applications
Of the 160 included studies, only 1 (0.6%) involved experience sampling, which was used to gain insight into information-seeking behavior during an EHEC (enterohemorrhagic Escherichia coli) outbreak [130].
Lessons Learned
Although experience sampling was used in only a single study, multiple lessons learned were provided. These lessons are not directly related to, but seem to be very valuable for, eHealth development. First, experience sampling was found to be a suitable method to explore an existing situation and thus to provide valuable input for further eHealth development [130]. Second, including participants who are willing and able to constantly provide input on their experience for a longer period of time can be challenging and time consuming, which also applies when it comes to ensuring that participants do not drop out during the study [130]. Third, when researchers want to study experiences during a specific event such as a virus outbreak, they ideally want to start collecting data from the beginning of the outbreak. However, in practice, this is often impossible. This can be compensated for by using a survey with retrospective questions; however, participants might be unable to correctly and completely recall their experiences [130].
Definition, Applicability, and Lessons Learned Per Product
Overview
Whereas methods are used to collect new data, products are based on, or summarize, these previously collected data. They can also be used as tools to collect new data. For each identified product, the definition, ways of applying it, and a maximum of 3 lessons learned are described in the following sections.
Prototype
Definition
Prototypes are visual representations of a to-be-developed technology, ranging from low- to high-fidelity representations. Low-fidelity prototypes often do not contain much detail, allow no automatic interaction between user and prototype, and can be relatively easy to create. Examples are paper-based sketches or wireframes, possibly combined with scenarios. High-fidelity prototypes are mostly digital; often cost more time, money, and technical skills to develop; and allow for user interactions, such as programmed apps or digital, interaction-enabled prototypes.
Examples of Applications
Prototypes were used in 20% (32/160) of the included studies, and in these studies, most prototypes were created during the design phase. However, it is also possible to create low-fidelity prototypes early in the development process, for example, when presenting initial ideas about a technology to participants [15]. Furthermore, end users or other stakeholders can create prototypes themselves in cocreation sessions to visualize their ideas and preferences [113]. Prototypes are often based on values and requirements [15,55,113]. In addition, multiple prototypes are often created and improved based on outcomes of usability testing [52,88].
Lessons Learned
First, the included studies showed that there is not 1 single way to create a prototype. Methods of creating a prototype can range from very quick and dirty without any content to very complex, creating a highly interactive product [40,55,121]. This depends on the goal, for example, to check the overall structure of a prototype or to identify usability problems. Second, it is important to have a rough idea of the costs of developing a technology as soon as possible, mostly to prevent the final version of the prototype from seeming to be too expensive. Inclusion of technology developers from the start of the development process was recommended to prevent these problems [46,113]. Third, because major changes might be made to prototypes in an iterative development process, changes to the requirements might also be necessary to ensure that they remain in line with the prototype of the eHealth technology. This highlights the importance of an iterative approach [55,113,115]. In line with this, making major changes to prototypes could require time and resources that are not available, which might result in a suboptimal prototype [88].
Requirements
Definition
Requirements are short statements that prescribe what is required of a technology: “They describe what a technology should do, what data it should store or retrieve, what content it should display, and what kind of user experience it should provide” [179].
Examples of Applications
Requirements were formulated in 6.9% (11/160) of the included studies. They were never formulated at the beginning of the development process because they should be based on outcomes of earlier activities. The included studies showed that requirements can be based on data generated by 1 method, such as interviews, or on a combination of data from different methods and scientific literature [91]. Requirements can be based on values, where values serve as a bridge between the previously conducted research and the specific requirements [129]. Requirements can be used to specify previously formulated values, to serve as foundations for prototypes, or to communicate needs and wishes to developers and discuss these needs and wishes with them [15,30].
Lessons Learned
First of all, authors stated that requirements should be elicited in a systematic way. Ideally, multiple sources of data should be combined, such as scientific literature and qualitative data collected from multiple types of stakeholders [91,114,115]. When combining sources, development teams need to be aware of conflicting requirements, for example, a discrepancy between user needs and scientific literature [91,92]. Second, it might not be possible to include all requirements in an eHealth technology because of practical, technical, or financial limitations. However, that does not mean that it is not worth the effort to further specify them: these requirements might be incorporated in the technology at a later point in time [38,39]. Third, it was mentioned that eliciting requirements from target groups comprising patients who are vulnerable and clinically complex, such as people with psychosis or dementia, might be challenging. Consequently, researchers should carefully select methods that fit the characteristics and skills of these populations [38,39,70].
Stakeholder Map
Definition
A stakeholder map is a visualized overview of stakeholders—people or organizations who affect or are affected by an eHealth technology—and their interrelationships.
Examples of Applications
Stakeholder maps were reported in 6.3% (10/160) of the included studies. Stakeholders should be identified from the start of the development process, and this overview should be updated throughout the entire process. Examples of stakeholders are patients, experts on a specific topic such as depression, a commercial company that can develop apps or VR, health care providers, researchers on eHealth or the health problem at hand, and employees of other organizations that could use the intervention later [15,129,153]. The roles and tasks of these stakeholders regarding the to-be-developed eHealth technology should be identified by means of methods such as interviews or desk research.
Lessons Learned
First, in the included papers it was shown that stakeholder maps are not created from scratch but are based on data. Often, a stakeholder map is created by means of stakeholder identification: the systematic process of finding out who the stakeholders of an eHealth development process are. Stakeholder identification can be supplemented by a stakeholder analysis, which refers to the analysis of interdependencies, responsibilities, and stakes of the identified stakeholders. To achieve this, ideally, a combination of research methods such as questionnaires, interviews, or literature reviews is used [27,129]. Second, the studies stated that the importance of a stakeholder map should not be underestimated. Thorough investigation of stakeholders and their context is very important not only for the entire development process, but also for implementation and evaluation of the to-be-developed eHealth technology [30,35]. Consequently, the stakeholder map should be constantly updated throughout the process because new insights might arise or new stakeholders might emerge [35,59]. Third, it was considered to be important to include a wide variety of stakeholders, where researchers should look beyond their own setting. For example, when developing a VR intervention for forensic mental health care, researchers should also include stakeholders in other organizations where VR is used, such as hospitals [15,35,153].
Values
Definition
Values refer to ideals or interests of stakeholders: they specify what stakeholders want to achieve or improve by means of an eHealth technology and capture what the added value of a technology should be for the people and organization involved.
Examples of Applications
Of the 160 included studies, 7 (4.4%) involved values, which were created based on outcomes of previously conducted research. This means that values are often formulated later in the development process, mostly during the CeHRes Roadmap’s value specification phase, hence the name. Values remain relevant throughout the remainder of the process because they can also serve as foundations for requirements and prototypes. Values can be used to summarize or synthesize outcomes of previously conducted studies such as interviews or questionnaires. They can also serve as foundations for prototypes, implementation plans, or evaluation goals [97]. Examples of values are improvement of skills, easy to use in current treatment, affordability, self-management, and positive self-image [79,97].
Lessons Learned
First, authors stated that values should be formulated in such a way that they are neither too specific to prevent overlap with requirements nor too broad and vague. A shared understanding among the eHealth development team members about what values are is essential to achieve this [15,64,97,152]. Second, values should capture the whole range of stakeholder needs and wishes, not merely those of end users of the eHealth technology [81,129]. To paint a complete picture, multiple methods should be used. In addition, when new insights arise, values should be updated by a multidisciplinary research team to ensure that they continue to align with the perspectives of the key stakeholders [15,62,79,97,129]. Third, conflicting values might arise related to, for example, costs or the focus of an eHealth technology. A good way to resolve these conflicts is by discussing them with a group consisting of multiple types of stakeholders [79,97]. In addition, researchers should not view a value map as static: it might have to be adjusted based on changes in the context and users [129].
Behavior Change Strategies
Definition
Behavior change strategies such as evidence-based BCTs or persuasive elements can be integrated in the design of eHealth technologies to increase their effectiveness.
Examples of Applications
Of the 160 included studies, 5 (3.1%) featured behavior change strategies, which were mostly used in the design phase. The product was used to determine which theory-based methods should be included in an intervention to increase the chances of achieving behavior change. Examples are the inclusion of BCTs such as goal setting and self-monitoring in a mobile app to increase vegetable consumption [69] or the application of 3 theory-based methods and 4 accompanying strategies to influence the attitudes and skills of patients to support them in their communication with health care professionals [109]. In the study by Asbjørnsen et al [79], researchers used methods from design thinking to translate values and needs of people with obesity into persuasive features and behavior change theories such as goals and planning, personalization, and self-monitoring.
Lessons Learned
First, behavior change theories should ideally be combined with outcomes of human-centered design methods, instead of being mostly researcher based. However, more insight is required into how this should be done in eHealth development [79,109]. Second, operationalizing theoretical strategies into a user-friendly eHealth technology might be challenging [79,91]. To overcome this, participatory approaches such as cocreation sessions or prototyping workshops can be used to determine how theoretical working mechanisms can be translated into features of an eHealth technology [69,79]. Third, traditional behavior change theories might be too static to integrate into adaptive eHealth technologies, especially in the case of just-in-time personalized interventions. This implies that new types of theories might be required [69].
Persona
Definition
Personas are user archetypes that summarize a representative person from the target group. They consist of a description of different types of characteristics of a future or actual user, often in the form of a story.
Examples of Applications
Of the 160 included studies, 5 (3.1%) described personas. In the study by Dick et al [73], 3 personas of users of an eHealth intervention for illicit substance use were created: the heavy user, abstainer, and occasional user, whereas the study by Derks et al [70] developed personas for an intervention for people with cardiovascular disease and the study by Breeman et al [81] developed personas for an intervention for people with borderline personality disorder.
Lessons Learned
First of all, it is recommended to use existing guidelines and frameworks such as that of LeRouge et al [180] when developing personas for eHealth technologies. These frameworks should be combined with human-centered research methods such as focus groups [67,70,81]. Second, to structure the persona-building process, researchers need to identify characteristic categories that need to be included in the persona, in which attention should also be paid to skills and attitudes related to the to-be-developed eHealth technology. Examples are demographics and personality of the service user; their medical and psychological profile, including fears and motivations for behavior; their abilities, (technological) skills, and coping strategies; and their needs and goals [67,70,73,81]. Finally, personas were seen as a useful tool to tailor the content and design of eHealth technologies and can be connected to the requirements or to BCTs [73].
Business Model
Definition
A business model captures how an organization creates, delivers, and captures values; it describes how an organization conducts its business. It is a conceptual and analytical framework to map, discuss, and help realize the added value of an eHealth technology, as well as to determine the key factors that are associated with a sound and sustainable implementation.
Examples of Applications
Of the 160 included studies, 4 (3%) described business models. Often, the development of business models is initiated during the first stages of a development process because development and implementation should be intertwined [84]. However, the studies showed that a business model is not finished during development: it should be updated throughout the entire development and implementation process. An example of its application is the use of the business model canvas for an eHealth portal for infection control. This model includes the technology’s key partners, key activities and key resources, cost structure, revenue streams, value proposition, customer relationships, customer segments, and channels [27].
Lessons Learned
First, researchers concluded that perspectives from all important stakeholders should be accounted for in a business model. This can be done by means of focus groups with multiple stakeholders, integration of earlier collected data, or in-depth interviews [84]. When collecting input for the business model, it was indicated that questions to participants should be very concrete because abstract questions will yield equally abstract and thus less useful answers [99]. Second, experience has shown that creating a business model is very time consuming, partly because there are no business models specifically for eHealth yet [27,99]. Third, a business model often does not have a fixed end and needs to be adapted continuously throughout the implementation and evaluation phases of an eHealth technology [27].
Discussion
Principal Findings
The main goal of this narrative scoping review is to create an initial overview of the methods used in eHealth development processes guided by the CeHRes Roadmap. Furthermore, we aim to identify for what purposes and in which phases these methods are used and provide an overview of the most relevant lessons learned. During the analysis, it became clear that a distinction between development methods and products can be made. In the 160 included studies, 10 overarching methods and 7 products were identified. Most of the identified methods were used in all 3 development phases of the CeHRes Roadmap. They were used for a broad range of goals, underlining the many different possibilities that exist for eHealth development. The lessons learned showed that most authors agreed that the methods and products contributed to eHealth development in a positive way by providing more insight into the users and context. However, a critical reflection on the methods or products and accompanying conclusions related to a method or product not being suitable were often not provided in the included studies. Regardless, authors mentioned multiple barriers and limitations that they had to account for, which differ per method and product. On the basis of the many lessons learned that we identified, there seems to be ample experience with, and knowledge about, different types of development methods. However, this knowledge remains mostly segregated and there could be more room for critical reflection on the suitability of a research activity. Furthermore, several potentially useful methods and products such as experience sampling, personas, or behavior change strategies seem to be underrepresented in the included studies. This underlines the need for better integration and broader dissemination of knowledge on eHealth development. Integrating and sharing knowledge can enable researchers to select the most fitting method or product, as opposed to using the one that is most easily available or well-known.
Comparison With Prior Work
Methods Versus Products
In this review, a distinction was made between methods and products in eHealth development. This distinction does justice to the diverging and converging nature of eHealth development. This is in line with design-thinking approaches such as the double diamond model, which pays attention to diverging, for example, by means of collecting data, and converging, for example, by means of integrating the findings in a set of requirements or a prototype [15,181]. Similarly, based on our findings, we recommend that methods and products should be seen not as separate activities but as 2 sides of the same eHealth development coin. This review showed that methods such as interviews, questionnaires, or literature studies are used to collect new data. These data can be translated into concrete products to synthesize collected data and to support and facilitate subsequent collection of new data. Thus, these products can serve as stepping stones among data collection methods. An example of using products as a synthesis approach is the formulation of values based on previously conducted interviews and questionnaires [15,97]. An example of the use of products to collect novel data is the use of a low-fidelity prototype in usability testing to gain insights into prospective users’ needs and wishes [131]. The combination of methods and products illustrates how researchers can continuously check assumptions and decisions with stakeholders in a concrete, specific way by means of formative evaluation cycles [11]. However, currently, the terminology used in literature does not reflect the distinction among the different types of development activities. Every development-related activity is referred to as a method, or products are only briefly described or mentioned as a tool within a method. On the basis of this review, we suggest that researchers should make an explicit distinction between methods and products when reporting on their eHealth development process.
Iterative eHealth Development
This review showed that there is no single ideal way to conduct a development process. The included studies illustrate that there are multiple methods that can be used in many different ways, for many different goals, and at many different points in the development process. To illustrate, for identifying points of improvement in a current situation according to stakeholders, researchers might use interviews, focus groups, and questionnaires. In addition, values and requirements can be elicited by means of both an interview study and focus groups in which prototypes are evaluated [96,97,178,179]. Consequently, multiple methods can be used to reach similar goals. This implies that creating an ideal, step-by-step guideline with predetermined methods is neither possible nor desirable because this would suggest that there is a single most optimal way to conduct a development process. However, this conclusion does not mean that anything goes. On the basis of the lessons learned, it can be concluded that a specific method or product should fit the context in which the data collection takes place as well as the characteristics and skills of the participants and the outcomes of earlier development activities [182]. To illustrate, although in-depth interviews might be a useful way to gather information from health care workers and experts, they seem to be less suitable for people with dementia because of the cognitive skills that would be required [48,61,178]. In addition, although these approaches can be used to collect valuable data, they also illustrate that over the past 10 years, there has not been much innovation in the methods and products used in eHealth development. This is especially striking considering the rapid changes in the possibilities of eHealth technologies such as mobile apps or VR over the past years. Scientific methods such as literature reviews, focus groups, and interviews have been used very often, whereas innovative methods of a more participatory nature, for example, generative methods such as photo diaries and mood boards, are hardly used, despite their potential added value.
In addition, this review has shown that often, multiple methods can—and should—be combined to paint a full picture of the current situation and possibilities of eHealth. If done well, the use of multiple methods to incorporate the stakeholders’ perspective throughout the entire process will result in an eHealth technology that fits their needs, wishes, skills, and context. This fit among technology, stakeholders, and context will also increase the chances of the uptake of eHealth in practice. Methods can be used in different ways to include the stakeholder perspective. On the one hand, methods can be used to validate findings; for example, interviews can be used to provide more context to the results of a literature review [15]. On the other hand, methods and products can also be used in a complementary way; for example, although usability tests can be used to elicit requirements from the end user, they do not provide insight into the demands of the organization. This knowledge can be obtained by means of complementary focus groups with managers. Consequently, when selecting methods throughout the process, researchers should account for the suitability of the methods and products for their specific context, participants, and earlier findings of the development process. Furthermore, this review showed that not all methods used in human-centered development processes have to be participatory to be useful; for example, literature reviews or log data analyses can also be valuable. Nonparticipatory methods can be complemented with human-centered methods such as interviews or focus groups to form a coherent whole.
Finally, the lessons learned illustrated that when shaping a multi-method or mixed methods development process, an iterative and flexible development approach is key [33,54,55,64,113,115]. Ideally, decisions for the next development activities should be based on findings, new insights, and experiences that arose from previous activities. Such an iterative approach ensures that the development process fits the context and the people involved. This is in line with agile science, which states that eHealth development should be iterative and flexible, with short sprints to allow for constant changes to the process and products [14,183]. An agile approach requires a flexible mindset of researchers because it is often not possible to plan the entire development process from the start. To support researchers, the results of this review can serve as input for the creation of a CeHRes Roadmap Toolkit for selecting the most appropriate and fitting methods, as opposed to choosing the method that they are most familiar with. To broaden the scope of this toolkit, it could be complemented with other research, for example, reviews such as this one, but ones that are focused on other frameworks. Another way to broaden the toolkit is by asking experts on, for example, human-centered design and eHealth development about other potentially useful methods. Such a broader toolkit can support researchers in adapting their development process in case of contextual changes or unexpected or new findings [14,183]. This can facilitate the iterative nature of the development process.
Points of Improvement for eHealth Development Models
On the basis of the findings of this review, multiple points of improvement for the CeHRes Roadmap can be formulated. These can also be partly applied to other eHealth development models. First, it is important to note that most studies did not use all phases of the CeHRes Roadmap; rather, they cherry-picked the most relevant phases or principles to use in their work. Often, the CeHRes Roadmap was combined with other development approaches such as design thinking or the Behavior Change Wheel. This illustrates that in eHealth development, (parts of) multiple models, theories, and approaches can be used to complement each other [91]. Second, although business modeling and value specification are important elements of the CeHRes Roadmap, they were underrepresented in the included studies. This indicates that in general, researchers might require more guidelines on how to not only develop an intervention, but also think about its implementation in practice from the start of development. Third, a main point of improvement for eHealth development using the CeHRes Roadmap is related to the use of behavior change theory: this approach was underrepresented in the included studies [81]. This is quite surprising, especially considering the relationship between behavior change theory and effectiveness of (eHealth) interventions [184,185]. A possible explanation for this gap is that design-oriented, development models such as the CeHRes Roadmap do not explicitly force or nudge developers to incorporate theory, as opposed to more human-centered activities such as stakeholder identification or prototyping. Consequently, separate goals or activities might be added to the CeHRes Roadmap to support developers in incorporating behavior change theory [186,187]. In line with this, there is a need for more guidelines on how behavior change theory can be combined or complemented with persuasive design features. In addition, there is a need for more research on how this can in turn be connected to stakeholders’ values and requirements [79,81,159]. An important point of attention in regard to this is that there is still much debate about the suitability of existing behavior change models for eHealth. Are existing theories suitable, or are they too static for highly personalized eHealth technologies? In other words: is there a need for more dynamic, personalized models for behavior change [188]? A final recommendation is related to applying development models to practice. As conducting development processes in practice is often very complex and challenging, multiple authors recommended forming a multidisciplinary development team. Such a team can consist of, for example, researchers, patients, health care providers, technology developers, designers, and managers [15,88,90]. A multidisciplinary team can prevent a top-down approach and tunnel vision; ensure that a constant eye is kept on the context; facilitate the coordination of large, complex development processes by combining skills and knowledge; and even support implementation of the to-be-developed eHealth technology in practice [182].
Strengths and Limitations
The main goal of this explorative narrative scoping review is to create an overview of development methods used in eHealth development guided by the CeHRes Roadmap. Although the CeHRes Roadmap is viewed as a suitable framework to guide eHealth development, there are many more useful development models. Because of our focus on studies that used the CeHRes Roadmap, several other suitable research methods or products might have been overlooked. Examples of such methods are observations, EMAs, or eye tracking [189-191]. Therefore, this overview of development methods and products is neither exhaustive nor unconditionally generalizable. Regardless, because of the interdisciplinary and broad applicability of the CeHRes Roadmap, much of the knowledge generated in this review is also relevant for other development models. However, creating a broader eHealth development toolkit would require additional similar reviews in which other development frameworks are put central. Subsequent analysis of differences and similarities among these reviews might also allow for generalization of knowledge. In line with this, because providing an elaborate, in-depth reflection on the included methods and products is beyond the scope of this paper, the provided lessons learned in this narrative scoping review are also not exhaustive. To do justice to the possibilities of all mentioned methods and products, a separate review per method is required. In addition, because of the goal and broad nature of our scoping review, we did not assess the quality of the included studies. The main reason for this was that the quality of a study does not say anything about the quality of a development method or product. In other words, if a study in which interviews were used in a contextual inquiry is of extremely high methodological quality, this does not automatically mean that interviews in themselves are always the most suitable method for contextual inquiries. On the basis of the reported lessons learned, researchers have to make their own choice for the most suitable method or product for their process. Furthermore, a pitfall of the qualitative approach that was used to describe the methods and products is that it is prone to potential bias of the researchers. To overcome this, the screening and summarizing processes were set up in a systematic way and conducted by 3 researchers. In addition, all findings were discussed among, and critically reflected upon by, the coauthors—all with ample experience in multidisciplinary, human-centered eHealth development. Finally, as is the case in any review, not all relevant information on development methods might have been published. On the one hand, this might be because, too often, development studies are not viewed as full-fledged research and are simply not published. An example of this is a stakeholder analysis, which is often conducted but hardly ever included in publications [135]. On the other hand, we noticed that there is much difference between the clarity and extensiveness of the way authors reported and reflected on development activities. Suboptimal or incomplete reflection by authors resulted in some lessons learned that remained quite general and were not specifically focused on eHealth development. In general, this might have influenced our findings and highlights the need for a larger number of, and more comprehensive, papers on eHealth development.
Future Directions
As stated earlier, the results of this review should be viewed not as an exhaustive overview of all existing eHealth development methods and products but as an initial overview of activities related to research based on the CeHRes Roadmap. First, in multiple studies, the methods used were described in very general terms or the papers only described a single method from a larger development process [78]. Consequently, additional interviews, focus groups, or questionnaires—the choice of method naturally depending on the type of context and characteristics of the users—can be conducted with researchers to gain more in-depth insight into their experiences with eHealth development. This can also be extended to experts working in practice because innovation in human-centered design often is initiated in practice. In line with this, more specific reviews or viewpoint papers on single development methods or products might be conducted.
Second, a more general recommendation is related to the method of reporting. On the basis of the included studies, it becomes clear that there are many differences in the way authors report on development processes. In addition, as can be seen in the relatively low number of reported products compared with reported methods, there is room for improvement in the description of products used. We recommend that future development studies explicitly substantiate choices for, and critically reflect on, the methods and products used. Future research might explore the desirability and feasibility of a standardized guideline for development studies in line with the standards for reporting on, for example, randomized controlled trials (CONSORT [Consolidated Standards of Reporting Trials]) [192], reviews (PRISMA) [193], or qualitative research such as interviews (Consolidated Criteria for Reporting Qualitative Studies) [194]. On the basis of our findings, we recommend that researchers at least report on (1) the specific goal of the method, (2) a replicable description of the materials and procedure of data collection, (3) the rationale behind choosing the method, (4) the participants, (5) the setting, (6) the main results and conclusions, and (7) the most important lessons learned about the use of the method within the specific context [24]. In case of products, we recommend that they be described in a clear way and, if possible, that the products themselves be provided in the paper or appendix. Examples are screenshots of prototypes, a list with values, and an overview of stakeholders. In addition, multiple included studies showed that a visualization of the development process might also be of added value [15].
Third, the COVID-19 pandemic has shown us the potential of online research. Organizing web-based usability testing, focus groups, or interviews is sometimes not only necessary, but they can also have practical and even substantive advantages [87]. Future research can focus on if and how research from a distance can be applied to make eHealth development more efficient and effective. Finally, the results from this review can be used in the development of a digital CeHRes Roadmap Toolkit. This can be achieved by means of a human-centered development process in which end users—in this case, researchers and developers—are actively involved using suitable methods. Before a more comprehensive and exhaustive eHealth toolkit can be developed, there is a need for additional reviews that take a similar approach as this review but focus on other development models such as intervention mapping or user-centered design. This might result in a more exhaustive overview of other development methods and products that were not represented in this review. Furthermore, in line with the interdisciplinary nature of eHealth development, this eHealth toolkit can also be based on existing, more generic toolkits such as the Communication and Media Design methods pack [195] and the Human-Centered Design Toolkit [196]. Developing a broad eHealth toolkit requires an interdisciplinary approach in which designers with both a research and commercial background, psychologists, health scientists, other relevant stakeholders, and researchers combine their expertise. Including expertise from commercial design companies might be useful or even necessary to innovate eHealth development and go beyond the more traditional, scientific approaches.
Conclusions
This narrative scoping review has resulted in an initial overview of multiple methods and products that can be used for development of eHealth, guided by the CeHRes Roadmap. It can be concluded that there is not a single straightforward, step-by-step way to conduct a development process: the way a process is shaped depends on the type of technology, context, and stakeholders. To support researchers in selecting the most appropriate and fitting methods and products for their development process, it is recommended to develop an eHealth development toolkit. To achieve this ultimate aim, there is a need not only for more research, but also for close collaboration with industry. In this way, eHealth development becomes less reliant on traditional research approaches such as literature reviews or interviews. Researchers can be challenged to create more creative and innovative development processes by using methods and products that are developed by industry. This review can serve as a blueprint for further reviews in which different frameworks are put central. In general, the decision for the method or product should be based on three different elements: (1) the specific goal of the current development phase, (2) the characteristics of the participants, and (3) practical requirements from the context. In this way, eHealth development processes can become more flexible and will be professionalized further. This in turn might result in more effective, personalized eHealth technologies that fit with, and are of added value for, the users and their context.
Abbreviations
- BCT
behavior change technique
- CeHRes
Center for eHealth Research
- CONSORT
Consolidated Standards of Reporting Trials
- COPD
chronic obstructive pulmonary disease
- EMA
ecological momentary assessment
- EHEC
enterohemorrhagic Escherichia coli
- ICT
information and communication technology
- PRISMA
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
- VR
virtual reality
Data extraction form.
Main goal of the study and described methods and products per included record.
Footnotes
Conflicts of Interest: None declared.
References
- 1.Van Gemert-Pijnen L, Kip H, Kelders SM, Sanderman R. eHealth Research, Theory and Development: A Multi-Disciplinary Approach. London: Routledge; 2018. Introducing eHealth; pp. 3–26. [Google Scholar]
- 2.van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, Seydel ER. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res. 2011 Dec 05;13(4):e111. doi: 10.2196/jmir.1672. https://www.jmir.org/2011/4/e111/ v13i4e111 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Eysenbach G. What is e-health? J Med Internet Res. 2001;3(2):E20. doi: 10.2196/jmir.3.2.e20. https://www.jmir.org/2001/2/e20/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, McKinstry B, Procter R, Majeed A, Sheikh A. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med. 2011 Jan 18;8(1):e1000387. doi: 10.1371/journal.pmed.1000387. https://dx.plos.org/10.1371/journal.pmed.1000387 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Van Gemert-Pijnen Kip L, Kelders SM, Kip H, Sanderman R, editors. eHealth Research, Theory and Development: A Multidisciplinary Approach. London: Routledge; 2018. [Google Scholar]
- 6.Kip H, Sieverink F, van Gemert-Pijnen JE, Bouman YH, Kelders SM. Integrating people, context, and technology in the implementation of a web-based intervention in forensic mental health care: mixed-methods study. J Med Internet Res. 2020 May 26;22(5):e16906. doi: 10.2196/16906. https://www.jmir.org/2020/5/e16906/ v22i5e16906 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res. 2015 Jan 30;17(1):e30. doi: 10.2196/jmir.4055. https://www.jmir.org/2015/1/e30/ v17i1e30 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Beerlage-de Jong N, van Gemert-Pijnen L, Wentzel J, Hendrix R, Siemons L. Technology to support integrated antimicrobial stewardship programs: a user centered and stakeholder driven development approach. Infect Dis Rep. 2017 Mar 30;9(1):6829. doi: 10.4081/idr.2017.6829. http://europepmc.org/abstract/MED/28458796 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Eldredge L, Markham C, Ruiter R, Kok G, Parcel G. Planning Health Promotion Programs An Intervention Mapping Approach. New York: John Wiley & Sons; 2016. [Google Scholar]
- 10.Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res. 2017 Jun 29;19(6):e232. doi: 10.2196/jmir.7126. https://www.jmir.org/2017/6/e232/ v19i6e232 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Kip H, Van Gemert-Pijnen L. Holistic development of ehealth technology. In: Van Gemert-Pijnen L, Kelders SM, Kip H, Sanderman R, editors. eHealth Research, Theory and Development: A Multi-Disciplinary Approach. London: Routledge; 2018. pp. 151–186. [Google Scholar]
- 12.Mohr DC, Lyon AR, Lattie EG, Reddy M, Schueller SM. Accelerating digital mental health research from early design and creation to successful implementation and sustainment. J Med Internet Res. 2017 May 10;19(5):e153. doi: 10.2196/jmir.7725. https://www.jmir.org/2017/5/e153/ v19i5e153 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Oinas-Kukkonen H, Harjumaa M. Persuasive systems design: key issues, process model, and system features. Commun Assoc Inf Syst. 2009 Mar;24(1):28. doi: 10.17705/1cais.02428. [DOI] [Google Scholar]
- 14.Hekler EB, Klasnja P, Riley WT, Buman MP, Huberty J, Rivera DE, Martin CA. Agile science: creating useful products for behavior change in the real world. Transl Behav Med. 2016 Jun;6(2):317–28. doi: 10.1007/s13142-016-0395-7. http://europepmc.org/abstract/MED/27357001 .10.1007/s13142-016-0395-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kip H, Kelders SM, Bouman YHA, van Gemert-Pijnen LJEWC. The Importance of Systematically Reporting and Reflecting on eHealth Development: Participatory Development Process of a Virtual Reality Application for Forensic Mental Health Care. J Med Internet Res. 2019 Aug 19;21(8):e12972. doi: 10.2196/12972. https://www.jmir.org/2019/8/e12972/ v21i8e12972 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Moore G, Wilding H, Gray K, Castle D. Participatory methods to engage health service users in the development of electronic health resources: systematic review. J Particip Med. 2019 Feb 22;11(1):e11474. doi: 10.2196/11474. https://jopm.jmir.org/2019/1/e11474/ v11i1e11474 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Maguire M. Methods to support human-centred design. Int J Human Comput Stud. 2001 Oct;55(4):587–634. doi: 10.1006/ijhc.2001.0503. [DOI] [Google Scholar]
- 18.Phases of the CeHRes Roadmap. University of Twente. [2022-01-25]. https://www.utwente.nl/en/bms/ehealth/cehres-roadmap-toolkit/cehres-phases/
- 19.Grant M, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009 Jun;26(2):91–108. doi: 10.1111/j.1471-1842.2009.00848.x. doi: 10.1111/j.1471-1842.2009.00848.x.HIR848 [DOI] [PubMed] [Google Scholar]
- 20.Van Dyk L, Wentzel M, Limburg, van M, Gemert-Pijnen, van JEWC, Schutte CS. Business models for sustained ehealth implementation: Lessons from two continents. Proceedings of International Conference on Computers and Industrial Engineering; Proceedings of International Conference on Computers and Industrial Engineering; Jul 16-18, 2012; Cape Town, South Africa. 2012. [Google Scholar]
- 21.Van Gemert-Pijnen L, Span M. Handbook of Smart Homes, Health Care and Well-Being. Cham: Springer; 2016. CeHRes roadmap to improve dementia care. [Google Scholar]
- 22.Kooistra LC, Ruwaard J, Wiersma JE, van Oppen P, van der Vaart R, van Gemert-Pijnen JE, Riper H. Development and initial evaluation of blended cognitive behavioural treatment for major depression in routine specialized mental health care. Internet Interv. 2016 May;4:61–71. doi: 10.1016/j.invent.2016.01.003. https://linkinghub.elsevier.com/retrieve/pii/S2214-7829(16)00003-8 .S2214-7829(16)00003-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Kulyk O, Van Gemert-Pijnen L, Daas Den C, David S. How persuasive are serious games, social media and mHealth technologies for vulnerable young adults? Design factors for health behavior and lifestyle change support: sexual health case. Proceedings of the 3rd International Workshop on Behavior Change Support Systems, BCSS 2015 - co-located with the 10th International Conference on Persuasive Technology, PERSUASIVE 2015; 3rd International Workshop on Behavior Change Support Systems, BCSS 2015 - co-located with the 10th International Conference on Persuasive Technology, PERSUASIVE 2015; Jun 3, 2015; Chicago, United States. 2015. https://abdn.pure.elsevier.com/en/publications/how-persuasive-are-serious-games-social-media-and-mhealth-technol . [Google Scholar]
- 24.Ossebaard HC, Seydel ER, van Gemert-Pijnen J. Online usability and patients with long-term conditions: a mixed-methods approach. Int J Med Inform. 2012 Jun;81(6):374–87. doi: 10.1016/j.ijmedinf.2011.12.010.S1386-5056(11)00270-X [DOI] [PubMed] [Google Scholar]
- 25.Wentzel J, van Velsen L, van Limburg M, de Jong N, Karreman J, Hendrix R, van Gemert-Pijnen J. Participatory eHealth development to support nurses in antimicrobial stewardship. BMC Med Inform Decis Mak. 2014 Jun 05;14:45. doi: 10.1186/1472-6947-14-45. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/1472-6947-14-45 .1472-6947-14-45 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Kulyk O, Akker RO, Klaassen R, Van Gemert-Pijnen J. Personalized virtual coaching for lifestyle support: principles for design and evaluation. Advances Life Sci. 2014;6:300–9. https://www.semanticscholar.org/paper/Personalized-Virtual-Coaching-for-Lifestyle-for-and-Kulyk-Akker/9cc75aa2bc8b8c06545a7aa1355c3e6c44d0c90e . [Google Scholar]
- 27.van Limburg M, Wentzel J, Sanderman R, van Gemert-Pijnen J. Business modeling to implement an eHealth portal for infection control: a reflection on co-creation with stakeholders. JMIR Res Protoc. 2015 Aug 13;4(3):e104. doi: 10.2196/resprot.4519. https://www.researchprotocols.org/2015/3/e104/ v4i3e104 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Wentzel J, Müller F, Beerlage-de Jong N, van Gemert-Pijnen J. Card sorting to evaluate the robustness of the information architecture of a protocol website. Int J Med Inform. 2016 Feb;86:71–81. doi: 10.1016/j.ijmedinf.2015.12.003.S1386-5056(15)30070-8 [DOI] [PubMed] [Google Scholar]
- 29.Beerlage-de Jong N, Eikelenboom-Boskamp A, Voss A, Sanderman R, van Gemert-Pijnen J. Combining user-centered design with the persuasive systems design model: the development process of a web-based registration and monitoring system for healthcare-associated infections in nursing homes. Int J Advances Life Sci. 2014 Jan;6:262–71. https://www.researchgate.net/publication/286645872_Combining_user-centered_design_with_the_persuasive_systems_design_model_The_development_process_of_a_web-based_registration_and_monitoring_system_for_healthcare-associated_infections_in_nursing_homes . [Google Scholar]
- 30.van Velsen L, Beaujean DJ, Wentzel J, Van Steenbergen JE, van Gemert-Pijnen JE. Developing requirements for a mobile app to support citizens in dealing with ticks and tick bites via end-user profiling. Health Informatics J. 2015 Mar;21(1):24–35. doi: 10.1177/1460458213496419. https://journals.sagepub.com/doi/10.1177/1460458213496419?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub%3dpubmed .1460458213496419 [DOI] [PubMed] [Google Scholar]
- 31.Kelders SM, Pots WT, Oskam MJ, Bohlmeijer ET, van Gemert-Pijnen J. Development of a web-based intervention for the indicated prevention of depression. BMC Med Inform Decis Mak. 2013 Feb 20;13:26. doi: 10.1186/1472-6947-13-26. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/1472-6947-13-26 .1472-6947-13-26 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Van de Dijk DD, Te Lintelo J, Willems CG, van Gemert-Pijnen J. ECOPD: user requirements of older people with copd for ehealth support at home, a user-centred study. Proceedings of the AAATE 2013; AAATE 2013; Sep, 2013; Villamoura Portugal. 2013. [DOI] [Google Scholar]
- 33.Gulmans J, Vollenbroek-Hutten M, Van Gemert-Pijnen J, Van Harten WH. Evaluating patient care communication in integrated care settings: application of a mixed method approach in cerebral palsy programs. Int J Qual Health Care. 2009 Feb;21(1):58–65. doi: 10.1093/intqhc/mzn053.mzn053 [DOI] [PubMed] [Google Scholar]
- 34.Verhoeven F, van Gemert-Pijnen J, Hendrix R. The development of a web-based tool for compliance with safe work practices; a case study about methicillin-resistant staphylococcus aureus. Proceedings of the 2009 International Conference on eHealth, Telemedicine, and Social Medicine; 2009 International Conference on eHealth, Telemedicine, and Social Medicine; Feb 1-7, 2009; Cancun, Mexico. 2009. [DOI] [Google Scholar]
- 35.Bossen D, Kloek C, Snippe HW, Dekker J, de Bakker D, Veenhof C. A blended intervention for patients with knee and hip osteoarthritis in the physical therapy practice: development and a pilot study. JMIR Res Protoc. 2016 Feb 24;5(1):e32. doi: 10.2196/resprot.5049. https://www.researchprotocols.org/2016/1/e32/ v5i1e32 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Ehrler F, Lovis C, Blondon K. A mobile phone app for bedside nursing care: design and development using an adapted software development life cycle model. JMIR Mhealth Uhealth. 2019 Apr 11;7(4):e12551. doi: 10.2196/12551. https://mhealth.jmir.org/2019/4/e12551/ v7i4e12551 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Cnossen IC, van Uden-Kraan CF, Eerenstein SE, Rinkel RN, Aalders IJ, van den Berg K, de Goede CJ, van Stijgeren AJ, Cruijff-Bijl Y, de Bree R, Leemans CR, Verdonck-de Leeuw IM. A participatory design approach to develop a web-based self-care program supporting early rehabilitation among patients after total laryngectomy. Folia Phoniatr Logop. 2015;67(4):193–201. doi: 10.1159/000441251. https://www.karger.com?DOI=10.1159/000441251 .000441251 [DOI] [PubMed] [Google Scholar]
- 38.Harder H, Holroyd P, Burkinshaw L, Watten P, Zammit C, Harris PR, Good A, Jenkins V. A user-centred approach to developing bWell, a mobile app for arm and shoulder exercises after breast cancer treatment. J Cancer Surviv. 2017 Dec;11(6):732–42. doi: 10.1007/s11764-017-0630-3. http://europepmc.org/abstract/MED/28741202 .10.1007/s11764-017-0630-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Honary M, Fisher NR, McNaney R, Lobban F. A web-based intervention for relatives of people experiencing psychosis or bipolar disorder: design study using a user-centered approach. JMIR Ment Health. 2018 Dec 07;5(4):e11473. doi: 10.2196/11473. https://mental.jmir.org/2018/4/e11473/ v5i4e11473 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Ragouzeos D, Gandrup J, Berrean B, Li J, Murphy M, Trupin L, Yazdany J, Schmajuk G. "Am I OK?" using human centered design to empower rheumatoid arthritis patients through patient reported outcomes. Patient Educ Couns. 2019 Mar;102(3):503–10. doi: 10.1016/j.pec.2018.10.016. http://europepmc.org/abstract/MED/30446358 .S0738-3991(18)30939-X [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Span M, Smits C, Jukema J, Groen-van de Ven L, Janssen R, Vernooij-Dassen M, Eefsting J, Hettinga M. An interactive web tool for facilitating shared decision-making in dementia-care networks: a field study. Front Aging Neurosci. 2015;7:128. doi: 10.3389/fnagi.2015.00128. doi: 10.3389/fnagi.2015.00128. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Ammerlaan JW, Mulder OK, de Boer-Nijhof NC, Maat B, Kruize AA, van Laar J, van Os-Medendorp H, Geenen R. Building a tailored, patient-guided, web-based self-management intervention 'ReumaUitgedaagd!' for adults with a rheumatic disease: results of a usability study and design for a randomized control trail. JMIR Res Protoc. 2016 Jun 23;5(2):e113. doi: 10.2196/resprot.5735. https://www.researchprotocols.org/2016/2/e113/ v5i2e113 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Corbett T, Walsh JC, Groarke A, Moss-Morris R, Morrissey E, McGuire BE. Cancer-related fatigue in post-treatment cancer survivors: theory-based development of a web-based intervention. JMIR Cancer. 2017 Jul 04;3(2):e8. doi: 10.2196/cancer.6987. https://cancer.jmir.org/2017/2/e8/ v3i2e8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Limburg M, Ossebaard H, Wentzel M, van Gemert-Pijnen J. Co-creating with stakeholders: ideating eHealth applications to support antibiotic stewardship in hospitals. Proceedings of the eTELEMED 2016: The Eighth International Conference on eHealth, Telemedicine, and Social Medicine 2016; eTELEMED 2016: The Eighth International Conference on eHealth, Telemedicine, and Social Medicine 2016; Apr 24 - 28, 2016; Venice, Italy. 2016. [Google Scholar]
- 45.Timmerman JG, Tönis TM, Dekker-van Weering MG, Stuiver MM, Wouters MW, van Harten WH, Hermens HJ, Vollenbroek-Hutten MM. Co-creation of an ICT-supported cancer rehabilitation application for resected lung cancer survivors: design and evaluation. BMC Health Serv Res. 2016 Apr 27;16:155. doi: 10.1186/s12913-016-1385-7. https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-016-1385-7 .10.1186/s12913-016-1385-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.McGrath SP, McGrath ML, Bastola D. Developing a concussion assessment mHealth app for certified Athletic Trainers. AMIA Annu Symp Proc. 2017;2017:1282–91. http://europepmc.org/abstract/MED/29854197 . [PMC free article] [PubMed] [Google Scholar]
- 47.Stuij SM, Labrie NH, van Dulmen S, Kersten MJ, Christoph N, Hulsman RL, Smets E, INSTRUCT project group Developing a digital communication training tool on information-provision in oncology: uncovering learning needs and training preferences. BMC Med Educ. 2018 Sep 24;18(1):220. doi: 10.1186/s12909-018-1308-x. https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-018-1308-x .10.1186/s12909-018-1308-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Span M, Hettinga M, Smits C, Groen-van de Ven L, Jukema J, Vernooij-Dassen M, Eefsting J. Developing a supportive tool to facilitate shared decision making in dementia. Proceedings of the The Sixth International Conference on eHealth, Telemedicine, and Social Medicine; The Sixth International Conference on eHealth, Telemedicine, and Social Medicine; Mar 23-27, 2014; Barcelona, Spain. 2014. [Google Scholar]
- 49.Pearson J, Walsh N, Carter D, Koskela S, Hurley M. Developing a web-based version of an exercise-based rehabilitation program for people with chronic knee and hip pain: a mixed methods study. JMIR Res Protoc. 2016 May 19;5(2):e67. doi: 10.2196/resprot.5446. https://www.researchprotocols.org/2016/2/e67/ v5i2e67 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Fledderus M, Schreurs KM, Bohlmeijer ET, Vollenbroek-Hutten MM. Development and pilot evaluation of an online relapse-prevention program based on acceptance and commitment therapy for chronic pain patients. JMIR Hum Factors. 2015 Jan 05;2(1):e1. doi: 10.2196/humanfactors.3302. https://humanfactors.jmir.org/2015/1/e1/ v2i1e1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Kloek C, van Tilburg M, Staal J, Veenhof C, Bossen D. Development and proof of concept of a blended physiotherapeutic intervention for patients with non-specific low back pain. Physiotherapy. 2019 Dec;105(4):483–91. doi: 10.1016/j.physio.2018.12.006. https://linkinghub.elsevier.com/retrieve/pii/S0031-9406(19)30005-7 .S0031-9406(19)30005-7 [DOI] [PubMed] [Google Scholar]
- 52.Wachtler C, Coe A, Davidson S, Fletcher S, Mendoza A, Sterling L, Gunn J. Development of a Mobile Clinical Prediction Tool to Estimate Future Depression Severity and Guide Treatment in Primary Care: User-Centered Design. JMIR Mhealth Uhealth. 2018 Apr 23;6(4):e95. doi: 10.2196/mhealth.9502. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Maguire R, Ream E, Richardson A, Connaghan J, Johnston B, Kotronoulas G, Pedersen V, McPhelim J, Pattison N, Smith A, Webster L, Taylor A, Kearney N. Development of a novel remote patient monitoring system: the advanced symptom management system for radiotherapy to improve the symptom experience of patients with lung cancer receiving radiotherapy. Cancer Nurs. 2015;38(2):E37–47. doi: 10.1097/NCC.0000000000000150. [DOI] [PubMed] [Google Scholar]
- 54.Al-Itejawi HH, van Uden-Kraan CF, Vis AN, Nieuwenhuijzen JA, Hofstee MJ, van Moorselaar RJ, Verdonck-de Leeuw IM. Development of a patient decision aid for the treatment of localised prostate cancer: a participatory design approach. J Clin Nurs. 2016 Apr;25(7-8):1131–44. doi: 10.1111/jocn.13120. [DOI] [PubMed] [Google Scholar]
- 55.Reblin M, Wu Y, Pok J, Kane L, Colman H, Cohen AL, Mendivil E, Warner EL, Meyer M, Agutter J. Development of the electronic social network assessment program using the center for eHealth and wellbeing research roadmap. JMIR Hum Factors. 2017 Aug 30;4(3):e23. doi: 10.2196/humanfactors.7845. https://humanfactors.jmir.org/2017/3/e23/ v4i3e23 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Gilbert M, Haag D, Hottes TS, Bondyra M, Elliot E, Chabot C, Farrell J, Bonnell A, Kopp S, Andruschak J, Shoveller J, Ogilvie G. Get checked… where? The development of a comprehensive, integrated internet-based testing program for sexually transmitted and blood-borne infections in British Columbia, Canada. JMIR Res Protoc. 2016 Sep 20;5(3):e186. doi: 10.2196/resprot.6293. https://www.researchprotocols.org/2016/3/e186/ v5i3e186 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Wentink M, van Bodegom-Vos L, Brouns B, Arwert H, Houdijk S, Kewalbansing P, Boyce L, Vliet Vlieland T, de Kloet A, Meesters J. How to improve eRehabilitation programs in stroke care? A focus group study to identify requirements of end-users. BMC Med Inform Decis Mak. 2019 Jul 26;19(1):145. doi: 10.1186/s12911-019-0871-3. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-019-0871-3 .10.1186/s12911-019-0871-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Skjøth MM, Hansen HP, Draborg E, Pedersen CD, Lamont RF, Jørgensen JS. Informed choice for participation in down syndrome screening: development and content of a web-based decision aid. JMIR Res Protoc. 2015 Sep 21;4(3):e113. doi: 10.2196/resprot.4291. https://www.researchprotocols.org/2015/3/e113/ v4i3e113 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Askedal K, Flak L. Stakeholder contradictions in early stages of eHealth efforts. Proceedings of the 50th Hawaii International Conference on System Sciences; 50th Hawaii International Conference on System Sciences; 2017; Hawaii. 2017. [DOI] [Google Scholar]
- 60.Galpin A, Meredith J, Ure C, Robinson L. "Thanks for letting us all share your mammogram experience virtually": developing a web-based hub for breast cancer screening. JMIR Cancer. 2017 Oct 27;3(2):e17. doi: 10.2196/cancer.8150. https://cancer.jmir.org/2017/2/e17/ v3i2e17 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Span M, Smits C, Groen-van de Ven L, Jukema J, Cremers A, Vernooij-Dassen M, Eefsting J, Hettinga M. Towards an interactive web tool that supports shared decision making in dementia: identifying user requirements. Int J Adv Life Sci. 2014;6(3&4):338–349. [Google Scholar]
- 62.Fico G, Teresa AM. Use of an holistic approach for effective adoption of User-Centred-Design techniques in diabetes disease management: experiences in user need elicitation. Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); Aug 25-29, 2015; Milan, Italy. 2015. [DOI] [PubMed] [Google Scholar]
- 63.Fico G, Hernandez L, Cancela J, Arredondo M. User Centered Design to incorporate predictive models for Type 2 Diabetes screening and management into professional decision support tools: preliminary results. Proceedings of the World Congress on Medical Physics and Biomedical Engineering; World Congress on Medical Physics and Biomedical Engineering; Jun 7-12, 2015; Toronto, Canada. 2015. pp. 1389–92. [DOI] [Google Scholar]
- 64.Fico G, Hernanzez L, Cancela J, Dagliati A, Sacchi L, Martinez-Millana A, Posada J, Manero L, Verdú J, Facchinetti A, Ottaviano M, Zarkogianni K, Nikita K, Groop L, Gabriel-Sanchez R, Chiovato L, Traver V, Merino-Torres JF, Cobelli C, Bellazzi R, Arredondo MT. What do healthcare professionals need to turn risk models for type 2 diabetes into usable computerized clinical decision support systems? Lessons learned from the MOSAIC project. BMC Med Inform Decis Mak. 2019 Aug 16;19(1):163. doi: 10.1186/s12911-019-0887-8. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-019-0887-8 .10.1186/s12911-019-0887-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Sjölinder M, Ehn M, Boman I, Folke M, Hansson P, Sommerfeld D, Nylander S, Borg J. Human-Computer Interaction. Cham: Springer; 2014. A multi-disciplinary approach in the development of a stroke rehabilitation tool. [Google Scholar]
- 66.Ledel Solem IK, Varsi C, Eide H, Kristjansdottir OB, Børøsund E, Schreurs KM, Waxenberg LB, Weiss KE, Morrison EJ, Haaland-Øverby M, Bevan K, Zangi HA, Stubhaug A, Solberg Nes L. A user-centered approach to an evidence-based electronic health pain management intervention for people with chronic pain: design and development of EPIO. J Med Internet Res. 2020 Jan 21;22(1):e15889. doi: 10.2196/15889. https://www.jmir.org/2020/1/e15889/ v22i1e15889 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67.Imhemed S, Kamaruddin A, Hasan S. Designing a Mobile App to support a Healthy Lifestyle (MAS-HeaL) for type-2 diabetes patients in Malaysia. J Eng Applied Sci. 2017;12(3):559–63. [Google Scholar]
- 68.Tieman JJ, Morgan DD, Swetenham K, To TH, Currow DC. Designing clinically valuable telehealth resources: processes to develop a community-based palliative care prototype. JMIR Res Protoc. 2014 Sep 04;3(3):e41. doi: 10.2196/resprot.3266. https://www.researchprotocols.org/2014/3/e41/ v3i3e41 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69.Mummah SA, King AC, Gardner CD, Sutton S. Iterative development of Vegethon: a theory-based mobile app intervention to increase vegetable consumption. Int J Behav Nutr Phys Act. 2016 Aug 08;13:90. doi: 10.1186/s12966-016-0400-z. https://ijbnpa.biomedcentral.com/articles/10.1186/s12966-016-0400-z .10.1186/s12966-016-0400-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 70.Derks YP, Visser TD, Bohlmeijer ET, Noordzij ML. mHealth in Mental Health: how to efficiently and scientifically create an ambulatory biofeedback e-coaching app for patients with borderline personality disorder. Int J Human Factor Ergonom. 2017;5(1):61–92. doi: 10.1504/ijhfe.2017.088418. [DOI] [Google Scholar]
- 71.Cheung KL, Hiligsmann M, Präger M, Jones T, Józwiak-Hagymásy J, Muñoz C, Lester-George A, Pokhrel S, López-Nicolás A, Trapero-Bertran M, Evers SM, de Vries H. Optimizing usability of an economic decision support tool: prototype of the equipt tool. Int J Technol Assess Health Care. 2018 Jan;34(1):68–77. doi: 10.1017/S0266462317004470.S0266462317004470 [DOI] [PubMed] [Google Scholar]
- 72.Yap MB, Lawrence KA, Rapee RM, Cardamone-Breen MC, Green J, Jorm AF. Partners in parenting: a multi-level web-based approach to support parents in prevention and early intervention for adolescent depression and anxiety. JMIR Ment Health. 2017 Dec 19;4(4):e59. doi: 10.2196/mental.8492. https://mental.jmir.org/2017/4/e59/ v4i4e59 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 73.Dick S, Heavin C, Vasiliou V. The design of a digital behaviour change intervention for third-level student illicit substance use: a persona building approach. Proceedings of the 53rd Hawaii International Conference on System Sciences; 53rd Hawaii International Conference on System Sciences; Jan 7-10, 2020; Hawaii. 2020. [DOI] [Google Scholar]
- 74.De Backere F, Ongenae F, Vannieuwenborg F, Van Ooteghem J, Duysburgh P, Jansen A, Hoebeke J, Wuyts K, Rossey J, Van den Abeele F, Willems K, Decancq J, Annema JH, Sulmon N, Van Landuyt D, Verstichel S, Crombez P, Ackaert A, De Grooff D, Jacobs A, De Turck F. The OCareCloudS project: toward organizing care through trusted cloud services. Inform Health Soc Care. 2016;41(2):159–76. doi: 10.3109/17538157.2014.965306. [DOI] [PubMed] [Google Scholar]
- 75.Vermeulen J, Neyens J, Spreeuwenberg MD, van Rossum E, Sipers W, Habets H, Hewson D, de Witte LP. User-centered development and testing of a monitoring system that provides feedback regarding physical functioning to elderly people. Patient Prefer Adherence. 2013;7:843–54. doi: 10.2147/PPA.S45897. doi: 10.2147/PPA.S45897.ppa-7-843 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 76.Kuosmanen T, Fleming TM, Barry MM. Using computerized mental health programs in alternative education: understanding the requirements of students and staff. Health Commun. 2018 Jun;33(6):753–61. doi: 10.1080/10410236.2017.1309620. [DOI] [PubMed] [Google Scholar]
- 77.Verhoeven F, van Gemert-Pijnen J. HCI in Work and Learning, Life and Leisure. Berlin, Heidelberg: Springer; 2010. Discount user-centered e-health design: a quick-but-not-dirty method. [Google Scholar]
- 78.Amann J, Fiordelli M, Scheel-Sailer A, Brach M, Rubinelli S. Opportunities and challenges of a self-management app to support people with spinal cord injury in the prevention of pressure injuries: qualitative study. JMIR Mhealth Uhealth. 2020 Dec 09;8(12):e22452. doi: 10.2196/22452. https://mhealth.jmir.org/2020/12/e22452/ v8i12e22452 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 79.Asbjørnsen RA, Wentzel J, Smedsrød ML, Hjelmesæth J, Clark MM, Solberg Nes L, Van Gemert-Pijnen J. Identifying persuasive design principles and behavior change techniques supporting end user values and needs in eHealth interventions for long-term weight loss maintenance: qualitative study. J Med Internet Res. 2020 Nov 30;22(11):e22598. doi: 10.2196/22598. https://www.jmir.org/2020/11/e22598/ v22i11e22598 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 80.Bogza L, Patry-Lebeau C, Farmanova E, Witteman HO, Elliott J, Stolee P, Hudon C, Giguere AM. User-centered design and evaluation of a web-based decision aid for older adults living with mild cognitive impairment and their health care providers: mixed methods study. J Med Internet Res. 2020 Aug 19;22(8):e17406. doi: 10.2196/17406. https://www.jmir.org/2020/8/e17406/ v22i8e17406 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 81.Breeman LD, Keesman M, Atsma DE, Chavannes NH, Janssen V, van Gemert-Pijnen L, Kemps H, Kraaij W, Rauwers F, Reijnders T, Scholte Op Reimer W, Wentzel J, Kraaijenhagen RA, Evers AW, BENEFIT consortium A multi-stakeholder approach to eHealth development: promoting sustained healthy living among cardiovascular patients. Int J Med Inform. 2021 Mar;147:104364. doi: 10.1016/j.ijmedinf.2020.104364. https://linkinghub.elsevier.com/retrieve/pii/S1386-5056(20)31900-6 .S1386-5056(20)31900-6 [DOI] [PubMed] [Google Scholar]
- 82.D'Addario M, Baretta D, Zanatta F, Greco A, Steca P. Engagement features in physical activity smartphone apps: focus group study with sedentary people. JMIR Mhealth Uhealth. 2020 Nov 16;8(11):e20460. doi: 10.2196/20460. https://mhealth.jmir.org/2020/11/e20460/ v8i11e20460 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 83.Jonathan GK, Dopke CA, Michaels T, Bank A, Martin CR, Adhikari K, Krakauer RL, Ryan C, McBride A, Babington P, Frauenhofer E, Silver J, Capra C, Simon M, Begale M, Mohr DC, Goulding EH. A smartphone-based self-management intervention for bipolar disorder (LiveWell): user-centered development approach. JMIR Ment Health. 2021 Apr 12;8(4):e20424. doi: 10.2196/20424. https://mental.jmir.org/2021/4/e20424/ v8i4e20424 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 84.Korpershoek YJ, Hermsen S, Schoonhoven L, Schuurmans MJ, Trappenburg JC. User-centered design of a mobile health intervention to enhance exacerbation-related self-management in patients with chronic obstructive pulmonary disease (Copilot): mixed methods study. J Med Internet Res. 2020 Jun 15;22(6):e15449. doi: 10.2196/15449. https://www.jmir.org/2020/6/e15449/ v22i6e15449 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 85.Liu LH, Garrett SB, Li J, Ragouzeos D, Berrean B, Dohan D, Katz PP, Barton JL, Yazdany J, Schmajuk G. Patient and clinician perspectives on a patient-facing dashboard that visualizes patient reported outcomes in rheumatoid arthritis. Health Expect. 2020 Aug;23(4):846–59. doi: 10.1111/hex.13057. doi: 10.1111/hex.13057. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 86.Nicklas JM, Leiferman JA, Lockhart S, Daly KM, Bull SS, Barbour LA. Development and modification of a mobile health program to promote postpartum weight loss in women at elevated risk for cardiometabolic disease: single-arm pilot study. JMIR Form Res. 2020 Apr 09;4(4):e16151. doi: 10.2196/16151. https://formative.jmir.org/2020/4/e16151/ v4i4e16151 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 87.Oberschmidt K, Grünloh C, Tunç S, Van VL, Nijboer F. You can’t always get what you want: streamlining stakeholder interests when designing technology-supported services for Active and Assisted Living. Proceedings of the 32nd Australian Conference on Human-Computer Interaction; 32nd Australian Conference on Human-Computer Interaction; Dec 2 - 4, 2020; Sydney NSW Australia. 2020. [DOI] [Google Scholar]
- 88.Rai HK, Schneider J, Orrell M. An individual cognitive stimulation therapy app for people with dementia: development and usability study of thinkability. JMIR Aging. 2020 Nov 16;3(2):e17105. doi: 10.2196/17105. https://aging.jmir.org/2020/2/e17105/ v3i2e17105 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 89.Rai HK, Griffiths R, Yates L, Schneider J, Orrell M. Field-testing an iCST touch-screen application with people with dementia and carers: a mixed method study. Aging Ment Health. 2021 Jun;25(6):1008–18. doi: 10.1080/13607863.2020.1783515. [DOI] [PubMed] [Google Scholar]
- 90.Schmidt T, Kok R, Andersen CM, Skovbakke SJ, Ahm R, Wiil UK, Frostholm L, Pedersen SS. Development of an internet-delivered program and platform for the treatment of depression and anxiety in patients with ischemic heart disease in eMindYourHeart. Inform Health Soc Care. 2021 Jun 02;46(2):178–91. doi: 10.1080/17538157.2021.1878185. [DOI] [PubMed] [Google Scholar]
- 91.Sporrel K, De Boer RD, Wang S, Nibbeling N, Simons M, Deutekom M, Ettema D, Castro PC, Dourado VZ, Kröse B. The design and development of a personalized leisure time physical activity application based on behavior change theories, end-user perceptions, and principles from empirical data mining. Front Public Health. 2020;8:528472. doi: 10.3389/fpubh.2020.528472. doi: 10.3389/fpubh.2020.528472. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 92.Stuij SM, Drossaert CH, Labrie NH, Hulsman RL, Kersten MJ, van Dulmen S, Smets EM, INSTRUCT project group Developing a digital training tool to support oncologists in the skill of information-provision: a user centred approach. BMC Med Educ. 2020 May 01;20(1):135. doi: 10.1186/s12909-020-1985-0. https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-020-1985-0 .10.1186/s12909-020-1985-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 93.Timmer MA, Kloek CJ, de Kleijn P, Kuijlaars IA, Schutgens RE, Veenhof C, Pisters MF. A blended physiotherapy intervention for persons with hemophilic arthropathy: development study. J Med Internet Res. 2020 Jun 19;22(6):e16631. doi: 10.2196/16631. https://www.jmir.org/2020/6/e16631/ v22i6e16631 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 94.Winterling J, Wiklander M, Obol CM, Lampic C, Eriksson LE, Pelters B, Wettergren L. Development of a self-help web-based intervention targeting young cancer patients with sexual problems and fertility distress in collaboration with patient research partners. JMIR Res Protoc. 2016 Apr 12;5(2):e60. doi: 10.2196/resprot.5499. https://www.researchprotocols.org/2016/2/e60/ v5i2e60 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 95.Carr EC, Babione JN, Marshall D. Translating research into practice through user-centered design: an application for osteoarthritis healthcare planning. Int J Med Inform. 2017 Aug;104:31–7. doi: 10.1016/j.ijmedinf.2017.05.007.S1386-5056(17)30110-7 [DOI] [PubMed] [Google Scholar]
- 96.Kip H, Kelders SM, Weerink K, Kuiper A, Brüninghoff I, Bouman YH, Dijkslag D, van Gemert-Pijnen J. Identifying the added value of virtual reality for treatment in forensic mental health: a scenario-based, qualitative approach. Front Psychol. 2019;10:406. doi: 10.3389/fpsyg.2019.00406. doi: 10.3389/fpsyg.2019.00406. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 97.Kip H, Kelders S, Van Gemert-Pijnen J. Putting the value in VR: how to systematically and iteratively develop a value-based vr application with a complex target group. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems; Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems; May 4 - 9, 2019; Glasgow Scotland UK. 2019. [DOI] [Google Scholar]
- 98.Verhoeven F, Hendrix RM, Daniels-Haardt I, Friedrich AW, Van Gemert-Pijnen JE. The development of a web-based learning tool for prevention and control of MRSA. Hygiene Medizin. 2008 Oct;33(10):412–6. https://www.researchgate.net/publication/283137593_The_development_of_a_web-based_learning_tool_for_prevention_and_control_of_MRSA . [Google Scholar]
- 99.van Limburg AH, Van Gemert-Pijnen J. Towards innovative business modeling for sustainable eHealth applications. Proceedings of the 2010 Second International Conference on eHealth, Telemedicine, and Social Medicine; 2010 Second International Conference on eHealth, Telemedicine, and Social Medicine; Feb 10-16, 2010; Saint Maarten, Netherlands Antilles. 2010. [DOI] [Google Scholar]
- 100.Beaujean D, van Velsen L, van Gemert-Pijnen JE, Maat A, van Steenbergen JE, Crutzen R. Using risk group profiles as a lightweight qualitative approach for intervention development: an example of prevention of tick bites and lyme disease. JMIR Res Protoc. 2013 Oct 30;2(2):e45. doi: 10.2196/resprot.2760. https://www.researchprotocols.org/2013/2/e45/ v2i2e45 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 101.van der Vaart R, Witting M, Riper H, Kooistra L, Bohlmeijer ET, van Gemert-Pijnen LJ. Blending online therapy into regular face-to-face therapy for depression: content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry. 2014 Dec 14;14(1):355–10. doi: 10.1186/s12888-014-0355-z. https://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-014-0355-z .s12888-014-0355-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 102.van Gemert-Pijnen J, Karreman J, Vonderhorst S, Verhoeven F, Wentzel J. Participatory development via user-involvement - A case study about the development of a web-based patient-communication system about methicillin-resistant staphylococcus aureus. Electron J Health Informatics. 2011;6(4):e28. doi: 10.1109/etelemed.2010.31. [DOI] [Google Scholar]
- 103.Lentferink A, Polstra L, de Groot M, Oldenhuis H, Velthuijsen H, van Gemert-Pijnen J. Persuasive Technology. Cham: Springer; 2018. The values of self-tracking and persuasive eCoaching according to employees and human resource advisors for a workplace stress management application: a qualitative study. [Google Scholar]
- 104.Cruz-Martínez R, Sieverink F, Wesselink R, van Gemert-Pijnen J. Towards data-driven persuasive coaching in a heart failure telemonitoring technology. Proceedings of the Sixth International Workshop on Behavior Change Support Systems (BCSS 2018). In conjunction with the 13th International Conference on Persuasive Technology; Sixth International Workshop on Behavior Change Support Systems (BCSS 2018). In conjunction with the 13th International Conference on Persuasive Technology; Apr 16-19, 2018; University of Waterloo, Canada. 2018. [Google Scholar]
- 105.Teles S, Bertal D, Kofler A, Ruscher S, Paul C. A multi-perspective view on AAL stakeholders' needs a user-centred requirement analysis for the activeadvice European project. Proceedings of the 3rd International Conference on Information and Communication Technologies for Ageing Well and e-Health - ICT4AWE; Proceedings of the 3rd International Conference on Information and Communication Technologies for Ageing Well and e-Health - ICT4AWE; Apr 28-29, 2017; Porto, Portugal. 2017. [DOI] [Google Scholar]
- 106.Børøsund E, Mirkovic J, Clark MM, Ehlers SL, Andrykowski MA, Bergland A, Westeng M, Solberg Nes L. A stress management app intervention for cancer survivors: design, development, and usability testing. JMIR Form Res. 2018 Sep 06;2(2):e19. doi: 10.2196/formative.9954. https://formative.jmir.org/2018/2/e19/ v2i2e19 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 107.Nobakht Z, Rassafiani M, Hosseini SA. A web-based caring training for caregivers of children with cerebral palsy: development and evaluation. Iran J Child Neurol. 2018;12(4):65–84. http://europepmc.org/abstract/MED/30279710 . [PMC free article] [PubMed] [Google Scholar]
- 108.Suffoletto B, Yanta J, Kurtz R, Cochran G, Douaihy A, Chung T. Acceptability of an opioid relapse prevention text-message intervention for emergency department patients. J Addict Med. 2017;11(6):475–82. doi: 10.1097/ADM.0000000000000351. http://europepmc.org/abstract/MED/28858888 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 109.van Bruinessen IR, van Weel-Baumgarten EM, Snippe HW, Gouw H, Zijlstra JM, van Dulmen S. Active patient participation in the development of an online intervention. JMIR Res Protoc. 2014 Nov 06;3(4):e59. doi: 10.2196/resprot.3695. https://www.researchprotocols.org/2014/4/e59/ v3i4e59 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 110.Duman-Lubberding S, van Uden-Kraan CF, Peek N, Cuijpers P, Leemans CR, Verdonck-de Leeuw IM. An eHealth application in head and neck cancer survivorship care: health care professionals' perspectives. J Med Internet Res. 2015 Oct 21;17(10):e235. doi: 10.2196/jmir.4870. https://www.jmir.org/2015/10/e235/ v17i10e235 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 111.Span M, Smits C, Groen-Van de Ven LM, Cremers A, Jukema J, Vernooij-Dassen M, Hettinga M. An interactive web tool to facilitate shared decision making in dementia: design issues perceived by caregivers and patients. Int J Adv Life Sci. 2014;6(3-4):107–21. https://www.researchgate.net/publication/271012853_An_Interactive_Web_Tool_to_Facilitate_Shared_Decision_Making_in_Dementia_Design_Issues_Perceived_by_Caregivers_and_Patients . [Google Scholar]
- 112.Hochstenbach LM, Courtens AM, Zwakhalen SM, Vermeulen J, van Kleef M, de Witte LP. Co-creative development of an eHealth nursing intervention: self-management support for outpatients with cancer pain. Appl Nurs Res. 2017 Aug;36:1–8. doi: 10.1016/j.apnr.2017.03.004.S0897-1897(16)30190-2 [DOI] [PubMed] [Google Scholar]
- 113.Rothgangel A, Braun S, Smeets R, Beurskens A. Design and development of a telerehabilitation platform for patients with phantom limb pain: a user-centered approach. JMIR Rehabil Assist Technol. 2017 Feb 15;4(1):e2. doi: 10.2196/rehab.6761. https://rehab.jmir.org/2017/1/e2/ v4i1e2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 114.Walters LE, Scott RE, Mars M. Design requirements for a teledermatology scale-up framework. South African Comput J. 2018 Jul 10;30(1):128–60. doi: 10.18489/sacj.v30i1.559. [DOI] [Google Scholar]
- 115.Doty JL, Brady SS, Monardez Popelka J, Rietveld L, Garcia-Huidobro D, Doty MJ, Linares R, Svetaz MV, Allen ML. Designing a mobile app to enhance parenting skills of Latinx parents: a community-based participatory approach. JMIR Form Res. 2020 Jan 24;4(1):e12618. doi: 10.2196/12618. https://formative.jmir.org/2020/1/e12618/ v4i1e12618 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 116.Pang PC, Chang S, Verspoor K, Pearce J. Designing health websites based on users' web-based information-seeking behaviors: a mixed-method observational study. J Med Internet Res. 2016 Jun 06;18(6):e145. doi: 10.2196/jmir.5661. https://www.jmir.org/2016/6/e145/ v18i6e145 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 117.Geraghty AW, Muñoz RF, Yardley L, Mc Sharry J, Little P, Moore M. Developing an unguided internet-delivered intervention for emotional distress in primary care patients: applying common factor and person-based approaches. JMIR Ment Health. 2016 Dec 20;3(4):e53. doi: 10.2196/mental.5845. https://mental.jmir.org/2016/4/e53/ v3i4e53 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 118.Crane D, Garnett C, Brown J, West R, Michie S. Factors influencing usability of a smartphone app to reduce excessive alcohol consumption: think aloud and interview studies. Front Public Health. 2017;5:39. doi: 10.3389/fpubh.2017.00039. doi: 10.3389/fpubh.2017.00039. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 119.Lubberding S, van Uden-Kraan CF, Te Velde EA, Cuijpers P, Leemans CR, Verdonck-de Leeuw IM. Improving access to supportive cancer care through an eHealth application: a qualitative needs assessment among cancer survivors. J Clin Nurs. 2015 May;24(9-10):1367–79. doi: 10.1111/jocn.12753. [DOI] [PubMed] [Google Scholar]
- 120.Downing KL, Best K, Campbell KJ, Hesketh KD. Informing active play and screen time behaviour change interventions for low socioeconomic position mothers of young children: what do mothers want? Biomed Res Int. 2016;2016:2139782. doi: 10.1155/2016/2139782. doi: 10.1155/2016/2139782. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 121.Noordman J, Driesenaar JA, van Bruinessen IR, van Dulmen S. ListeningTime; participatory development of a web-based preparatory communication tool for elderly cancer patients and their healthcare providers. Internet Interv. 2017 Sep;9:51–6. doi: 10.1016/j.invent.2017.05.002. https://linkinghub.elsevier.com/retrieve/pii/S2214-7829(17)30028-3 .S2214-7829(17)30028-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 122.Boele FW, van Uden-Kraan CF, Hilverda K, Weimer J, Donovan HS, Drappatz J, Lieberman FS, Verdonck-de Leeuw I, Sherwood PR. Neuro-oncology family caregivers' view on keeping track of care issues using eHealth systems: it's a question of time. J Neurooncol. 2017 Aug;134(1):157–67. doi: 10.1007/s11060-017-2504-y. http://europepmc.org/abstract/MED/28550505 .10.1007/s11060-017-2504-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 123.Merolli M, Marshall CJ, Pranata A, Paay J, Sterling L. User-centered value specifications for technologies supporting chronic low-back pain management. Stud Health Technol Inform. 2019 Aug 21;264:1288–92. doi: 10.3233/SHTI190434.SHTI190434 [DOI] [PubMed] [Google Scholar]
- 124.Floch J, Zettl A, Fricke L, Weisser T, Grut L, Vilarinho T, Stav E, Ascolese A, Schauber C. User needs in the development of a health app ecosystem for self-management of cystic fibrosis: user-centered development approach. JMIR Mhealth Uhealth. 2018 May 08;6(5):e113. doi: 10.2196/mhealth.8236. https://mhealth.jmir.org/2018/5/e113/ v6i5e113 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 125.van der Horst H. Vreemde vogels. HUWE. 2005 Mar;48(3):883. doi: 10.1007/bf03084505. [DOI] [Google Scholar]
- 126.Liao G, Chien Y, Chen Y, Hsiung H, Chen H, Hsieh M, Wu W. What to build for middle-agers to come? Attractive and necessary functions of exercise-promotion mobile phone apps: a cross-sectional study. JMIR Mhealth Uhealth. 2017 May 25;5(5):e65. doi: 10.2196/mhealth.6233. https://mhealth.jmir.org/2017/5/e65/ v5i5e65 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 127.Derks YP, Klaassen R, Westerhof GJ, Bohlmeijer ET, Noordzij ML. Development of an ambulatory biofeedback app to enhance emotional awareness in patients with borderline personality disorder: multicycle usability testing study. JMIR Mhealth Uhealth. 2019 Oct 15;7(10):e13479. doi: 10.2196/13479. https://mhealth.jmir.org/2019/10/e13479/ v7i10e13479 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 128.Keizer J, Beerlage-De Jong N, Al Naiemi N, van Gemert-Pijnen J. Finding the match between healthcare worker and expert for optimal audit and feedback on antimicrobial resistance prevention measures. Antimicrob Resist Infect Control. 2020 Aug 05;9(1):125. doi: 10.1186/s13756-020-00794-7. https://aricjournal.biomedcentral.com/articles/10.1186/s13756-020-00794-7 .10.1186/s13756-020-00794-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 129.Lentferink A, Polstra L, D'Souza A, Oldenhuis H, Velthuijsen H, van Gemert-Pijnen L. Creating value with eHealth: identification of the value proposition with key stakeholders for the resilience navigator app. BMC Med Inform Decis Mak. 2020 Apr 27;20(1):76. doi: 10.1186/s12911-020-1088-1. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-020-1088-1 .10.1186/s12911-020-1088-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 130.van Velsen L, van Gemert-Pijnen J, Beaujean DJ, Wentzel J, van Steenbergen JE. Should health organizations use web 2.0 media in times of an infectious disease crisis? An in-depth qualitative study of citizens' information behavior during an EHEC outbreak. J Med Internet Res. 2012 Dec 20;14(6):e181. doi: 10.2196/jmir.2123. https://www.jmir.org/2012/6/e181/ v14i6e181 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 131.Beerlage-de Jong N, Wentzel J, Hendrix R, van Gemert-Pijnen L. The value of participatory development to support antimicrobial stewardship with a clinical decision support system. Am J Infect Control. 2017 Apr 01;45(4):365–71. doi: 10.1016/j.ajic.2016.12.001.S0196-6553(16)31092-6 [DOI] [PubMed] [Google Scholar]
- 132.Scholten MR, Kelders SM, Van Gemert-Pijnen J. An empirical study of a pedagogical agent as an adjunct to an eHealth self-management intervention: what modalities does it need to successfully support and motivate users? Front Psychol. 2019;10:1063. doi: 10.3389/fpsyg.2019.01063. doi: 10.3389/fpsyg.2019.01063. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 133.Nijland N, Cranen K, Boer H, van Gemert-Pijnen J, Seydel ER. Patient use and compliance with medical advice delivered by a web-based triage system in primary care. J Telemed Telecare. 2010;16(1):8–11. doi: 10.1258/jtt.2009.001004.16/1/8 [DOI] [PubMed] [Google Scholar]
- 134.van Velsen L, Beaujean DJ, van Gemert-Pijnen JE, van Steenbergen JE, Timen A. Public knowledge and preventive behavior during a large-scale Salmonella outbreak: results from an online survey in the Netherlands. BMC Public Health. 2014 Jan 31;14:100. doi: 10.1186/1471-2458-14-100. https://bmcpublichealth.biomedcentral.com/articles/10.1186/1471-2458-14-100 .1471-2458-14-100 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 135.van Woezik AF, Braakman-Jansen LM, Kulyk O, Siemons L, van Gemert-Pijnen JE. Tackling wicked problems in infection prevention and control: a guideline for co-creation with stakeholders. Antimicrob Resist Infect Control. 2016;5:20. doi: 10.1186/s13756-016-0119-2. https://aricjournal.biomedcentral.com/articles/10.1186/s13756-016-0119-2 .119 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 136.Bernecker SL, Banschback K, Santorelli GD, Constantino MJ. A web-disseminated self-help and peer support program could fill gaps in mental health care: lessons from a consumer survey. JMIR Ment Health. 2017 Jan 19;4(1):e5. doi: 10.2196/mental.4751. https://mental.jmir.org/2017/1/e5/ v4i1e5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 137.Mirk SM, Wegrzyn NM. Apps for health-related education in pharmacy practice: needs assessment survey among patients within a large metropolitan area. JMIR Res Protoc. 2017 Jul 19;6(7):e133. doi: 10.2196/resprot.5886. https://www.researchprotocols.org/2017/7/e133/ v6i7e133 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 138.Jansen F, van Uden-Kraan CF, van Zwieten V, Witte BI, Verdonck-de Leeuw IM. Cancer survivors' perceived need for supportive care and their attitude towards self-management and eHealth. Support Care Cancer. 2015 Jun 26;23(6):1679–88. doi: 10.1007/s00520-014-2514-7. [DOI] [PubMed] [Google Scholar]
- 139.Whiteside U, Lungu A, Richards J, Simon GE, Clingan S, Siler J, Snyder L, Ludman E. Designing messaging to engage patients in an online suicide prevention intervention: survey results from patients with current suicidal ideation. J Med Internet Res. 2014 Feb 07;16(2):e42. doi: 10.2196/jmir.3173. https://www.jmir.org/2014/2/e42/ v16i2e42 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 140.Noordman J, Driesenaar JA, van Bruinessen IR, Portielje JE, van Dulmen S. Evaluation and implementation of ListeningTime: a web-based preparatory communication tool for elderly patients with cancer and their health care providers. JMIR Cancer. 2019 Jan 30;5(1):e11556. doi: 10.2196/11556. https://cancer.jmir.org/2019/1/e11556/ v5i1e11556 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 141.van Uden-Kraan CF, Jansen F, Lissenberg-Witte BI, Eerenstein SE, Leemans CR, Verdonck-de Leeuw IM. Health-related and cancer-related Internet use by patients treated with total laryngectomy. Support Care Cancer. 2020 Jan;28(1):131–40. doi: 10.1007/s00520-019-04757-6. http://europepmc.org/abstract/MED/30993449 .10.1007/s00520-019-04757-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 142.Da Costa D, Zelkowitz P, Letourneau N, Howlett A, Dennis C, Russell B, Grover S, Lowensteyn I, Chan P, Khalifé S. HealthyDads.ca: what do men want in a website designed to promote emotional wellness and healthy behaviors during the transition to parenthood? J Med Internet Res. 2017 Oct 11;19(10):e325. doi: 10.2196/jmir.7415. https://www.jmir.org/2017/10/e325/ v19i10e325 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 143.Grunberg PH, Dennis C, Da Costa D, Zelkowitz P. Infertility patients' need and preferences for online peer support. Reprod Biomed Soc Online. 2018 Aug;6:80–9. doi: 10.1016/j.rbms.2018.10.016. https://linkinghub.elsevier.com/retrieve/pii/S2405-6618(18)30038-8 .S2405-6618(18)30038-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 144.Van de Belt TH, Engelen LJ, Berben SA, Teerenstra S, Samsom M, Schoonhoven L. Internet and social media for health-related information and communication in health care: preferences of the Dutch general population. J Med Internet Res. 2013 Oct 02;15(10):e220. doi: 10.2196/jmir.2607. https://www.jmir.org/2013/10/e220/ v15i10e220 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 145.Leafman JS, Mathieson K. Perceptions of telemedicine for patient education among online support group patients with chronic or rare conditions. Cogent Medicine. 2018 Oct 12;5(1):1525148. doi: 10.1080/2331205x.2018.1525148. [DOI] [Google Scholar]
- 146.Duplaga M. Searching for a role of nursing personnel in developing landscape of eHealth: factors determining attitudes toward key patient empowering applications. PLoS One. 2016;11(4):e0153173. doi: 10.1371/journal.pone.0153173. https://dx.plos.org/10.1371/journal.pone.0153173 .PONE-D-15-41054 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 147.von Rosen AJ, von Rosen FT, Tinnemann P, Müller-Riemenschneider F. Sexual health and the internet: cross-sectional study of online preferences among adolescents. J Med Internet Res. 2017 Nov 08;19(11):e379. doi: 10.2196/jmir.7068. https://www.jmir.org/2017/11/e379/ v19i11e379 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 148.Kristan J, Suffoletto B. Using online crowdsourcing to understand young adult attitudes toward expert-authored messages aimed at reducing hazardous alcohol consumption and to collect peer-authored messages. Transl Behav Med. 2015 Mar;5(1):45–52. doi: 10.1007/s13142-014-0298-4. http://europepmc.org/abstract/MED/25729452 .298 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 149.Wentink MM, VAN Bodegom-Vos L, Brouns B, Arwert HJ, Vlieland TP, DE Kloet AJ, Meesters JJ. What is important in e-health interventions for stroke rehabilitation? A survey study among patients, informal caregivers and health professionals. Int J Telerehabil. 2018;10(1):15–28. doi: 10.5195/ijt.2018.6247. http://europepmc.org/abstract/MED/30147840 .ijt-10-15 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 150.Hughes CM, Padilla A, Hintze A, Raymundo TM, Sera M, Weidner S, Ontiveros J, Peng T, Encarcion A, Cruz ZA, Warner S, Sany K, Baye M. Developing an mHealth app for post-stroke upper limb rehabilitation: feedback from US and Ethiopian rehabilitation clinicians. Health Informatics J. 2020 Jun;26(2):1104–17. doi: 10.1177/1460458219868356. https://journals.sagepub.com/doi/10.1177/1460458219868356?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub%3dpubmed . [DOI] [PubMed] [Google Scholar]
- 151.Ihmhemed SM, Kamaruddin RA, Atan R. Using empathy approach to design type-2 diabetic user persona. J Advanced Sci Eng Res. 2017;7(1):27–42. [Google Scholar]
- 152.Verhoeven F, Steehouder MF, Hendrix RM, Van Gemert-Pijnen JE. From expert-driven to user-oriented communication of infection control guidelines. Int J Human Comput Stud. 2010 Jun;68(6):328–43. doi: 10.1016/j.ijhcs.2009.07.003. [DOI] [Google Scholar]
- 153.Lambert JD, Greaves CJ, Farrand P, Haase AM, Taylor AH. Development of a web-based intervention (eMotion) based on behavioural activation to promote physical activity in people with depression. Mental Health Physical Activity. 2017 Oct;13:120–36. doi: 10.1016/j.mhpa.2017.10.003. [DOI] [Google Scholar]
- 154.Gude WT, Simon AC, Peute LW, Holleman F, Hoekstra JB, Peek N, Jaspers MW. Formative usability evaluation of a web-based insulin self-titration system: preliminary results. Stud Health Technol Inform. 2012;180:1209–11. [PubMed] [Google Scholar]
- 155.King S, Boutilier JA, MacLaren Chorney J. Managing chronic pain in the classroom: development and usability testing of an eHealth educational intervention for educators. Canadian J School Psychol. 2016 Oct 23;33(2):95–109. doi: 10.1177/0829573516674308. [DOI] [Google Scholar]
- 156.Verhoeven F, Steehouder M, Hendrix R, van Gemert-Pijnen JE. How nurses seek and evaluate clinical guidelines on the internet. J Adv Nurs. 2010 Jan;66(1):114–27. doi: 10.1111/j.1365-2648.2009.05175.x.JAN5175 [DOI] [PubMed] [Google Scholar]
- 157.Scholten M, Kelders SM, van Gemert-Pijnen J. Persuasive Technology: Development and Implementation of Personalized Technologies to Change Attitudes and Behaviors. Cham: Springer; 2017. A scoped review of the potential for supportive virtual coaches as adjuncts to self-guided web-based interventions. [Google Scholar]
- 158.Kip H, Bouman YH, Kelders SM, van Gemert-Pijnen JE. eHealth in treatment of offenders in forensic mental health: a review of the current state. Front Psychiatry. 2018;9:42. doi: 10.3389/fpsyt.2018.00042. doi: 10.3389/fpsyt.2018.00042. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 159.Asbjørnsen RA, Smedsrød ML, Solberg Nes L, Wentzel J, Varsi C, Hjelmesæth J, van Gemert-Pijnen JE. Persuasive system design principles and behavior change techniques to stimulate motivation and adherence in electronic health interventions to support weight loss maintenance: scoping review. J Med Internet Res. 2019 Jun 21;21(6):e14265. doi: 10.2196/14265. https://www.jmir.org/2019/6/e14265/ v21i6e14265 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 160.Scholten MR, Kelders SM, Van Gemert-Pijnen JE. Self-guided web-based interventions: scoping review on user needs and the potential of embodied conversational agents to address them. J Med Internet Res. 2017 Nov 16;19(11):e383. doi: 10.2196/jmir.7351. https://www.jmir.org/2017/11/e383/ v19i11e383 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 161.Verhoeven F, Tanja-Dijkstra K, Nijland N, Eysenbach G, van Gemert-Pijnen L. Asynchronous and synchronous teleconsultation for diabetes care: a systematic literature review. J Diabetes Sci Technol. 2010 May 01;4(3):666–84. doi: 10.1177/193229681000400323. http://europepmc.org/abstract/MED/20513335 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 162.Lentferink AJ, Oldenhuis HK, de Groot M, Polstra L, Velthuijsen H, van Gemert-Pijnen JE. Key components in eHealth interventions combining self-tracking and persuasive eCoaching to promote a healthier lifestyle: a scoping review. J Med Internet Res. 2017 Aug 01;19(8):e277. doi: 10.2196/jmir.7288. https://www.jmir.org/2017/8/e277/ v19i8e277 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 163.Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res. 2012 Nov 14;14(6):e152. doi: 10.2196/jmir.2104. https://www.jmir.org/2012/6/e152/ v14i6e152 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 164.Verhoeven F, van Gemert-Pijnen JE, Dijkstra K, Nijland N, Seydel E, Steehouder M. The contribution of teleconsultation and videoconferencing to diabetes care: a systematic literature review. J Med Internet Res. 2007 Dec 14;9(5):e37. doi: 10.2196/jmir.9.5.e37. https://www.jmir.org/2007/5/e37/ v9i5e37 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 165.Walters LE, Mars M, Scott RE. A review and critique of teledermatology in the South African public health sector. Stud Health Technol Inform. 2016;231:143–51. [PubMed] [Google Scholar]
- 166.Organ D, Dick S, Hurley C, Heavin C, Linehan C, Dockray S, Davoren M, Bryne M. A systematic review of user-centred design practices in illicit substance use interventions for higher education students. Proceedings of the European Conference on Information Systems 2018: Beyond Digitization-Facets of Socio-Technical Change; European Conference on Information Systems 2018: Beyond Digitization-Facets of Socio-Technical Change; Jun 23 - 28, 2018; Portsmouth, UK. 2018. [Google Scholar]
- 167.Radomski A, Wozney L, McGrath P, Huguet A, Hartling L, Dyson M, Bennett K, Newton AS. Design and delivery features that may improve the use of internet-based cognitive behavioral therapy for children and adolescents with anxiety: a realist literature synthesis with a persuasive systems design perspective. J Med Internet Res. 2019 Feb 05;21(2):e11128. doi: 10.2196/11128. https://www.jmir.org/2019/2/e11128/ v21i2e11128 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 168.Norris A, Martinez S, Labaka L, Madanian S. Disaster e-health: a new paradigm for collaborative healthcare in disasters. Proceedings of the ISCRAM 2015 - Information Systems for Crisis Response and Management; ISCRAM 2015 - Information Systems for Crisis Response and Management; May 24-27, 2015; Krystiansand, Norway. 2015. [DOI] [Google Scholar]
- 169.Meiland F, Innes A, Mountain G, Robinson L, van der Roest H, García-Casal JA, Gove D, Thyrian JR, Evans S, Dröes R-M, Kelly F, Kurz A, Casey D, Szcześniak D, Dening T, Craven MP, Span M, Felzmann H, Tsolaki M, Franco-Martin M. Technologies to support community-dwelling persons with dementia: a position paper on issues regarding development, usability, effectiveness and cost-effectiveness, deployment, and ethics. JMIR Rehabil Assist Technol. 2017 Jan 16;4(1):e1. doi: 10.2196/rehab.6376. https://rehab.jmir.org/2017/1/e1/ v4i1e1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 170.Spanakis EG, Santana S, Tsiknakis M, Marias K, Sakkalis V, Teixeira A, Janssen JH, de Jong H, Tziraki C. Technology-based innovations to foster personalized healthy lifestyles and well-being: a targeted review. J Med Internet Res. 2016 Jun 24;18(6):e128. doi: 10.2196/jmir.4863. https://www.jmir.org/2016/6/e128/ v18i6e128 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 171.Wahle F, Bollhalder L, Kowatsch T, Fleisch E. Toward the design of evidence-based mental health information systems for people with depression: a systematic literature review and meta-analysis. J Med Internet Res. 2017 May 31;19(5):e191. doi: 10.2196/jmir.7381. https://www.jmir.org/2017/5/e191/ v19i5e191 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 172.Kramer LL, Ter Stal S, Mulder BC, de Vet E, van Velsen L. Developing embodied conversational agents for coaching people in a healthy lifestyle: scoping review. J Med Internet Res. 2020 Feb 06;22(2):e14058. doi: 10.2196/14058. https://www.jmir.org/2020/2/e14058/ v22i2e14058 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 173.Stara V, Santini S, Kropf J, D'Amen B. Digital health coaching programs among older employees in transition to retirement: systematic literature review. J Med Internet Res. 2020 Sep 24;22(9):e17809. doi: 10.2196/17809. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 174.Wentzel M, Karreman J, van GJ, van VanGemert-Pijne JE, Ossebaard H, Hamalainen P. Towards an internet-based infectious disease management platform to increase patient safety. Proceedings of the Third International Conference on eHealth, Telemedicine, and Social Medicine; Third International Conference on eHealth, Telemedicine, and Social Medicine; Feb 23-28, 2011; Guadeloupe, France. 2011. [Google Scholar]
- 175.Van Gemert-Pijnen JE, Kelders SM, Bohlmeijer ET. Understanding the usage of content in a mental health intervention for depression: an analysis of log data. J Med Internet Res. 2014 Jan 31;16(1):e27. doi: 10.2196/jmir.2991. https://www.jmir.org/2014/1/e27/ v16i1e27 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 176.Sieverink F, Kelders S, Poel M, van Gemert-Pijnen L. Opening the black box of electronic health: collecting, analyzing, and interpreting log data. JMIR Res Protoc. 2017 Aug 07;6(8):e156. doi: 10.2196/resprot.6452. https://www.researchprotocols.org/2017/8/e156/ v6i8e156 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 177.Sewitch MJ, Fallone CA, Ghali P, Lee GE. What patients want in a smartphone app that supports colonoscopy preparation: qualitative study to inform a user-centered smartphone app. JMIR Mhealth Uhealth. 2019 Jul 02;7(7):e12242. doi: 10.2196/12242. https://mhealth.jmir.org/2019/7/e12242/ v7i7e12242 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 178.Keizer J, Braakman-Jansen LM, Kampmeier S, Köck R, Al Naiemi N, Te Riet-Warning R, Jong NB, Becker K, Van Gemert-Pijnen J. Correction to: cross-border comparison of antimicrobial resistance (AMR) and AMR prevention measures: the healthcare workers' perspective. Antimicrob Resist Infect Control. 2019;8:133. doi: 10.1186/s13756-019-0589-0. https://aricjournal.biomedcentral.com/articles/10.1186/s13756-019-0589-0 .589 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 179.Van Velsen L, Wentzel J, Van Gemert-Pijnen JE. Designing eHealth that matters via a multidisciplinary requirements development approach. JMIR Res Protoc. 2013 Jun 24;2(1):e21. doi: 10.2196/resprot.2547. https://www.researchprotocols.org/2013/1/e21/ v2i1e21 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 180.LeRouge C, Ma J, Sneha S, Tolle K. User profiles and personas in the design and development of consumer health technologies. Int J Med Inform. 2013 Nov;82(11):e251–68. doi: 10.1016/j.ijmedinf.2011.03.006.S1386-5056(11)00072-4 [DOI] [PubMed] [Google Scholar]
- 181.Tschimmel K. Design thinking as an effective toolkit for innovation. Proceedings of the XXIII ISPIM Conference: Action for Innovation: Innovating from Experience; XXIII ISPIM Conference: Action for Innovation: Innovating from Experience; Jun 17-20, 2012; Barcelona, Spain. 2012. [Google Scholar]
- 182.Kip H. The added value of eHealth: improving the development, implementation and evaluation of technology in treatment of offenders. University of Twente. [2022-01-20]. https://research.utwente.nl/en/publications/the-added-value-of-ehealth-improving-the-development-implementati .
- 183.Patrick K, Hekler EB, Estrin D, Mohr DC, Riper H, Crane D, Godino J, Riley WT. The Pace of Technologic Change. American Journal of Preventive Medicine. 2016 Nov;51(5):816–824. doi: 10.1016/j.amepre.2016.05.001. [DOI] [PubMed] [Google Scholar]
- 184.Morrison LG. Theory-based strategies for enhancing the impact and usage of digital health behaviour change interventions: a review. Digit Health. 2015 Jul 17;1:2055207615595335. doi: 10.1177/2055207615595335. https://journals.sagepub.com/doi/10.1177/2055207615595335?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub%3dpubmed .10.1177_2055207615595335 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 185.Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res. 2010 Feb 17;12(1):e4. doi: 10.2196/jmir.1376. https://www.jmir.org/2010/1/e4/ v12i1e4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 186.Moller AC, Merchant G, Conroy DE, West R, Hekler E, Kugler KC, Michie S. Applying and advancing behavior change theories and techniques in the context of a digital health revolution: proposals for more effectively realizing untapped potential. J Behav Med. 2017 Feb;40(1):85–98. doi: 10.1007/s10865-016-9818-7. http://europepmc.org/abstract/MED/28058516 .10.1007/s10865-016-9818-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 187.Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, Eccles MP, Cane J, Wood CE. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013 Aug;46(1):81–95. doi: 10.1007/s12160-013-9486-6. [DOI] [PubMed] [Google Scholar]
- 188.Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl Behav Med. 2011 Mar;1(1):53–71. doi: 10.1007/s13142-011-0021-7. http://europepmc.org/abstract/MED/21796270 .21 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 189.Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform. 2019 Jun;126:95–104. doi: 10.1016/j.ijmedinf.2019.03.018.S1386-5056(18)31318-2 [DOI] [PubMed] [Google Scholar]
- 190.Lorenz P, Schindler L, Steudte-Schmiedgen S, Weidner K, Kirschbaum C, Schellong J. Ecological momentary assessment in posttraumatic stress disorder and coping. An eHealth study protocol. Eur J Psychotraumatol. 2019;10(1):1654064. doi: 10.1080/20008198.2019.1654064. http://europepmc.org/abstract/MED/31528269 .1654064 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 191.Kip H, Beerlage-de JN, Wentzel J. eHealth Research, Theory and Development: A Multi-Disciplinary Approach. London: Routledge; 2018. The contextual inquiry. [Google Scholar]
- 192.Schulz K, Altman D, Moher D, CONSORT Group CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Trials. 2010 Mar 24;11:32. doi: 10.1186/1745-6215-11-32. https://trialsjournal.biomedcentral.com/articles/10.1186/1745-6215-11-32 .1745-6215-11-32 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 193.Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009 Jul 21;6(7):e1000097. doi: 10.1371/journal.pmed.1000097. https://dx.plos.org/10.1371/journal.pmed.1000097 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 194.Guidelines for Reporting Health Research: A User's Manual. Hoboken, New Jersey, United States: Wiley; 2014. COREQ (Consolidated Criteria for Reporting Qualitative Studies) [Google Scholar]
- 195.CMD Methods Pack: find a combination of research methods that suit your needs. CMD Methods Pack. [2022-01-20]. https://cmdmethods.nl/
- 196.HCD Design Kit California. Design Kit. [2022-01-19]. https://www.designkit.org/
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data extraction form.
Main goal of the study and described methods and products per included record.