Assessing the Involvement of Users During Development of Lower Limb Wearable Robotic Exoskeletons: A Survey Study

Abstract

Objective

To explore user-centered design methods currently implemented during development of lower limb wearable robots and how they are utilized during different stages of product development.

Background

Currently, there appears to be a lack of standardized frameworks for evaluation methods and design requirements to implement effective user-centered design for safe and effective clinical or ergonomic system application.

Method

Responses from a total of 191 experts working in the field of lower limb exoskeletons were analyzed in this exploratory survey. Descriptive statistics were used to present responses and measures of frequency, and chi-square tests were used to contrast the answers of respondents who identified as clinicians versus engineers.

Results

A vast majority of respondents involve users in their development, in particular at the initial and iterative stages, although some differences were found between disciplines. A variety of methods and metrics are used to capture feedback from users and test devices, and although valuable, some methods used may not be based on validated measures. Guidelines regarding tests on safety of exoskeletons also lack standardization.

Conclusion

There seems to be a consensus among experts regarding the importance of a user-centered approach in exoskeleton development; however, standardized frameworks with regard to appropriate testing methods and design approaches are lacking. Such frameworks should consider an interdisciplinary focus on the needs and safety of the intended user during each iteration of the process.

Application

This exploratory study provides an overview of current practice among engineers and clinicians regarding the user-centered design of exoskeletons. Limitations and recommendations for future directions are identified.

Introduction

The development of wearable robots (WRs) has evolved substantially over the past decade, and their application is expected to grow exponentially (de Looze, Krause, & O’Sullivan, 2017; Veneman, Burdet, Van Der Kooij, & Lefeber, 2016; Windrich, Grimmer, Christ, Rinderknecht, & Beckerle, 2016; Young & Ferris, 2017). The technology may be applied across diverse fields to augment, train, supplement, or even replace motor functions, both in individuals with and without impairments of their motor functions. Recent reviews have reported that lower extremity robotic exoskeletons have potential for benefiting the neuromusculoskeletal system in terms of function and mobility, as well as secondary medical conditions, for example, such as those relating to the circulatory and digestive systems (Federici, Meloni, Bracalenti, & De Filippis, 2015; Mekki, Delgado, Fry, Putrino, & Huang, 2018; Paleg & Livingstone, 2015). In the widened application of exoskeletons, such as the domain of worker support, results have indicated a preventive effect of these devices for work-related musculoskeletal disorders (WRMDs) such as low back and shoulder pain (de Looze, Bosch, Krause, Stadler, & O’Sullivan, 2016). Many trials that involve users are currently underway to enhance understanding in this matter (Gorgey, 2018), but although evidence points to positive effects of lower extremity exoskeletons as well as of lifting-supportive exoskeletons, objective outcomes need to be delineated (Lazzaroni, Toxiri, Ortiz, De Momi, & Caldwell, 2018; Mekki et al., 2018). Anticipating a rise of WR accessibility in the near future, human–robot interaction and human factors, such as ergonomic aspects and safety, ought to be of great concern, as should factors such as the users’ acceptance of the technology and the usability of the devices (Kaleshtari, Ciobanu, Seiciu, Marin, & Berteanu, 2016; Koumpouros, 2016; Victores, Jardón, Bonsignorio, Stoelen, & Balaguer, 2010).

User-centered designs acknowledge and highlight factors relating to the user’s personal, environmental, and social influence, in addition to technological factors (Beckerle, Christ, et al., 2017; Kaleshtari et al., 2016). For medical application, the World Health Organization (WHO) has encouraged this type of contextual approach with its biopsychosocial model, the International Classification of Functioning, Disability and Health (ICF), because a medical diagnosis alone does not adequately define ability or needs. The ICF model acknowledges that environmental and personal factors will influence a person’s health condition in terms of function and level of activity and participation (WHO, 2001, 2015), hence the importance of a well-formed design process for the application of WRs for individuals with disabilities, in their specific context. In relation to the use of exoskeletons in industry, a user-centered approach may be of great importance to enhance compliance and acceptability of WRs for workers, considering their potential in reducing risk of WRMDs (Bosch, van Eck, Knitel, & de Looze, 2016; Huysamen et al., 2018; Koopman, Kingma, Faber, de Looze, & van Dieen, 2019).

Involving the users by obtaining relevant feedback concerning the device’s capabilities and usability while taking into account contextual factors is considered fundamental during all stages of its development (Alabdulkarim & Nussbaum, 2019; Hill, Holloway, Morgado Ramirez, Smitham, & Pappas, 2017; Victores et al., 2010). This ensures that after identifying users’ abilities, needs, and preferences, as well as safety issues during development, the end product is assessed. End-product assessment would involve not only mechanical and functional testing but also an evaluation in terms of the user’s opinion of whether specific needs or preferences, on different levels, have been met. This type of iterative approach would optimally involve an interdisciplinary team, consisting of professionals with a background in engineering, human movement sciences, biomechanics, and ergonomics, depending on the type of device under development, throughout all stages of the design process. Ideally, standardized methods should be applied to obtain measures from both the device and the user, taking into account risk management and safety regulations, thereby working toward more computational modeling, bench testing, and reporting of adverse events involving users (He, Eguren, Luu, & Contreras-Vidal, 2017).

There is a lack of evidence in the literature with respect to how and to what extent developers involve users during different stages of development and whether needs or desires of the end-user are being fully identified (Federici et al., 2015; Hill et al., 2017; Koumpouros, 2016). Similarly, methods that assess potential users’ perceived usability of the device for their own purposes or consider their opinion of its appearance have received limited attention (Kaleshtari et al., 2016). This lack of evidence and absence of consensus in terms of appropriate evaluation methods in the field seems, in part, to be due to the heterogeneity of studies in the area (He et al., 2017; Hill et al., 2017). The process of standardizing evaluation methods involving users may be complicated by ethical, financial, or methodological challenges, as well as personal and environmental factors, irrespective of whether the application of the device is to aid movement in a medical setting, at home, or as an ergonomic application.

Therefore, the purpose of this exploratory study was to evaluate current involvement of users in the development of robotic lower limb exoskeletons, in particular WR with applications such as locomotor training, ambulation assistance, fall prevention, supporting physical labor, and/or augmenting power, or as a research tool, for example, for movement science. Moreover, this survey was designed to give an overview of the outcome measures most commonly used to assess the performance and safety of devices. An attempt was made to reach individuals who were involved with the development, in the broadest sense, of applications suited for various tasks such as rehabilitation (for institutions or at home), functional assistance (compensation of motor deficits to enhance residual skill or enable alternative skills), or other applications (military, industry, or sports). The results may guide user-centered development of robotic exoskeletons for walking, in particular future work regarding classification and standardization of outcome measures for the type of device under development.

Method

Study Design

An interdisciplinary team with a background in engineering, biomechanics, rehabilitation, and user experience research, all involved with COST Action CA16116 on Wearable Robots (www.WearableRobots.eu), collaborated in the development of the questions used in the survey (see Appendix A), which was published online. COST (European Cooperation in Science and Technology) is a funding agency for research and innovation networks. COST Actions help connect research initiatives across Europe and enable scientists to grow their ideas by sharing them with their peers, and CA16116 (https://www.cost.eu/actions/CA16116/) titled “Wearable Robots for Augmentation, Assistance or Substitution of Human Motor Functions” is targeting the field of WRs. The aim is to integrate different underlying disciplines in science and engineering, including ethical, legal, and societal studies, and to engage stakeholders to improve WR technology and its societal impact.

The survey questions were formulated with the goal of providing answers to specific key questions relating to user involvement, intended use of the device under development, types of outcome measures used during the development of exoskeletons, and safety. Preliminary analysis was presented at the first INBOTS conference in 2018 (Ármannsdóttir et al., 2019), but data collection continued to more than double the sample size and enable the extended analysis presented in this paper.

Participants

Professionals working in the field of WRs were invited to participate, anonymously, via various web-based platforms such as Exoskeleton Report (exoskeletonreport.com), LinkedIn, and Twitter, in addition to various email lists associated with the topic of WR and robotic rehabilitation. The survey was launched in July 2018 with QuestionPro Inc (www.Questionpro.com) and was open till the end of November 2018. It included a total of 12 questions, many of them allowing respondents to choose multiple answer options, as well as open text answers for clarification. A total of 328 persons responded to the survey, and all responses were considered for analysis, regardless of whether the survey was fully completed or not. However, as inclusion criteria, a minimum of three out of the total 12 questions were required to be answered. Furthermore, answers that indicated that the participant was involved in the development of devices outside the scope of this study (i.e., upper limb robotics, surgical robotics, etc.) were also eliminated.

A total of 191 responses met the inclusion criteria. Of the total 189 responses regarding occupation, 125 (66%) identified themselves as engineers, whereas 36 (19%) were clinicians and 28 (15%) were students. Two individuals did not indicate their occupation. Participants were given the option to provide additional details in an open text box, but these were only partially filled out, and so a more precise demographic overview of all participants is lacking. However, the demographic data show that the survey did reach out to researchers and developers from academic, industrial, and clinical user environments. Answers from students were excluded and the main focus put on exploring the current practice of experienced experts involved in the development of WRs. For each of the questions, 121 to 180 participants provided answers. A large majority (70%) of the participants were Europeans, followed by participants from North America (18%), Asia (8%), South America (2%), Oceania (1.5%), and Africa (0.5%).

Data Analysis

In addition to descriptive presentation, the answers of respondents from the two disciplines (engineer vs. clinician) were contrasted using the chi-square test (significance set at p < .05), which compares the observed distribution of answers with a theoretical, expected distribution. In cases where respondents had the option of multiple responses, chi-square results were analyzed using multiple dichotomy, allowing for a single analysis for each answer possibility. As not all respondents answered all questions, the data presented rely on varying sample sizes. Presentation and analysis of results were simplified when answers were strongly related and were seen to be chosen concurrently (e.g., “Functional assistance” in Figure 2 includes answers indicating assistance both indoors and outdoors, and “Brainstorming” in Figure 3 includes answers for brainstorming and for thinking aloud).

Figure 2.

Answers (% of engineers [n = 123] and clinicians [n = 35]) to the question, “What is the final aim of your exoskeleton?”

*Distribution of answers is significantly different from expected.

Figure 3.

Answers (% of engineer [n = 114] and clinician respondents [n = 33]) to the question, “What method(s) do you use to involve users in your exoskeleton development?”

*Distribution of answers is significantly different from expected.

Results

User Involvement

For an analysis of how much and at which stages participants are involving users during development and/or for testing of exoskeletons for walking, the total number of respondents was 157. Overall, respondents indicated overwhelmingly (97%) that they do involve users, more so in the initial stages such as identifying functional requirements (78%) and during the development/iteration of the device (75%), rather than at later stages such as when assessing the end product (42%) (Table 1).

Table 1.

Distribution of Answers for the Question, “Do You Involve Users During Your Exoskeleton Development to Gain Insight Regarding User Requirements and Usability?”

Answer Choices	Percentage of Respondents (N = 157)
Yes, to identify functional requirements	78
Yes, to provide feedback during the development/iteration of the device	75
Yes, for assessing prototypes	69
Yes, to define context of use	57
Yes, to identify technical requirements	46
Yes, for assessing the end product	42
No	3

Note. Percentages represent the proportion of respondents (N = 157).

Differences Between Disciplines (Engineers vs. Clinicians)

Responses per discipline for the intended target users may be seen in Figure 1, shown as the percentage of respondents—engineers n = 123 and clinicians n = 36.

Figure 1.

Answers (% of engineers [n = 123] and clinicians [n = 36]) to the question, “Who are your target users?”

*Distribution of answers is significantly different from expected.

There was a statistically significant difference in the engineers’ and clinicians’ distribution of answers regarding the target user of the device under development (Figure 1). The chi-square test demonstrated that engineers had a higher than expected answer count for the answer “workers” (p < .001; χ² = 11.5) and for “elderly/frail” (p = .04; χ² = 4.4), whereas the case was opposite for clinicians. The distribution of answers for “neurologically injured” target users was as expected (n.s.).

Responses per discipline regarding the final aim of the device(s) that respondents are involved in developing are presented in Figure 2, shown as the percentage of respondents—engineers n = 123 and clinicians n = 35.

There was a statistically significant difference in the engineers’ and clinicians’ distribution of answers shown by the chi-square test for “rehabilitation” as the final aim of the device (p = .001; χ² = 10.5) and for “other” (p < .001; χ² = 11.8). The observed values among engineers were lower than expected for “rehabilitation” and higher than expected for “other” devices, whereas the opposite was seen for clinicians. Observed values for “functional assistance” were similar to the expected values (n.s.).

Methods Applied to Involve Users During Development

Respondents were asked what method(s) they utilize to involve users in their exoskeleton development. Responses per discipline are presented in Figure 3, shown as the percentage of respondents—engineers n = 114 and clinicians n = 33.

The chi-square test demonstrated a significant difference in the observed versus expected distribution of answers between engineer and clinician respondents for “questionnaires” (p = .003; χ² = 8.6), for “prototype testing” (p = .01 χ² = 5.5), and for “psychophysical methods” (p = .005; χ² = 7.9). Observed values among engineers were lower than expected for the use of “questionnaires” and “psychophysical methods” and higher than expected for the use of “prototype testing,” whereas the opposite was seen with respect to the distribution of answers from clinician respondents.

Metrics or Measures Applied During the Design Process

Functional tests or measures were identified by respondents as the most commonly used when asked “how they ensure that the exoskeletons meet the user requirements during the design process of the prototype/system” (by 71% of 110 respondents). Pain measures and emotional responses were the least used (by 28% of respondents). Table 2 gives an overview of the results from that question, where participants were given the option of specifically answering which methods were used.

Table 2.

Answers (% of Respondents; n = 110)

Specific Requirements; Answer Possibilities	% of Respondents (n = 110)	Stated Methods to Meet User Requirements	Number of Examples Given
Functional	71	Unspecified tasks such as “easy to use,” “efficiency,” “agility,” and ADL (20% of all examples given) and unspecified gait and stair analysis methods Specific methods: 6MWT; 10MWT; TUG; gait speed; sit to stand; ROM; stability; EMG; metabolic; temporal-spatial parameters; durability; battery; stiffness; posture; Tinetti; Barthel; MMSE; FIM; Fugl-Meyer; PUL; FAC; Romberg; SCIM II; WISC II; GMFM; PODCI; Bealey; SOT-2	101
Comfort	53	Unspecified questionnaires and general discussion with the user (69% of all examples given) Specific methods: skin examination; VAS; Borg; force interaction; weight of device; ROM; EMG; SUS; NASA TLX; QUEST 2.0	45
Biomechanical	49	Unspecified gait or motion analysis with or without 3D capture (63% of all examples given) Specific methods: EMG; force interaction; REBA; balance; metabolic; posture assessment	49
Satisfaction	49	Unspecified questionnaire, discussions, nonstructured interviews (82% of all examples given) Specific methods: QUEST; VAS; Likert-type scale; SUS; Pittsburgh participation scale; TAM2	38
Usability	32	Unspecified tests, questionnaires, or discussion (70% of all examples given) Specific methods: Likert-type scale, QUEST 2.0	7
Fatigue	29	Unspecified questionnaire or tests (25% of all examples given) Specific methods: oxygen consumption; heart rate; how long can user wear device; endurance parameters while wearing device; VAS; Borg; 6MWT; EMG; IPAQ; Böhr score	29
Pain	28	Unspecified questions and general discussion with the user (48% of all examples given) Specific methods: VAS; pressure perception; swelling examination; NPS; NRS; Tampa Scale; Fugl-Meyer	23
Emotional responses	28	Unspecified questionnaires/discussion/observation methods under development (89% of all examples given) Specific methods: SAM; VAS; NASA TLX; SUS	19

Note. The examples are ordered by the number of times they were mentioned. Not all of those who chose a specific answer provided an example, so the number of examples is given for each answer. A list of abbreviations is provided in Appendix B.

When asked whether they experienced any ambiguity or doubts regarding safety evaluation of the devices under development, 45% of the 103 respondents indicated that they did. The same proportion of 102 respondents, when asked how they evaluated the safety of the device under development, stated that they used their own test protocols, and 32% claimed that limited testing was done due to a lack of available, standardized methods for assessing product safety.

Discussion and Conclusion

The broad spectrum of WRs and large variability in types of users call for the identification of current practice regarding application of user-centered approaches during development and testing of exoskeletons. A survey was conducted to explore the current standard of practice with respect to the degree and methods of user involvement among experts involved in the development of WR. Answers from a total of 191 respondents were analyzed, which likely provides a fairly accurate overview of common practice in the field. The vast majority (97%) of experts indicated that they do consider user outcomes both at the initial and during iterative stages of their devices’ development, which indicates a positive shift when contrasted with a recent review of studies published before 2016, which clearly revealed a lack of users’ perspective in exoskeleton technology (Hill et al., 2017). The results further demonstrate that the respondents’ background (engineer vs. clinician) seems to correspond with the type of development they are involved in, that a great variety of methods are being used to involve users, and, importantly, that more specific guidelines regarding safety testing of devices under development may be required.

A large proportion of respondents from both disciplines indicated neurologically injured persons as target users (Figure 1), although significant differences between engineers and clinicians were found. The biggest difference was seen for workers as the target user, where a greater number of engineers than expected named workers, whereas the opposite held true for the clinicians. This is to some extent expected, as engineers are more likely to be recruited to identify and develop solutions aiming at primary injury prevention and productivity of workers than clinicians. However, clinicians typically have a background in functional anatomy and movement sciences, so they may be underutilized in the area of ergonomic design for workers, as less than 10% of clinician respondents stated that their end-users were workers. Somewhat surprisingly, the same pattern was seen for the elderly/frail as end-users, although the statistical result was not as strong. Motor impairment and/or frailty of the elderly are clinical conditions, so one might expect clinicians to be heavily involved in the design and evaluation of WRs for this population. However, it is possible that the survey did not reach clinicians working within geriatrics.

Answers regarding the final aim of the devices that respondents were involved in developing were somewhat consistent with their answers regarding the target users (Figure 2). Although the distribution of the engineers’ answers for the aim of the exoskeleton design was lower than expected regarding rehabilitation, the fact remains that two thirds of engineer respondents indicated that they were involved in the development of devices in that area, as well as for functional assistance. As expected, clinicians’ answers reflect their strong presence in the field of rehabilitation, and the largest difference between the two disciplines was for end products developed for use in the military, industry, and/or sports. As noted earlier, this may indicate that greater representation of clinicians might be encouraged in areas that are not associated with disease or dysfunction to boost interdisciplinary collaboration and secure comprehensive expertise within the team.

The methods used to involve users during development likely vary depending on the stage of development, but overwhelmingly involved observations, interviews, and brainstorming sessions, and this was seen for both disciplines (Figure 3). The differences seen between disciplines revealed that engineers apply questionnaires and psychophysical methods less and prototype testing more than expected, with the opposite seen in clinicians. This reflects the background and perspectives of professionals who generally work in different settings and indicates that an interdisciplinary approach is needed to ensure the use of diverse outcome measures throughout the entire design process. Despite broad acceptance of the value of engineer/clinician collaboration to address challenges related to human mobility (Gorgey, 2018; Koumpouros, 2016), this is not yet a reality for several applications. Nevertheless, important steps have been taken in the direction of a multifaceted design and testing approach for certain areas of exoskeleton application (ASTM International, 2019; EUROBENCH: European Robotic Framework for Bipedal Locomotion Benchmarking, 2019; Mudie et al., 2018).

Unsurprisingly, most respondents claimed to involve users at the initial and developmental stages, more so than during the end-product testing (Table 1). Determining needs and preferences of WR users is fundamental to the development of new technology; however, user involvement in assessing the end products is necessary to verify that they meet developers’ goals. Reasons given by respondents for not involving users during the development of WRs include lack of patient availability and/or funding, and the possible need to validate a system before permission to test with users is granted, which might be especially inhibitive for industrial development. The options for user involvement during different stages, such as for testing of uncertified devices without a research purpose (e.g., for usability evaluation), may vary greatly depending on the country (regulations and ethical/practical procedures) as well as the intended end-users (e.g., healthy workers vs. specific patient groups) and may hinder end-user involvement in product development altogether.

Limiting the involvement of intended end-users has obvious implications with respect to how testing of a particular device translates to the population it is intended to serve. However, user testing during the development phases of WRs should involve the adequate standards for safe and informative human testing. Regarding safety, respondents often stated that they were unaware of standardized methods that would be applicable for safety evaluations. This apparent lack of guidelines may yield an underreporting of adverse events involving users, such as falls or injuries (He et al., 2017). In addition to perceived and actual safety of the user (number of adverse events), important ethical concerns include cases where the end product may not meet user needs in terms of function, usability, and contextual factors.

As seen in Table 2, a variety of metrics or measures are used to ensure that the exoskeletons meet the user requirements during development. Respondents frequently gave examples of unspecified methods, such as “questionnaires,” “general discussions with users,” and “visual observation.” Similar results have been reported, and researchers have concluded that originally developed questionnaires are frequently used to fit investigators’ desired research questions due to the lack of standardized tools in the field of WRs (Koumpouros, 2016; Poritz, Taylor, Francisco, & Chang, 2019). Successful assessment of the users’ opinion and perception in a broad context is important, and this may also explain the large number of different metrics listed by survey participants. However, although many of these methods are less structured than tests assessing prototypes or end products, they may yield useful information in identifying users’ expectations and needs and in exploring potential solutions during the early stages of development.

The field of WR is essentially young and there are many different aspects of sophisticated systems such as WR that need to be evaluated. The multitude of measurements being used likely reflects the degree of innovation in the field and heterogeneity of users tested, which makes comparability between studies difficult. Importantly, the efficacy of devices in improving function and/or performance or minimizing risk of injury of the intended user needs to be determined within the context of the individual’s ability/disability in addition to personal and environmental factors. Therefore, it inevitably proves very complex to choose and apply a valid and relevant metric/test that will grasp the different aspects of human–device interaction and performance, which is considered one of the limitations to current WR technology (Beckerle, Salvietti, et al., 2017; Deng et al., 2018).

Functional tests were among the specific outcome measures most commonly listed by survey participants (Table 2), specifically the 6-Minute Walk Test (6MWT), the 10-Meter Walk Test (10MWT), gait speed, and the “Timed Up and Go” (TUG), in addition to generalized terms including “gait analysis” and “temporal-spatial parameters.” This is consistent with reports of useful parameters relating to functional effectiveness of WR (Lajeunesse, Vincent, Routhier, Careau, & Michaud, 2016; Louie, Eng, & Lam, 2015). Quantitative metrics like the ones mentioned earlier are all relevant in assessing the functional capacities of the devices under development, whether to evaluate training effects of an exoskeleton for rehabilitation or for direct augmentation of movement. Some measures have been validated across many patient populations and their limits with respect to minimal detectable change, and more importantly, what constitutes a clinically meaningful change is often well defined (Poncumhak, Saengsuwan, Kamruecha, & Amatachaya, 2013). Results from such metrics, however, need to be put into perspective relating to not only the emerging field of WR but also the user’s ability/disability and contextual factors, as well as user satisfaction or emotion with WR use. That way, as long as no adverse effects due to the device are registered, improvements in, for example, walking speed can actually be interpreted as a positive factor for the user. Factors such as emotional responses, however, are the ones the survey respondents identified as the least used, which is in accordance with previous publications (Federici et al., 2015), and may reflect the lack of metrics specific to emotional responses.

In terms of product testing, reliability, validity, and the appropriateness of methods used are oftentimes unclear and respondents of the survey were not asked to provide information regarding specific implementation of the tests used. The need for a holistic evaluation of devices has been approached, with suggestions for a benchmarking framework to support ongoing research and marketing activities (Torricelli et al., 2015). Future steps should focus on advancing the use of optimum techniques for a user-centered approach during all stages of development, with ways of effectively gathering, analyzing, and utilizing information from users in an unbiased manner for the development of systems that achieve users’ needs and preferences.

Important limitations need to be kept in mind when interpreting results. The majority of respondents were European, and although they indicated their occupation, it is unclear whether their work environment related primarily to academic or industrial research, exactly which profession “clinicians” belonged to, or to what extent they are involved in research and/or rehabilitation practice. However, the response rate was strong, which should ensure a realistic insight into current, unpublished research methods in different laboratories, clinics, and/or companies. Although the use of a limited number of questions does not provide an in-depth analysis, it may have prompted a stronger response rate, and the open text answers did allow respondents to answer freely beyond the options provided. Although we have limited information regarding those who opened the survey but opted out, anonymity of respondents likely ensures honest answers to the questions posed.

Conclusion

The results of this exploratory survey signal that although the need of user involvement during different stages of development and adoption of WRs is recognized, there is a broad heterogeneity of methods used and a lack of clarity as to which ones are valid, reliable, and indicated. This limits the effectiveness of the user involvement and limits the transfer of knowledge among different developers, hence resulting in the adequate design and adoption of WRs in different applications. The field will have to clarify which objective outcome measures, specific to an application and stage of development, best demonstrate whether the WR technology is safe and fulfills its function for the target user in terms of functional performance improvement, user satisfaction, and effect on participation levels. This will inform future directions with respect to defining standards that are relevant for all stakeholders and may serve as important guidelines for future development practices, benchmarking, and testing.

Key Points

• There seems to be an increase in user involvement in the development of exoskeleton when results of this survey are compared with published literature.

• The data of this exploratory survey further indicate that although most developers are aware of the importance of user involvement, there is a lack of knowledge and consistency on what method of user involvement is adequate for specific applications, for specific user groups, or for specific phases of development.

• Slight differences in answers between engineer and clinician respondents underscore the importance of an interdisciplinary team during all developmental stages and testing of lower limb exoskeletons.

• Testing methods need to be standardized, but still be specific for the intended users and their contextual factors, and for the type of device and its intended use.

• A clearer, user-centered policy needs to be created to involve end-users in development not merely as potential users but as expert collaborators, designers, and developers helping to frame the product design or research project.

• Future works might take an in-depth look at the differences in current practices of user involvement between application cases and consider using both quantitative and qualitative data to help reveal ability/disability-specific best practices.

Biography

Anna L. Ármannsdóttir is a physical therapist and currently a PhD student at the University of Iceland’s Faculty of Medicine, Reykjavík, Iceland. She holds a post-professional MSc in movement science (2014) from the University of Iceland.

Philipp Beckerle is an assistant professor in the faculty of electrical engineering and information technology at Technische Universität Dortmund, Germany, and adjunct researcher in the Department of Mechanical Engineering at Technische Universität Darmstadt, Germany. He received his Dr.-Ing. in mechatronics from the Technische Universität Darmstadt in 2014.

Juan C. Moreno holds a position as scientist and is head of the Neural Rehabilitation Group in the Cajal Institute at CSIC in Madrid, Spain. He received a PhD degree in industrial engineering from the School of Industrial Engineering of the Polytechnic University of Madrid (UPM) in 2006.

Edwin H. F. van Asseldonk is an associate professor in the Department of Biomechanical Engineering at the University of Twente, Enschede, The Netherlands. He received his PhD in biomechanical engineering from the University of Twente in 2008.

Maria-Teresa Manrique-Sancho is a senior UX researcher in Vector ITC Group and an assistant professor at the International University of La Rioja (UNIR), Spain. She received a PhD in engineering/UX design from the Polytechnic University of Madrid (UPM) in 2016.

Antonio J. del-Ama received his PhD in electric, electronics and control engineering from the Carlos III University (UC3M) of Madrid in 2013. Until 2019, he was group leader of the Biomechanics and Assistive Technology of the National Hospital for Paraplegics in Toledo, Spain, and is an associate professor at the Rey Juan Carlos University, Spain.

Jan F. Veneman is a technical project manager at Hocoma AG, Volketswil, Switzerland. He received his PhD in biomechatronics from the University of Twente in 2007. He is general chair of the COST Action CA16116.

Kristín Briem is a professor in the Faculty of Medicine at the University of Iceland, Reykjavík, Iceland. She received her PhD in biomechanics and human movement science from the University of Delaware in 2008.

Appendix A

Online Questionnaire for Developers of Wearable Robotic Exoskeletons for Walking

1. Please provide information regarding your background/current occupation before going on to answer the questionnaire.

Student of _______________________

Clinician working in the area of ______________

Engineer currently working in the area of ____________________________

• 2. What type of wearable lower limb robotic exoskeleton for rehabilitation/compensation of walking are you developing? (please mark all that apply)

  • a. Medical for locomotor training
  • b. Assistive device to enable ambulation
  • c. Assistive device for fall prevention
  • d. Exoskeleton to support physical labor and/or augment power
  • e. Research tool for movement science/biomechanics
  • f. Other: (a text box to specify it)
• 3. Who are your target users? (please mark all that apply)

  • a. Neurologically injured (stroke, SCI, CP, other)
  • b. Elderly/frail persons
  • c. Workers
  • d. Other: (a text box to specify it)
• 4. What is the final aim of your exoskeleton? (please mark all that apply)

  • a. Clinical rehabilitation
  • b. Rehabilitation at home
  • c. Rehabilitation outdoors
  • d. Functional assistance/compensation indoors
  • e. Functional assistance/compensation outdoors
  • f. Military applications
  • g. Sports
  • h. Industrial/ergonomic application
  • i. Other: (a text box to specify it)
• 5. Do you involve users during your exoskeleton development to gain insight regarding user requirements and usability? (please mark all that apply)

  • a. No, please tell us why (text box):
  • b. Yes, please indicate in which stage do you use it:

    • i. To identify technical requirements
    • ii. To identify functional requirements
    • iii. To define context of use (scenarios, tasks, and environment)
    • iv. To provide feedback during the development/iteration of the device
    • v. For assessing prototypes
    • vi. For assessing the end product
    • vii. Other: (a text box to specify it)
• 6. If the answer was yes to the previous question, what method do you use to involve users in your exoskeleton development? (please mark all that apply)

  • a. Questionnaires
  • b. Brainstorming
  • c. Interviews
  • d. Thinking aloud during the exoskeleton use
  • e. Contextual inquiry
  • f. Diaries
  • g. User observation
  • h. Prototype testing
  • i. Focus groups
  • j. Post-release testing
  • k. Psychophysical methods
  • l. Other: (a text box to specify it)
• 7. What are the top three user requirements* you consider in the development of your exoskeletons (open text option)?

  • *by user requirement we mean the characteristic or capacity that the user needs from the exoskeleton to solve a problem or achieve an objective from their point of view.

For example, the user wants to be autonomous, so the user requirement could be “The user shall be able to put on the exoskeleton without external assistance in less than 1 min.”

• 8. How do you ensure that the exoskeletons meet the user requirements during the design process of the prototype/system (please mark all that apply)?

  • a. Comfort tests/measures, such as _______
  • b. Functional tests, such as __________
  • c. Biomechanical tests, such as ________
  • d. Pain tests/measures, such as _________
  • e. Fatigue tests/measures, such as ________
  • f. Requesting users’ emotional responses (happiness, surprise, sadness, anger, disgust, fear), such as _______________
  • g. Requesting users’ satisfaction of use, such as _______________
  • h. Usability measures, such as ___________
  • i. Other: (a text box to specify it)
• 9. What metric outcomes do you use to evaluate your exoskeleton, once the design has been finalized? (please mark all that apply)

  • a. Technical tests (e.g., control performance. Please specify.)
  • b. Functional tests (e.g., Timed Up and Go. Please specify.)
  • c. Biomechanical tests (specific kinematic/kinetic parameters. Please specify.)
  • d. Therapeutic outcome measures (neurological and/or functional. Please specify.)
  • e. Dropout rate among users
  • f. Users’ emotional responses (happiness, surprise, sadness, anger, disgust, fear. Please specify.)
  • g. Users’ satisfaction of use
  • h. Maintenance of users’ independence
  • i. Degree of the burden of care (of professional health care providers, family members)
  • j. Other: (a text box to specify it)
• 10. What actual features of your exoskeleton or functionality may, in your opinion, be a limitation/barrier for satisfactory use? (please mark all that apply)

  • a. Putting on–taking off the device prior to and after its use
  • b. Size of the exoskeleton, for example, being able to sit easily in a regular chair
  • c. Appearance of the device
  • d. Degree of necessary maintenance
  • e. Comfort of the device (e.g., its weight and fit)
  • f. Level of sound while in use
  • g. (Remote) control features
  • h. Battery life
  • i. Security of the user (e.g., after a fall; the user’s ability to take the device off with ease to get up from the floor
  • j. Other: (a text box to specify it)
• 11. How do you evaluate safety? (please mark all that apply)

  • a. Using medical device norms, such as NEN-EN-IEC 60601-1
  • b. Using industrial norms, such as ISO-TC 15066 and ISO 10218-1
  • c. Using electrical safety test protocols for medical devices
  • d. Using self-developed test protocols
  • e. Limited testing, due to lack of clarity regarding how to evaluate certain safety aspects
  • f. Other: (a text box to specify it)
• 12. Do you experience any ambiguity/doubts concerning safety evaluation?

  • a. No
  • b. Yes; please describe what the most pressing issues are (a text box to specify it)

Appendix B

List of Abbreviations

ADL: activities of daily living

EMG: electromyography

FAC: Functional Ambulation Category

FIM: Functional Independence Measure

GMFM: Gross Motor Function Measure

IPAQ: International Physical Activity Questionnaires

MMSE: Mini-Mental State Examination

NASA TLX: NASA Task Load Index

NPS: Neuropathy Pain Scale

NRS: Numeric Rating Scale

PODCI: Pediatric Outcomes Data Collection Instrument

PUL: Performance of Upper Limb

QUEST: Quebec User Evaluation of Satisfaction with assistive Technology

REBA: Rapid Entire Body Assessment

ROM: range of motion

SAM: Self-Assessment Manikin

SCIM II: Spinal Cord Independence Measure II

SOT-2: Sensory Organization Test 2

SUS: System Usability Scale

TAM: Technology Acceptance Model

TUG: Timed Up and Go

VAS: Visual Analogue Scale

WISCI II: Walking Index for Spinal Cord Injury II

10MWT: 10-Meter Walk Test

6MWT: 6-Minute Walk Test

ORCID iDs

Philipp Beckerle https://orcid.org/0000-0001-5703-6029

Juan C. Moreno https://orcid.org/0000-0001-9561-7764

Kristín Briem https://orcid.org/0000-0002-0606-991X