Evaluation and Test Methods of Industrial Exoskeletons In Vitro, In Vivo, and In Silico: A Critical Review

Abstract

Industrial exoskeletons have been used to assist workers during occupational activities, such as overhead work, tool-use, mobility, stooping/squatting, and/or load carrying in various industries. Despite the promise of reducing the risk of work-related musculoskeletal disorders, there is a lack of sufficient evidence to support the safe and effective use of industrial exoskeletons. To assess the merits and residual risks of various types of exoskeletons in different work settings, more comprehensive evaluation procedures are needed. This review study aims to provide an overview of the existing viable and promising methods for evaluating the effectiveness of industrial exoskeletons. The different evaluation methods are organized into three categories—in vitro, in vivo, and in silico studies. The limitations and challenges in different types of evaluation approaches are also discussed. In summary, this review sheds light on choosing appropriate evaluation approaches and may help with decision-making during the development, evaluation, and application of industrial exoskeletons.

I. INTRODUCTION

An exoskeleton is defined as a wearable device that augments, enables, assists, and/or enhances physical activity, either static or dynamic, through mechanical interaction with the body.¹ An exoskeleton may include rigid and/or soft components; so, the term “exoskeleton” is also inclusive of exosuits. Exoskeleton devices have been introduced in various domains, such as military, rehabilitation medicine, and industry, and have different designs or performance expectations for various applications. Regarding field military application, exoskeletons are mainly used to enhance the physical capability of dismounted soldiers. For rehabilitation medical applications, exoskeletons are primarily used to assist or treat medically diagnosed conditions, with the users often injured or physically disabled. Lastly, for industrial/occupational application, exoskeletons have been used to assist workers during occupational activities, such as overhead work, tool use, mobility, stooping/squatting, and/or load carrying in multiple industries (e.g., logistics, warehouse, factory, and healthcare industrial settings).²

From an occupational health and safety perspective, the expectation of using industrial exoskeletons is to protect workers by reducing risk factors of work-related musculoskeletal disorders (MSDs) during physically demanding jobs. Although the rise of a commercial market may inflate other expectations about the benefits of using industrial exoskeletons (e.g., productivity gains and quality enhancements),³ increased efficiency and productivity of workers are suggested to be a secondary benefit expected from using exoskeletons. Many industrial exoskeletons tend to be task-specific, in which different types of exoskeletons are designed to assist with various tasks. In general, exoskeletons are typically categorized in terms of sources of power and areas of support. Based on the sources of power, industrial exoskeletons are commonly categorized into active (e.g., electrically powered) or passive (e.g., springs and counterweights) devices. However, there are new types of power sources added to this category, such as pseudo-passive exoskeletons, which contain batteries, sensors, and other electronics, but do not actuate the devices, or semi-active exoskeletons, which contain a combination of passive and active actuators. Exoskeletons can also be categorized based on body regions of assistance, such as upper-extremity, lower-back, lower-extremity, or whole-body exoskeletons. Additionally, exoskeletons can be worn to extend the body by providing extra (supernumerary) limbs.⁴ In regard to industrial exoskeletons, the main purpose is to augment, amplify, or reinforce the performance of a worker’s existing body components, primarily the lower back and the upper extremities.³

Despite the promise of reducing fatigue and the risk of MSDs, there is a lack of sufficient evidence to support the safe and effective use of industrial exoskeletons. Potential risks of using exoskeletons in the workplace should be better understood before the widespread implementation of industrial exoskeletons, especially their long-term and cumulative effects. For example, although users may benefit from wearing upper extremity exoskeletons during manual handling tasks, adverse physiological consequences (e.g., increased antagonist muscle activity, postural strains, cardiovascular demand, and modified kinematics) were reported.⁵ There are potential risks of injuries while wearing exoskeletons, such as the exoskeleton malfunctioning; a slip, trip, or fall incident, limitations of the user’s overall mobility; and possible collisions between the exoskeleton and work equipment.⁶ Other risks of wearing exoskeletons include thermal discomfort, pressure wounds, and/or compressed nerves from prolonged use.⁶ To assess the merits and residual risks of various types of exoskeletons in different work settings, more comprehensive evaluation procedures are needed. Although previous studies do not cover all the evaluation aspects, attempts have been made to leverage assessment methods and techniques to accommodate the different types of exoskeletons and various working tasks and environments individually.

This review study aims to provide an overview of the existing viable and promising methods for evaluating the effectiveness of industrial exoskeletons. An effective industrial exoskeleton is defined as one that reduces the exertion by the user or the physical load on the user, and shows positive effects on performance, acceptance, or usability with minimal or acceptable adverse effects on users’ well-being. Based on the general categories of experimental approaches, this review organizes the different evaluation methods into three categories— in vitro, in vivo, and in silico studies. In vitro (Latin for within the glass) refers to the technique of performing a given procedure in a controlled environment outside of a living organism. Here, it refers to exoskeleton tests on a physical model or smart-controlled mannequin instead of on a human subject. In vivo (Latin for within the living) here refers to experimentation using live human subjects. In silico refers to testing the devices using computational or biomechanical modeling and simulations. Since exoskeleton technology is emerging and evolving rapidly, the methods represented in this review are those found in the current peer-reviewed literature on the evaluation of exoskeletons in industrial applications. This list of methods and evaluation tools can be expanded as technology advances. These findings are applicable to further consensus on the standards for industrial exoskeletons and exosuits, e.g., ASTM International F48.²

II. EXISTING EVALUATION APPROACHES

A. in Vitro Approach

Similar to crash test dummies in automobile collision tests, an instrumented body dummy was used to test the performance of back-assist exoskeletons in a standardized way [Fig. 1(a) and Table 1].⁷ This performance test has been included in the Japanese Industrial Standards (JIS B 8456–1) for lumbar-support exoskeletons,⁸ which specify performance and labeling criteria for lumbar support wearable robots based on a consensus between manufacturers, consumers, and other neutral bodies (i.e., national research institutes and certification bodies). Additionally, the performance test is also included in the International Organization for Standardization (ISO) 18646–4.⁹

FIG. 1:

Representative evaluation methods of exoskeletons illustrated in three categories: (a) in vitro approach using the instrumented human-shaped body dummy (reprinted from Nabesima et al. with permission from IEEE, copyright 2018)⁷; (b) in vivo approach using human subject testing (reprinted from Maurice et al. with permission from IEEE, copyright 2020)¹¹; and (c) in silico approach using biomechanical modeling (reproduced from Tröster et al. under a Creative Commons License)⁴⁸

TABLE 1:

Overview of three types of evaluation approaches—in vitro, in vivo, and in silico. The representative research studies that performed evaluations of industrial exoskeletons or exoskeleton-related conditions/systems were summarized, and selected example measures were put into different categories accordingly. Note that studies may have other measures that are not listed in the table, or the same studies may appear multiple times in different categories

Refs.	Exoskeletons or relevant conditions/systems	Evaluation methods or tasks	Selected example measures
In Vitro
7	Powered back-assist exoskeleton (HAL)	Instrumented human-shaped body dummy to simulate trunk movement during lifting	Assistive torque index (ATI); lumbar compression reduction (LCR)
In Vivo
Job-performance related
In Dynamic Tasks
12	Powered back-assist exoskeleton (HAL)	Repetitive stoop lifting (12-kgbox) until subjects were fatigued	Number of lifts; total lifting time
13	Powered back-assist exoskeleton (HAL)	Snow shoveling until subjects were fatigued	Number of scoops; total snow-shoveling time and distance
14	Passive back-assist exoskeleton (Spexor prototype)	Trunk movements for assessing range of motions in three-dimensional directions	Range of motion in lumbar spine
15	Passive back-assist exoskeletons (backX and Laevo)	Simulated manual assembly task	User’s movement; working postures
16	Passive upper-extremity exoskeleton (EksoVest prototype)	Simulated overhead drilling task	Number of errors; task completion time
In static/quasi-static tasks
17	Passive back-assist exoskeleton (Laevo)	Simulated assembly task (static holding of a forward bending posture)	Endurance time
18	Passive upper-body exoskeleton (Paexo)	Static and semi-static overhead work	Shoulder angle; elbow angle
In non-task activities
19	Passive back-assist exoskeleton (Laevo)	Walking (non-task activity)	Metabolic cost; walking speed; step length
20	Passive back-assist exoskeleton (Laevo)	12 functional activities including non-task activities, such as walking, sit to stand, bending, and stair climbing	Performance metrics (e.g., range of motion, performance time, walking distance, fingertip to floor distance while bending); adjustability (e.g., donning/doffing, length adjustment)
User-response related
Psychological measures
27	Three passive back-assist exoskeletons (Laevo, CORFOR, BackX) and one active back-assist exoskeleton (CrayX)	Prone positioning procedure for patients with severe COVID-19-related acute respiratory distress syndrome	Questionnaires on perceived effort, safety, comfort, efficacy, installation, and intention of use
20	Passive back-assist exoskeleton (Laevo)	12 functional activities including lifting, carrying, walking, and climbing	Discomfort level; perceived task difficulty
30	Passive back-assist exoskeleton (Laevo)	Four-week field study for investigating automobile manufacturing tasks including static forward bending and dynamic manual material handling	Discomfort; usability; user acceptance
32	Different padding materials of exoskeletons	Pressure tolerance testing at subject’s thigh, chest, and pelvic areas	Algometry measures, e.g., the first discomfort threshold (FDT), the first pain threshold (FPT), and time to FDT
Physiological measures
33	Passive back-assist exoskeleton (Spexor)	Five-min repetitive lifting task	Metabolic cost; muscle activity
18	Passive upper-body exoskeleton (Paexo)	Simulated sustained static and semi-static overhead tasks	Oxygen consumption; heart rate; muscle activity
11	Passive upper-body exoskeleton (Paexo)	Overhead pointing task with a hand-held power drill	Heart rate; oxygen consumption; muscle activity in shoulder and lower back
17	Passive back-assist exoskeleton (Laevo)	Simulated assembly task (static holding of a forward bending posture)	lower back muscle activity
34	Passive back-assist exoskeleton (VT-Lowe’s)	Symmetric and asymmetric lifting.	Peak and mean EMG in back and leg muscles
35	Different circumferential compression conditions	Circumferential compression testing that simulates different soft-exoskeleton-human interaction	Deep tissue oxygenation
Neurophysiological measures
43	Passive back-assist exoskeleton (Laevo)	Asymmetric lifting tasks	Brain connectivity based on fNIRS; neural cost of human-exoskeleton interaction
In Silico
47	Spine-inspired continuum soft exoskeleton prototype	Biomechanical modeling and simulation for stoop lifting	Disc compression/shear forces; muscle forces
48	Active shoulder-assist exoskeleton design	Biomechanical-model-based approach for developing and evaluating an exoskeleton design in assisting with manual patient transferring	Joint compression, muscle forces in shoulder complex.
49	Steadicam vest system	Dynamic EMG-assisted spine model for investigating occupational tasks using powered hand tools	Compressive spinal loads

The aforementioned human-shaped body dummy⁷ has one fixed lumbar joint and two active joints at the knee and hip, and the shapes of each body part can be customized to fit the back-assist exoskeleton being tested. The dummy was equipped with force and motion sensors and was controlled by standard proportional derivative controllers. As an example product, a powered back-assist exoskeleton was tested using this instrumented body dummy. Two performance metrics were reported: (i) assistive torque index (ATI) and (ii) lumbar compression reduction (LCR), which may not be easily recorded when testing on a human user with existing measurement sensors.⁷ However, the performance values of ATI and LCR may not be identical to the assistive torque and the reduction of the lumbar compression force received by a living human user since the human user might respond differently to the exoskeletons. In other words, exoskeletons might have both physical and psychological effects on the exoskeleton users, and the interactions between the users and the exoskeletons are not considered in this method. Nevertheless, the in vitro dummy testing method provides a standard approach to compare some physical effects of different exoskeletons to good repeatability and consistency by eliminating human variation that is seen during live subject testing.

A similar study¹⁰ proposed more complex test dummies for repeatable and improved measurements of exoskeleton effects on humans, which could be modular and consist of other required human body parts. Joint actuation, sensing, and combined system motions, including feedback from the exoskeleton, can be integrated and potentially controlled via system modeling language. These concepts may also provide useful short- and long-term effect measurements on both human users and exoskeletons, which can be used to optimize wearable exoskeleton designs to improve safety and performance. For instance, test dummies can be beneficial for testing mean time to failure of exoskeletons, exoskeleton component contact with the human body, effects of off-axis joint rotation, and other areas where fatigue or unsafe conditions may place human participants at too great of risk in human-in-the-loop tests.

B. In Vivo Approach

In vivo testing [Fig. 1(b) and Table 1] is often employed over in vitro, because it is better suited for observing the overall effects of an experiment on a living subject (i.e., exoskeleton user), resulting in potentially more external validity. In vivo subject testing can include various in vivo metrics (e.g., muscle activity, metabolic cost, kinematics, brain responses, etc.) that can be measured when a subject is performing a task while wearing an industrial exoskeleton. The in vivo measurements for exoskeleton evaluations often fall into two categories: job performance and user response. Job-performance-related measurements are often related to job completion time and/or job quality. User-response-related measurements primarily focus on how an exoskeleton may affect the physical and mental responses of the users.

1. Job-Performance Measures

Since reduced productivity is costly for companies and can generate stress among workers exposed to a high work pace and/or quality pressure,¹¹ an expectation of using exoskeletons is to improve, or at least not degrade, task performance and/or productivity. Performance-related measures may be different due to different requirements in various dynamic and static working tasks. Workers’ kinematics and postures can be considered job-performance-related measures, because they may affect production- or service-related outcomes, workflow, and resulting quality of the product or service. In addition, the “transparency” of an exoskeleton—whether the exoskeleton impedes mobility or interferes with other non-task activities—is another important measure for evaluating exoskeletons’ performance. How easy it is to don/doff or adjust the exoskeleton should also be considered as part of performance evaluation.¹⁰

In dynamic tasks, several parameters, such as completion time, quantity of products, work quality, or range of motion, may be evaluated to understand the effects of industrial exoskeletons on job performance. For example, a powered back-assist exoskeleton was evaluated by lifting performance (i.e., number of lifts and lifting time) with and without wearing the exoskeleton during repetitive lifting movements until the subjects felt fatigued.¹² The same exoskeleton was also evaluated by measuring total snow-shoveling time and distance during a snow-shoveling task.¹³ Additionally, the range of motion in the human lumbar spine was used to assess a new back-assist exoskeleton design—the novel exoskeleton using flexible beams compared to a previous product with a rigid structure. It was found that the novel design increased the range of motion of the trunk in the sagittal plane by > 25%.¹⁴ However, other studies have noted that using a back-assist exoskeleton produced some unexpected effects on the user’s movement or posture, including small changes in working postures (e.g., lumbar flexion changes 14 deg).¹⁵ Other negative performance-related effects, such as an increased number of errors, were found while a subject was wearing an exoskeletal vest during an overhead drilling task, despite nearly 20% of the reduction in the task completion time compared to the no-vest condition.¹⁶

In static or quasi-static tasks, duration is one of the main factors used to evaluate the effect of exoskeletons on job performance. During a static forward bend posture, endurance time was found to be three times higher when a passive exoskeleton was used compared to without an exoskeleton.¹⁷ Joint angle is another performance-related parameter, and similar joint angles and postures are expected while wearing an exoskeleton compared to the no-exoskeleton condition. For example, shoulder and elbow angles were found to be comparable in static and semi-static overhead work activities while wearing a passive upper-body exoskeleton.¹⁸

In addition, if exoskeletons are designed to be worn during work and break times, they should provide support or not interfere with non-task activities. For instance, a passive exoskeleton was suggested to be beneficial for lifting (a “task” activity); although, subjects tended to walk (a non-task activity) slower and with shortened steps.¹⁹ In a similar study, a passive trunk exoskeleton was tested in 12 activities, including non-task activities, such as walking, sit to stand, and stair climbing.²⁰ Significant reductions in performance between the exoskeleton and control (without exoskeleton) conditions were found for carrying time, walking distance, stair climbing time, ladder climbing time, and fingertip-to-floor distance when bending the trunk. No significant differences were found in maximum holding time of forward bending, sit-to-stand time, and maximum distance in wide standing. The adjustability of the exoskeleton, including donning/doffing and length adjustment, was also used as a metric to evaluate the performance of an exoskeleton.²⁰

2. User-Response Measures

From a user’s perspective, the psychological, physiological, and neurophysiological responses are three types of crucial measures for evaluating the efficacy of the exoskeletons. Most of those measures are related to exertion, fatigue, and discomfort, which are key elements in the development of pain and potentially culminates in MSDs.^21–23

a. Psychological Measures

According to the latest definition, fatigue is defined as subject sensation and can only be measured by self-report.²⁴ Thus, users’ perceptions and feelings are first-hand important measures for exoskeleton assessments. Ratings of pain, soreness, and discomfort were suggested to assess an exoskeleton at different areas of the users’ bodies.^25,26 A previous study used a five-point Likert scale to evaluate the perceived effort, safety, comfort, efficacy, installation, and intention to use.²⁷ One back-assist exoskeleton was selected to use by participants, because it received the highest and only positive score of intention-to-use in a special patient-handling procedure (i.e., prone positioning for patients with severe COVID-19–related acute respiratory distress syndrome).

Another study used a visual analog scale (VAS) to rate exoskeletons, which was considered to allow a finer distinction between participants’ opinions by reducing the variation of individual interpretation compared to numerical rating scales.^20,28 Furthermore, different exoskeleton users may have different expectations or tolerance for discomfort brought by wearing an exoskeleton. Medical exoskeleton users—who may not be able to regain certain abilities (e.g., walk or grasp) without the assistance of the exoskeletons—often focus more on functionality rather than comfort level. In contrast, industrial exoskeleton users often have a much lower tolerance to exoskeleton-associated discomfort—any level of discomfort, even minimal, might hinder user acceptance.²⁹ Thus, to assess industrial exoskeletons, psychological measures (e.g., user acceptance) are important components and expected to be investigated in a long-term application. The use of a passive back-assist exoskeleton was investigated over four weeks in automobile manufacturing, finding a significant concern expressed by workers about discomfort at the chest.³⁰

The exoskeleton user and developer communities have recognized the importance of exoskeleton fit and comfort.³¹ Specifically, the algometry method has been applied for assessing pressure tolerance and recommendations for acceptable contact pressures between exoskeleton components and the user’s body regions.³² In this method, forces are applied to a human participant with a pneumatic actuator transferred by standard point-algometry surfaces (1 cm², 20 cm², and with thigh and pelvic pads and chest strap components commonly used in exoskeletons). The participant is subjected to gradually increasing contact pressure until they indicated an unacceptable level of discomfort and the onset of pain. Pain onset thresholds were often twice that of the onset of discomfort, and regional anatomical differences in pressure tolerance were notable (e.g., chest region being more sensitive to the onset of discomfort than the thigh region and the pelvis region being the least sensitive).

b. Physiological Measures

There are various favorable physiological measures for different tasks, which include the global physiological metrics (e.g., metabolic cost and cardiac dynamics) and the local metrics (e.g., activity and tissue oxygenation of working muscles). Metabolic cost is commonly used to evaluate the efficacy of exoskeletons during prolonged physical tasks, such as evaluating the effectiveness of a lower-extremity exoskeleton during gait. Similar to the industrial upper-body exoskeletons’ evaluation, the metabolic cost was used to evaluate a passive back-assist exoskeleton during a 5 min repetitive lifting task, where it was found that metabolic cost was significantly reduced by 18% while wearing the exoskeleton compared to the no-exoskeleton condition.³³

In a recent study, significant reductions in metabolic parameters (e.g., oxygen rate) while wearing exoskeletons were also found during sustained static and semi-static overhead tasks.¹⁸ In the same study, heart rate significantly decreased by 5–6% for the exoskeleton conditions. Heart rate and oxygen consumption appeared to show a larger reduction when comparing the exoskeleton condition to the no-exoskeleton condition.¹¹ Muscle activity of working muscles, measured by surface electromyography (EMG), is one of the popular metrics to assess industrial exoskeletons. In an overhead task, wearing a passive upper-limb exoskeleton significantly reduced shoulder muscle activity and did not affect lower back muscle activity.¹¹ During a simulated assembly task, with the trunk in a forward-bent position and a static holding task, lower back muscle activity significantly decreased when subjects wore a passive back-assist exoskeleton.¹⁷ During symmetric and asymmetric lifting, a passive back-assist exoskeleton significantly reduced peak and mean EMG in the back and leg muscles.³⁴ Near-infrared spectroscopy (NIRS) could be another promising measure to assess users’ responses related to fatigue or pain, as it quantifies localized tissue oxygenation changes and provides a sensitive and complementary measure to monitor the progression of fatigue during physically demanded tasks.²¹ The occurrence of pain appears to be related to decreases in tissue oxygenation during the circumferential compression, which simulates the pressure that soft exoskeletons exert on the wearer’s body.³⁵

c. Neurophysiological Measures

In addition to traditional physiological measures, increasing attention was paid to neurophysiological metrics using neuroimaging, such as electroencephalography (EEG) or functional near-infrared spectroscopy (fNIRS), to assess work parameters and fatigue during working tasks.³⁶ For example, EEG-derived motor-related cortical potential revealed the role of cortical motor commands on the control of voluntary muscle activations.³⁷ Several fNIRS studies reported a significant decrease in relative levels of oxygenated hemoglobin in the prefrontal cortex (PFC) at exhaustion during submaximal and maximal fatiguing contraction, accompanied by muscular impairment. This suggests that the PFC may be one of the critical brain regions to monitor fatigue.^38–40 With more advanced analysis methods and neuroimaging technology, neurophysiological metrics appeared to provide additional information that traditional measures might not be able to detect. For example, EEG- and EMG-derived corticomuscular coupling measures were affected by attention and precision of exerted forces, despite no significant changes observed in traditional measures, such as EMG and force production, during a sustained motor contraction task.⁴¹ In a fNIRS study that measured fatigue-related functional connectivity across frontal and sensorimotor areas, neural adaptation strategies, indicated by motor steadiness, were found to be obesity- and sex-specific, which was not revealed in the conventional indicators of fatigue (e.g., endurance and strength loss).⁴²

Recently, attempts have been made to assess the effectiveness of industrial exoskeleton applications using neurophysiological measures and analysis. When the subject wore a passive back-assist exoskeleton during asymmetric lifting tasks, the fNIRS-based brain connectivity analyses were used to investigate the neural cost of human-exoskeleton interaction. The exoskeleton appeared to impose greater neurocognitive and motor adaptation efforts by engaging action monitoring and error processing brain network.⁴³ Although there are only a few studies to evaluate the effectiveness of exoskeletons using neuroimaging techniques, the previous findings shed light on using psychophysical metrics to evaluate the effects of exoskeletons on both physical and cognitive demands during designated working tasks. Using portable neuroimaging techniques as assessment tools may have advantages, such as providing “cerebral” objective measures that reflect psychological responses (e.g., the sensation of fatigue or acceptance of intervention); evaluating the physical parameters without interfering with assisted areas (in contrast, EMG sensors might need to be placed on working muscles where exoskeletons attach); and potentially assessing concurrency of physical and cognitive work at naturalistic work settings (where industrial workers are always exposed to combined physical and cognitive demands—never either one in isolation).

C. In Silico Approach

Biomechanical and computational modeling and simulation [Fig. 1(c) and Table 1] are powerful tools to evaluate the effects of exoskeletons on muscle forces and joint loads, which are difficult or impossible to be measured during in vivo testing or plausibly estimated from in vitro testing. There are two common approaches to estimate muscle forces and joint loadings: (i) the EMG-driven forward dynamics approach and (ii) the inverse dynamics approach. Depending on different modeling approaches, the inputs to models often include in vivo measurements, such as kinematics, muscle activity, or external forces. With appropriate verification and valuation process, the in silico approach is able to not only help develop and optimize product design before building physical prototypes, but also help evaluate the performance of the exoskeleton—both assistive and “iatrogenic” effects—under various conditions, even if under extreme situations when in vivo testing cannot be safely performed.

Numerous modeling work has been conducted for designing and evaluating lower-extremity exoskeletons.^44–46 As for upper-body exoskeletons, there are a few related modeling studies. In one study, biomechanical modeling and simulation were used to help design and evaluate a spine-inspired continuum soft exoskeleton, with disc compression/shear forces and muscle forces quantitatively analyzed in the simulation.⁴⁷ Similarly, a shoulder-assist exoskeleton was designed using a model-based approach for assisting with manual patient transfer in a surgery waiting room. The exoskeleton design was then optimized and evaluated using the same simulation framework, which later showed reduced joint compression and muscle forces in the shoulder complex during selected patient transfer scenarios.⁴⁸ In addition to the assisted joints, models can also be used to quantify biomechanical loads to joints where exoskeletons are not specifically designed to support. As a relevant example, a dynamic EMG-assisted spine model was used to evaluate loadings of the lumbar spine while wearing a vest with an attached non-anthropomorphic mechanical arm for supporting powered hand tools. This steady-cam vest system, which is technically not considered as a contemporary anthropomorphic shoulder-assist exoskeleton, was found to transfer mechanical loads to the lower back and increase compressive spinal loads.⁴⁹

III. DISCUSSION

This review primarily focuses on the existing viable and promising evaluation and test methods of industrial exoskeletons and summarizes them into three categories: in vitro, in vivo, and in silicon (Fig. 1 and Table 1). Although it is expected that not all evaluation methods would be used at once in a study, strategically combining different approaches may provide more comprehensive evaluations for various exoskeleton products. Furthermore, despite the promising results from the existing evaluation methods, we are aware of the limitations and challenges in different types of evaluation approaches.

The in vitro approach, using test dummies, can provide standardized tests to evaluate the performance and functions of various exoskeletons without being influenced by “human” factors or risking the subject’s safety. However, it still cannot replace in vivo human subject testing, which can reflect real users’ feelings and responses. Additionally, to obtain reliable and detailed interaction force or actuator forces of exoskeletons during in vitro testing, we expect that future test dummies will be “smarter” and equipped with more advanced sensors and sophisticated controls.

The in vivo approach appears dominant in existing exoskeleton assessment methods. However, while most evaluations were tested in laboratories, there are a number of inherent disadvantages to laboratory testing. For example, the subjects are often selected from a nonrepresentative working population (e.g., inexperienced workers) who have no performance pressure or productivity demands. Study populations tend to be homogeneous and do not reflect anthropometric and cultural/ethnic differences seen in industry.⁵⁰ They are also often asked to perform fewer working cycles of less duration and can return to a neutral posture more frequently after experimental trials. Additionally, there is often not enough training or practicing time for subjects to adjust to exoskeletons, because testing sessions typically last only a few hours. Similar to other interventions, assistive devices may offer a short-term benefit but may harm the user in a long-term application. Therefore, more comprehensive studies, specifically long-term studies, need to be conducted to help us fully understand how exoskeletons can maintain and/or improve human performance throughout an entire day and for multiple days.

There are several limitations in different types of in vivo measures. For example, for the localized measurement tools (e.g., EMG or NIRS on muscles), the placements of sensors often interfere with the areas where the exoskeleton covers, which could affect recordings on these sensors. Additionally, the interaction forces (i.e., among exoskeletons, users, and objects to be worked on) may be difficult to measure, especially for soft exoskeletons There are more complex psychological factors associated with different backgrounds in human subjects and social or cultural influence from various industries, which are often hard to control and may affect evaluation results when the effects of exoskeletons in acceptance, usability, or performance are studied. Moreover, new applications of exoskeletons bring new challenges to evaluation approaches and methods. When exoskeletons are used by healthcare workers, many aspects of human-centered care work must be taken into consideration, such as the effects of exoskeletons on patient’s feelings and caregiver–patient interactions.⁵¹

The in silico approach is viewed to have many advantages to investigate “what-if” questions, without building physical models or performing human subject experiments. However, limitations occur in different modeling approaches and numerous assumptions and simplifications in modeling and simulations are inevitable. In general, simplified geometries of musculoskeletal structures are often assumed in biomechanical models and the subject-specific material properties or characters of those structures are difficult to obtain. In addition, the errors in testing data will be carried over to the modeling results, not to mention missing experimental data in some situations (e.g., interaction forces among exoskeleton, user, and working objects or “unreachable” deep muscles’ activity). Generally, modeling is not reliable to predict absolute magnitudes of the biomechanical loads but it is good for analyzing general trends or effects of different biomechanical parameters in musculoskeletal loadings.

To evaluate exoskeletons under realistic jobsite conditions, it is required to look into evaluation approaches and systems with less interference to users’ work, such as markerless motion capture systems or video-based joint-loading estimation methods. With advancements in science and technology, next-generation exoskeletons may integrate new technologies (e.g., artificial intelligence, augmented reality, virtual reality, or the internet of things) and the evaluation methods will need to evolve accordingly. In summary, this review sheds light on choosing appropriate evaluation approaches and may help with decision-making during the development, evaluation, and application of industrial exoskeletons.