Functional assessment and performance evaluation for assistive robotic manipulators: Literature review

Abstract

Context

The user interface development of assistive robotic manipulators can be traced back to the 1960s. Studies include kinematic designs, cost-efficiency, user experience involvements, and performance evaluation. This paper is to review studies conducted with clinical trials using activities of daily living (ADLs) tasks to evaluate performance categorized using the International Classification of Functioning, Disability, and Health (ICF) frameworks, in order to give the scope of current research and provide suggestions for future studies.

Methods

We conducted a literature search of assistive robotic manipulators from 1970 to 2012 in PubMed, Google Scholar, and University of Pittsburgh Library System – PITTCat.

Results

Twenty relevant studies were identified.

Conclusion

Studies were separated into two broad categories: user task preferences and user-interface performance measurements of commercialized and developing assistive robotic manipulators. The outcome measures and ICF codes associated with the performance evaluations are reported. Suggestions for the future studies include (1) standardized ADL tasks for the quantitative and qualitative evaluation of task efficiency and performance to build comparable measures between research groups, (2) studies relevant to the tasks from user priority lists and ICF codes, and (3) appropriate clinical functional assessment tests with consideration of constraints in assistive robotic manipulator user interfaces. In addition, these outcome measures will help physicians and therapists build standardized tools while prescribing and assessing assistive robotic manipulators.

Introduction

The research and development of assistive robotic manipulators, including wheelchair- and desktop-mounted robotic manipulators, can be traced to the 1960s. Over the past 50 years, nearly a dozen assistive robotic manipulators have been developed and evaluated for their performance in usability and functionality. Different user interfaces have been designed for each of these assistive robotic manipulators to improve the performance of accomplishing functional activities of daily life (ADLs). The involvement of user experiences and feedback from the target population has kept the design and development progressing. In addition, with the benefit of increasing computational power, more research groups have developed automation and artificial intelligence for object recognition and path planning so that people with disabilities may perform ADLs and vocational tasks more independently and efficiently. However, despite of these attempts, only few commercialized assistive robotic manipulators are currently available on the market.

According to the most recent studies, the number of people with disabilities in the United States who could benefit from using an assistive robotic manipulator is estimated at 150 000, which is 0.06% of the population. This population includes people with muscular dystrophy (MD), spinal cord injury (SCI), spinal muscular atrophy, multiple sclerosis (MS), amyotrophic lateral sclerosis, cerebral palsy (CP), rheumatoid arthritis, postpolio syndrome, locked-in syndrome, and other severe motor paralysis.¹ Also, the number of people in the United States over age 65 years will double from 34.7 million to 69.4 million by 2030.² As these people begin to show degenerative symptoms, the need for assistance in object manipulation will increase.

User needs for assistive robotic manipulators were acquired from interviews with target populations. We reviewed eight studies and surveys of people with disabilities, from which we compiled the following design requirements.^3,4

The location was suggested to be wheelchair-mounted with an inconspicuous parked position and no occlusion of user's vision, front space, and steering. The manipulator has to assert force larger enough to lift books and open doors, reach to the floor and high shelves, and move rapidly. In terms of functionality, stored positions and repetitive movements were suggested, such as stirring, interrupts for manual fine control, and simultaneous multiple-joint control. Aesthetic design and auxiliary gripper were also suggested.

The successful commercialization of assistive robotic manipulators depends on reliability, cost-efficiency, appearance, functionality, and usability. Reliability and cost-efficiency are factors that can be defined and evaluated by design requirements.⁵ However, the appearance, functionality, and usability require applying the concept of user-centered design. User-centered design is defining and evaluating design requirements with the participation of end-users.⁶ The goal of functionality is to duplicate the functionality of a human's arm. Most assistive robotic manipulators have seven degrees of freedom (DOF), including the gripper, to achieve the DOF of human upper extremities (if neglecting fine finger movement). Usability, including the user interfaces and automation, is designed to facilitate the users in accomplishing tasks in an acceptable time period.⁵

Safety is another critical factor. Risk of injuries may increase while the robotic manipulator is mounted on the wheelchair or desktop and the user is inside the working space of the manipulator. Complexity in user interfaces is another safety factor. Thus, the Multiple-Master, Multiple-Slave (M3S) safe communication concept design is suggested. This M3S concept was suggested to be adopted by integrating the Controller Area Network (CAN) communication protocol into robotic manipulator systems.⁷

In almost all studies, dimensional and kinematic requirements were the first considerations for meeting user needs. For example, a 7 degree-of-freedom wheelchair-mounted robotic manipulator (WMRM) called FRIEND II at the Institute of Automation, University of Bremen, was designed as a personal assistant in ADLs and working environments.⁸ Three different mounting location manipulators: the desk-based “Wolfson” robot, the trolley-mounted “Wessex” robot, the wheelchair-mounted “Weston” robot, developed by Bath Institute of Medical Engineering;^9,10 and a 9 DOF WMRM with Cartesian control developed and optimized by the University of South Florida in fulfilling kinematic and dynamic requirements for ADL tasks like opening and holding a spring-loaded door.^11–15

Two commercialized assistive robotic manipulators (Manus ARM and JACO manipulator, shown in Fig. 1) are both WMRMs with more than six DOF and minimized fold-in position. The Manus ARM (Assistive Robotic Manipulator) has a two-finger gripper manufactured by Exact Dynamics (Didom, The Netherlands), which also manufactures an updated version called iARM. It can be controlled by keypad, joystick, or single-button switches through a CAN bus to meet the M3S safety concept.^16,17 Alternatively, the JACO manipulator has a three-fingered hand manufactured by Kinova Technology (Montreal, Canada). The hand can grasp objects using either two or three fingers. It can be controlled by its own 3-DOF joystick.^18,19

Figure 1.

Two commercialized wheelchair-mounted robotic manipulators: JACO manipulator (left) and Manus ARM (right).

The Personal Mobility and Manipulation Appliance (PerMMA) by the University of Pittsburgh is the first wheelchair to integrate bimanual manipulation for enhancing the quality of life for people with severe physical impairments.²⁰ PerMMA utilizes two WMRMs on a novel mounting system to enhance its manipulability and mobility.^21,22

The development work of PerMMA has shown that robot performance effectiveness became a major concern for consumers. Performance primarily relates to the human–machine user interface, which makes the connection from the user's intention to the robot actuation. If the interface is difficult to learn and use, the task performance will not be effective – for example, some tasks, such as eating, require a time limit. The pleasure would be reduced if conveying food from the plate to user's mouth was so slow that the food becomes cold. Moreover, the user should not need to focus on how to control the robot. Instead, the user should only need to focus on the task on hand. Distributing the cognitive load and robot control between users and controllers became a major research question.²³

However, this cooperative control and sharing work loading highly depends on the task to be performed. For example, if the user retrieves a cup for drinking, the user may focus on the control of getting more or less liquid instead of retrieving the cup, aligning the cup to mouth, and putting the cup away. To classify the ADL tasks, we include the concept of the International Classification of Functioning, Disability, and Health (ICF) here.

The ICF, released by the World Health Organization in 2001, provides a comprehensive view of health status from different perspectives: Body Functions, Body Structures, Activities and Participation, and Environmental Factors. Body Functions and Structures express physiological functions of body systems and anatomical elements such as organs, limbs, and their components. Activities describe the execution of a task or actions by an individual. Participation is the involvement in life situations. Environmental Factors comprise physical, social, and attitudinal features.²⁴

The primary objective of this review was to explore quantitative and qualitative ADL performance evaluation measurements in assistive robotic manipulators while developing user interfaces. The secondary objective was to compare the user's needs and performance evaluation under the classification of ICF. The associated ICF codes to these measurements and needs are also included as classifications of the key ADL functions. The findings will not only help researchers in making decisions using appropriate tools for evaluating performance during the user-interface development phase, but also help develop new studies for the uncovered ICF functions. Moreover, this also provides manufacturers, physicians, and therapists with an overview of performance evaluation options used for training and identifying the improvement for ADLs and vocational rehabilitation.

Method

Literature search

We conducted a systematic search of articles from 1970 to 2012 in Pub MED, Google Scholar, and University of Pittsburgh Library System-PITTCat. We used the following keywords in the literature search: wheelchair mounted; desktop mounted; rehabilitation robotics; assistive robot; assisting manipulator; Manus arm; JACO arm; iARM; clinical evaluation; performance; robotic manipulator; upper limb; upper extremity impairment; functional assessment; disability. Both forward and backward search strategies were used to find related references. In the forward search, we also searched the literatures that have references to the identified literatures. The backward search was used to explore the related references in the identified literatures. This forward and backward search was continuously conducted until no new literature was identified.

Study selection

To be selected for further review, a study had to fulfill at least one of the following criteria:

1. At least one clinical trial was conducted (i.e. compare user's performance with or without assistive robotic manipulators), controlled trial (i.e. clinical trial with a control group, either randomized or not), or interview with consumers (i.e. survey of task priorities, either pre- or post-development).

2. Wheelchair users or people with experiences caring for wheelchair users including caregivers, family members, or clinical practitioners were involved.

3. ADL tasks were used as evaluation.

4. A performance evaluation or questionnaire in the outcome measures was provided.

5. Full-length publication in English language in a peer-reviewed journal was required.

Studies on the performance assessment of desktop-mounted robotic manipulators for performing ADL or vocational tasks were also included. To include the most complete scope of the current literature, we did not limit the search by type of disabilities, user interfaces, or country.

Data extraction

We analyzed the contents from the selected studies with a structured matrix. The structural items we used for filling this matrix is the following: descriptive information of the subjects and target population, user interface(s) implemented in the study, outcome measures for performance evaluation or preference, statistical methods for demonstrating clinical evidences, and relevant outcomes based on the results.

International Classification of Functioning Code

In addition, to demonstrate the clinical relevance of assistive robotic manipulators, ICF codes²⁴ related to the functional assessment and performance evaluation used in the selected studies were also included to indicate the specific functions in the selected studies.

Results

Study selection

From the systematic literature search, we identified 20 studies (Table 1). Of these 20 were studies from the group that implemented user interfaces for the assistive robotic manipulators. These publications included several consecutive clinical trials and summaries of those clinical trials and often used the same subjects. They were separated into two broad categories: user task preferences and user-interface performance measurements of commercialized and developing assistive robotic manipulators. The subsets of articles in each category are discussed in more detail below.

Table 1.

List of the selected studies with tested robotic manipulator, user interfaces and participants

Study	User interface	Robot	Participants
Corker26	Joystick	Golden Arm	n = 3 (SCI, Guillain Barre, and MS)
	Tongue-actuated switches
Hammel et al.27	Voice control	Desktop vocational assistant robotic workstation	n = 24 (SCI)
Bach et al.28	12-key keypad for joint control	Industrial robot	n = 6 (DMD)
	2 joysticks with scanning command selection	Cobra RS2
		Microbot 453-H
Buhler29; Bühler et al.30	Standard 4 × 4 keypad	Manus ARM	First test: n = 13
			MD: n = 2
			Spastic tetraplegia: n = 4
			Polyomyelitis: n = 1
			Intracranial pressure (ICP): n = 5
			Spina bifida: n = 1
			2nd test: n = 2
			(MD & Spastic tetraplegia)
Chen et al.31	Head-operate user interface	PerForce1 made by the Cybernet Systems Corporation	n = 6 (able-bodied)
Schuyler and Mahoney25	GUI interface with 3 modes: Joint, Program, and Procedure	Desktop-mounted UMI-RTX	n = 9 (CP)
	Computer access: intellikeys, WiViK scanning
Römer et al.32,33	Standard 4 × 4 keypad	Manus ARM	n = 13 (experienced)
	Wheelchair joystick		n = 21 (non-experienced)
Driessen et al.17	Wheelchair joystick with 3 modes: Cartesian, pilot, and joint modes and collaborative with camera on fingertip	Manus ARM	n = 4 (experienced)
Tijsma et al.34	Costumed GUI	Manus ARM	n = 4 (powered wheelchair users with weak upper limb strength)
Romer et al.33	Wheelchair joystick with 3 modes: Cartesian, pilot, and joint modes and collaborative with camera on fingertip	Manus ARM	n = 4 (experienced)
Laffont et al.1	GUI with panoramic camera	Manus ARM	n = 20
	Computer access: trackball, simple mouse, and head tracking		MD: n = 5
			Traumatic tetraplegia: n = 13
			Guillain-Barré syndrome: n-2
Tsui et al.; Tsui and Yanco; Matsumoto et al.2,36,40	Touchscreen to select object	Manus ARM	n = 12
	Computer access: touchscreen, touchscreen with key-guard, single switch scanning, and head pointer		Traumatic brain injury: n = 5
			CP: n = 6
			Spina Bifida: n = 1
Routhier and Archambault38	Standard 3-axis joystick	JACO manipulator	n = 22
			SCI: n = 11
			MD: n = 5
			Others: n = 7
Maheu et al.18	Standard 3-axis joystick	JACO manipulator	n = 31
Kim et al.35	Manual and autonomous mode	UCF-MANUS	n = 10 (SCI)
Cooper et al.22	Manual mode using touchscreen	PerMMA	n = 15 (power wheelchairs users with upper and lower extremity impairments)
	Teleoperation mode using Phantom Omni haptic joystick

DMD, Duchenne muscular dystrophy.

Discussion

In the user task preferences section, one article was reviewed to provide the priority list of ADL tasks selected by target population, family members, or caregivers. In the user interface performance measurements section, 19 articles were reviewed. User interfaces included are keypad, joystick, voice control, touch-screen, haptic device, vision-based autonomous control, and computer access methods such as mouse, single switch with scanning, or head pointer.

User task preferences

User preferences in ADL tasks with assistive robotic manipulators reflect consumers’ main concerns. The user priority tasks are the primary indicators for developing user interfaces that allow users to perform the desired function.

A review article⁵ in 1994 summarized the users’ preference from nine surveys. Pre- and post-development surveys were conducted to investigate task priority of assistive robotic manipulators from potential users and to retrieve users’ feedback after using a specific robotic manipulator. More than 200 potential users – including people with SCI, MD, CP, MS, arthritis, polio syndrome, and other neuromuscular diseases – were interviewed in the nine studies.

During the pre-development surveys, 128 participants were interviewed (age 8–87 years, from 1986 through 1991) to select top five tasks from 14 to 80 items under the categories of Personal Hygiene, Domestic, Recreation, and Work/School as shown in Table 2. Table 2 shows that reaching, gripping, picking up objects from a shelf or the floor, eating and drinking, and preparing food were rated the most important tasks. Gardening, hobbies and crafts, and leisure activities were rated relatively high importance. Vocational tasks were rated low importance probably because of the high unemployment rate among the subjects (75–93%).

Table 2.

List of the top five preferred tasks from pre- and post-development surveys

	Pre-development survey		Post-development survey
Rank	Task	ICF codes	Tasks	ICF codes
1	Cooking, fixing food, drinks	d630 Preparing meals,	Work/school fetch & carry objects	d2100 Undertaking a simple task
		d6300 Preparing simple meals		d4301 Carrying in the hands,
		d6301 Preparing complex meals		d4400 Picking up
		d550 Eating		d4401 Grasping
		d560 Drinking		d4452 Reaching
				d4300 Lifting
				d4305 Putting down objects
2	Reaching, stretching, gripping, picking up objects	d2100 Undertaking a simple task	Personal hygiene	d510 Washing oneself
		d4301 Carrying in the hands
		d4401 Grasping
		d4400 Picking up
3	Gardening/hobbies and crafts/leisure	d6505 Taking care of plants, indoors and outdoors	Feeding or eat/drink	d630 Preparing meals
		d9204 Hobbies	Prepare meal	d6300 Preparing simple meals
		d9203 Crafts		d6301 Preparing complex meals
		d920 Recreation and leisure		d550 Eating
				d560 Drinking
4	Reach or pick up from the floor	d4452 Reaching	Communication/phone	d3600 Using telecommunication devices
		d4300 Lifting
		d4305 Putting down objects
		d4400 Picking up
5	Personal hygiene	d510 Washing oneself	Domestic opening doors, drawers, windows, closets, etc.	d4401 Grasping
	Dressing	d540 Dressing		d445 Hand and arm use
				d4450 Pulling
				d640 Doing housework

As shown in Table 2, during the post-development surveys, 65 individuals (age 6–65 years, from 1986 through 1993) were interviewed after they had experienced the developed prototypes. The results revealed an opposite order of preferences. In contrast to the pre-development surveys, the work/school fetch and carry objects were rated high, as were preparing food and personal hygiene (Table 2). One interesting finding was that tasks rated high priority were generally the tasks subjects were unable to do such as food preparation and personal hygiene. The other interesting finding was that the tasks that participants foresee they will be unable to do because of their degenerative disease was also rated moderately high, such as drinking or holding the phone.

Because the target population is dependent on the medical, vocational, and support resources, it is also important to discover the perspectives from the medical personnel and professional agencies that authorize the purchase, and the attendant caregivers and family members who perform set-up and auxiliary care. Family members rated eating and drinking tasks of higher importance, but the task of pick-and-place objects was not identified as important. Similarly, caregivers rated drinking from a straw, pouring liquid, brushing teeth, operating a CD player, playing video games, typing on a keyboard, and turning the computer on and off of much higher importance than did the users. Lower importance was reported for the tasks of drinking from a glass, opening doors, gardening, and manipulating printouts. Experienced occupational therapists rated the following tasks as highly important: cooking and meal preparation, feeding, lifting and carrying heavy objects, dressing, grooming, hygiene, turning, transfers, shifting weight, and elimination functions. These non-user surveys revealed different priorities from those listed by people with disabilities. However, the tasks selected by non-users also must be taken into consideration during development because they may affect the acceptance of the robotic manipulator; furthermore, these may be the tasks with major burdens in ADLs. This study provided broader user preference for researchers in developing user interfaces for assistive robotic manipulators.⁵

It is noteworthy that the manipulation within work/school environments is rated high after experiencing assistive robotic manipulators. This item was rated low in the surveys before usage and non-user surveys. This suggests that users see the enhanced assistance as a way to achieve their vocational/academic goals. However, the question is how many work opportunities and what kind of job users with WMRM are capable of performing with the enhanced robotic manipulation assistance. To answer this question, one study shows that the numbers and types of jobs increase significantly with only nominal increases in manipulation ability.²⁵ Although there are other factors in the vocational accommodation process, individuals with severe manipulation disabilities may acquire more job opportunities once they have the use of assistive robotic manipulator.

User interface performance measurements of commercialized and developing assistive robotic manipulators

The University of California conducted a long-term study with three participants, one with SCI, one with Guillain Barré syndrome, and one with MS (two men and one woman, age: 47–54 years). The control interface of the Rancho Los Amigos (Golden Arm) Manipulator for two participants was a proportional joystick that sequentially drove each joint motor. The other participant with SCI used tongue-actuated switches. Performance was evaluated using a modified peg-in-hole test. The test was to put different shapes of blocks into an associated hole on the lap tray with constant tolerance and trajectory distance. A Fitts’ Index of Difficulty (Id) measure was utilized for standardization:

The practice task was to bring a stand and a book on the lap tray where it was reachable with a mouth-stick. The completion time of every hour of the peg-in-hole test was recorded to demonstrate the learning effect. During the long-term study, an integrated circuit system recorded the frequency of daily usage to reflect the objective view of the usefulness of the manipulator in ADL. The problems were also investigated periodically with a Critical Incident Technique interview.

Participants were able to complete the peg-in-hole test within 4 minutes after using the manipulator for 13 hours. The number of control commands to complete a task was highly correlated with the total task time (Pearson's r = 0.91). Significantly, more horizontal and vertical movements were reported than rotational movements.²⁶

Voice control can be useful for individuals who have difficulty using joysticks or keypads. A study of 24 individuals with high-level tetraplegia (C1–C5, age 20–73 years) from the Palo Alto Veterans Affairs Spinal Cord Injury Center evaluated a desktop vocational assistant robotic workstation. Participants were asked to prepare a meal, feed themselves, wash their face, shave, and brush their teeth using trained voice recognition interface. Tasks were rated on a three-point scale. Measurements of performance were recorded with in-house designed pre- and post-questionnaires, interviews, and observer assessments during training and evaluation. Task completion time was recorded during every task:

1. Prepare soup and feed self: 9:24 ± 1.89, range 7:00–13:00 with 78% reported satisfied and 22% neutral.

2. Brush teeth and rinse: 5.25 ± 2.33, range 1:54–9:00 with 95% reported satisfied and 5% neutral.

3. Shave face: 9:82 ± 4.98, range 4:31–14:00 with 62% reported satisfied and 38% neutral.

4. Wash and dry face: 8:00 ± 1.73, range 7:00–10:00 with 73% reported satisfied, 20% neutral, 7% dissatisfied.

The comparison between pre- and post-questionnaire showed the shift of acceptance from undecided to accepted.²⁷

A long-term study of six participants with Duchenne muscular dystrophy referred from University Hospital (age 19–28 years, mean age: 24 years) was conducted by the University of Medicine and Dentistry of New Jersey. The participants were all wheelchair users without functional movement of the shoulder and elbow, but with finger excursion for pressing push buttons (three used ventilators). Two types of industrial robotic manipulators, the Cobra RS2 manipulator (Cobra, Darmstadt, West Germany) and the Microbot 453-H manipulator (Movemaster, Mountain View, CA, USA), were mounted on a stand fixed to the lap tray. Two kinds of user interfaces modified from the industrial control panel were evaluated: keypad and joystick. The keypad consisted of 12 touch-sensitive buttons on a circuit board (4″ × 5″) to control the motors geared to shoulder, elbow, wrist, and terminal device functions. The joystick user interface consisted of two joysticks and two toggle switches. One joystick and two toggle switches were designed for selecting commands using scanning mode, and the other joystick was used to activate the robotic manipulator. The participants used the WMRM for 1.5–6 years (3 years on average). The measurement includes the time of daily usage (5–12 hours/day, 8.6 hours/day in average), eating time (1–2 hours using keypad, 1.5–2.5 hours using joystick with scanning selection, 1–2 hours with attendant assisted), and average reduced caregiving time (2–4 hours/day, 3 hours/day in average). Participants were also interviewed about frequently used ADL tasks (Table 3). The three most important tasks of the WMRM usage were assistance with eating, manipulation of remote and environmental control devices, and recreational activities. The capability to scratch oneself was also found to be important. Participants’ family members noticed improved independence.²⁸

Table 3.

List of tasks from long-term study survey

Tasks	ICF codes	Usage rank
Eating pre-cut slices of bread, hot meals and soup (using modified utensil such as a Z-shaped spoon and a cup rather than a bowl)	d550 Eating	1
	d560 Drinking
Environmental control system operation including remote controls	e1151 Assistive products and technology for personal use in daily living	1
	e1250 General products and technology for communication
Recreational use such as electronics, carpentry and miscellaneous activities (drilling holes in electronic printed circuit boards, soldering and sandpapering wooden components for ship models)	d9204 Hobbies	2
	d9203 Crafts
	d920 Recreation and leisure
Pages turned (rubber tip on gripper)	N/A	3
Use of light switches (highly placed switches were reached by placing a stick in the terminal device)	e1151 Assistive products and technology for personal use in daily living	4
Opening doors (by coordinating manipulator with power wheelchair)	d4401 Grasping	5
	d445 Hand and arm use
	d4450 Pulling
	d640 Doing housework
Books manipulated	d2100 Undertaking a simple task	6
Telephone use	d3600 Using telecommunication devices	6
Use of electrical outlets	N/A	6
Use of electric shaver	d5202 Caring for hair	7
Independence with elevator buttons	d4601 Moving around within buildings other than home	7
Manipulation and use of cups and glasses	d4400 Picking up	8
	d4401 Grasping
	d4452 Reaching
	d4300 Lifting
	d4305 Putting down objects
	d560 Drinking
Manipulation of records and cassette tapes	d4400 Picking up	8
	d4401 Grasping
	d4452 Reaching
	d4300 Lifting
	d4305 Putting down objects
Drawer opening and object manipulation	d640 Doing housework	8
Coffee preparation	d630 Preparing meals	9
Use of modified water faucets	N/A	10

A study conducted by the Forschungsinstitut Technologie-Behindertenhilfe in Germany explored the capability of users with different disabilities to operate the Manus ARM after a short training period. The standard 4 × 4 keypad from Manus ARM was used for WMRM operation. Participants with different disabilities were recruited to perform simple test tasks of driving to a work position and building a tower of three wooden pieces. Five participants (age 20–40 years) were unable to finish this task within the requested time. Satisfactory results with Cartesian mode were reported (good: n = 3, medium: n = 4, bad: n = 6). A negative response in switching menus with the standard keypad led to refusal or rejection in most participants. Two of the most skilled participants were evaluated for ADL tasks with their own choices of typical tasks: taking care of oneself (e.g. shaving); eating, drinking, and pouring out liquid; opening doors and drawers; grabbing and handling objects; retrieving papers out of a file; and lifting up objects from the floor/ground. One participant with MD did finish all these tasks properly and quickly without major problems, while the other with spastic tetraplegia requested to operate the robot at a workstation. Although simplified keypad design was discussed in the study, no clinical results were reported with it.^29,30

A study was conducted to evaluate a head-operated user interface with force feedback to control a PerForce1 made by the Cybernet Systems Corporation. Six able-bodied (age: 20–40 years) participants were evaluated in performing a Fitts’ movement task. Three tasks were used to evaluate the user's performance. In the first task, participants were asked to touch two targets on a board in front with the head-stick or end-effector on the robotic arm for six trials. The completion time and Fitts’ Id were recorded. The Id was calculated as

The second task was a page-turning task in which the participant continuously turned five pages of a large book with either the head-stick or the robotic manipulator. The task completion time was recorded. The third task was to draw two diagonal straight lines by following an X mark on the paper. Linear error was used to determine the accuracy in control.³¹

An exploratory study was conducted by duPont Hospital for Children, Wilmington, with the use of standard occupational therapy assessment tests to measure the effective manipulation performance of individuals with disabilities using a desktop-mounted robotic manipulator (UMI-RTX). Nine participants with severe physical manipulation impairments (CP) were identified through local rehabilitation centers. Three modes of user interfaces were tested: joint, program, and procedure mode. Appropriate computer input method was selected by preliminary evaluation (6 with intelli-keys, 1 with WiViK scanning, and 2 with both). Three functional assessment measurements were used: Jebsen Hand Test, Block and Box Test, and Minnesota Rate of Manipulation Test (MRMT). In all the three functional assessments, the results showed that human–robot performance is significantly slower than test results by people with stroke or other disabilities in comparison with other studies. However, participants were not able to complete any of the tests if there was no assistive robotic manipulator used.²⁵

A study by the Dutch Institute for Rehabilitation Research (iRV) compared 13 long-term Manus ARM users with more than 4 years experience with 21 non-ARM users who have similar levels of impairments. The user interface used was either the standard 4 × 4 keypad or wheelchair joystick. Participation in ADL tasks was observed for 1 week every 3 months for 12 months and the average daily usage and assistance time was reported. Results showed that one participant applied the ARM for more than 4 hours/day, four participants for 2–2.5 hours/day, and eight users for less or equal than 2 hours/day, range from 0.6 to 3.7 hours/day. It showed that the ARM users perform 40% more ADL tasks than the other group. The Manus ARM was used average 2 hours/day (0.7–1.8 hours).^32,33

A vision-based interface with autonomous planning transfers the loading in positioning and fine adjustment to the computer. A study by TNO Science & Industry, The Netherlands, evaluated four experienced Manus users with the pre- and post-test in retrieving a colored cup located at a location not seen by the users. The wheelchair joystick was implemented with three control modes: Cartesian mode, pilot mode, and joint mode. A camera was mounted on the fingertip to provide visual feedback within a graphical user interface displayed on a 7-inch widescreen TFT display. A visual serving function was developed for guiding the WMRM toward the target cup. Rates of success and difficulties were reported. Participants reported the difficulties in pilot mode and operating the WMRM through the camera's view. All participants were able to finish the task with no difficulty using the visual serving function.¹⁷

Another study using the same graphical user interface was conducted by TNO and Delft University to evaluate Manus ARM with four powered-wheelchair users (one woman) with weak upper limb strength. The Manus ARM was mounted on a stand-alone support beside the wheelchair user with adjustment of arm speed and switching method by user's choice. Three tasks were used to evaluate performance: (1) to stack two cups on the table, then pick up a pen and insert into the piled cups using Cartesian, pilot, and collaborative modes; (2) to move two blocks into a with normal and adjusted center of rotation modes; (3) to pick up two pens located out of the user's sight with pilot and collaborative modes. The measurements included the number of mode switches, task time, Rating Scale of Mental Effort, and interviews of suggestion to new interface. Data were analyzed using 2 × 2 (method of mode switching: original and new; control modes: Cartesian and pilot) analysis of variance (ANOVA). Owing to limited trials for training and a small sample size, results showed no significance among these four conditions.³⁴

A multi-center study was conducted to evaluate the efficacy of a graphic user interface with a panoramic camera to identify out-of-sight objects to be retrieved by Manus ARM automatically. There were 20 participants recruited (seven women; mean age: 44 years, range: 26–67 years) from four physical medicine and rehabilitation units of French hospitals (Coubert, Reims, Berck sur Mer, and Garches), all members of the French Association for the Promotion of New Technologies for Disabled People (Approche). This group was compared with 24 able-bodied control participants (16 women; mean age: 33 years, range: 19–55 years). Participants were asked to grasp six objects previously placed around their wheelchair. Using the WMRM, they selected the objects through the graphic user interface using a computer access method they were comfortable with (12 with trackball, 6 with a simple mouse, and 2 with head tracking). The measurements used were global success rate, completion time in object selection, number of clicks, and satisfaction. Data were analyzed using repeated-measures ANOVA with the group (disability group, control group) as the between-subjects factor, the object/location (six possibilities) and the trial number (first, second, third), as the within-subjects factors. Significant higher success rate was found in control group (88.7% for control group and 81.1% for people with disability). Significantly higher completion time was found in the disability group (71.6 seconds) in comparison with the control group (39.1 seconds). Both groups showed no significant difference in number of clicks. A high satisfaction rate was reported.¹

A study by the University of Central Florida investigated the utility using the UCF-MANUS, a WMRM designed with two operation modes: manual and autonomous. Ten participants with SCI (mean age: 41.1 years, range: 25–54 years) were divided into two groups to compare the performance of two operational modes in the pick-and-place task for three weeks. Task completion time, number of clicks, command inefficiency, and planning inefficiency calculated from trajectory data were used to measure performance, user's effort, and efficiency. A modified Psychosocial Impact of Assistive Devices Scale (PIADS) and an interview were administered as assessment of satisfaction and responses of issues with user interfaces. Results showed a significant reduction in the number of clicks and task completion time in the auto mode group. The learning effect was found in this 3-week study. The average completion time and number of clicks were reduced more on the third week in the manual mode group. In contrast, the satisfaction scores in auto mode group were found to be slightly lower than in the manual mode group. The authors concluded that with auto mode, the tasks were performed easier and faster, but less satisfactorily.³⁵

The University of Massachusetts-Lowell conducted a study with 12 participants (four women; age 17–60 years; seven using manual wheelchairs and five using power wheelchairs) in testing the performance of a developed vision-based autonomous object-retrieving system. Among the participants, nine participants used a touchscreen, one used a touchscreen with key-guard, one used single-switch scanning, and one used a head pointer to select an object on the display in front of them. The task was to select an object of the researcher's choice and have the robotic manipulator recognize and retrieve the selected object from a shelf. Mood rating scales were recorded before and after a task session. A post-session questionnaire was conducted. Psychometric measurements were the pre- and post-session mood rating scales, post-session questionnaires, and a shortened version of the PIADS. Performance measurements were the time for user selection, gross motion, visual alignment, object identification, fine motion, grasping, and return of the object to the participant. User selection time included perception time (i.e. time in identifying the object location on the shelf and on the display), and motor time (i.e. time in selecting the object on the display). The autonomous system success rate was 65% in 198 trials. The average total time was 164.72 ± 61.71 seconds. However, in the comparison of the perception time and PIADS with cognition levels, the authors used one-tail t-test without any adjustment. Therefore, there might be risk in the inflation of type I error in the reported p value.^2,36,37

A study with 27 participants by the Center for Interdisciplinary Research in Rehabilitation and Social Integration (CIRRIS) evaluated the usability of the JACO robotic manipulator. Among these 27 participants, 22 participants (four women, mean age: 40 ± 16.4 years, years in wheelchair (WC): 16.5 ± 13.5 years) have completed the evaluation tasks. Six tasks were performed for the evaluation with the JACO arm mounted on the tabletop: grasping a bottle located on the left side on the table, grasping a bottle located on the right on a surface near the ground and bringing it on the table, pushing the buttons of a calculator, taking a tissue from a box on the table, taking a straw from a glass on the table, and pouring water from a bottle into a glass. Participants were asked to perform each task twice successfully and the success rate to achieve two trials was recorded. Perceived mental loading and importance were surveyed using a 4-point ordinal scale. A socio-demographic questionnaire was also used. The success rate for easier tasks was more than 95% and for tasks that are more difficult was more than 80%. On the perceived easiness scale, taking a tissue was the easiest and grasping a bottle from the ground and putting it on the table was the hardest. Pouring out bottled water into a glass was rated the highest importance and pushing a calculator's buttons was rated low importance.³⁸

A larger sample-sized (n = 31; mean age: 45.6 ± 14.7 years) study of the JACO manipulator reported a similar success rate, easiness, and importance. In addition, estimated time saved in eating, drinking, preparing meal, dressing, and washing was calculated from self-reported time.¹⁸

The PerMMA developed by University of Pittsburgh consists of manual, tele-operation, and autonomous operation modes. PerMMA, equipped with two Manus ARMs by using combinations of the three modes and two types of user interfaces: touchscreen for the local user and haptic devices (Phantom Omni by Geomagic (Morrisville, NC, USA)) for the teleoperation user.²² This study recruited 15 participants (six women; mean age: 42.9± 15.8 years) with both upper and lower extremity impairments and using power wheelchairs to evaluate the performance within a laboratory environment. Participants were asked to complete as many of five tasks independently using the touchscreen interface and in cooperation with a teleoperator. The five tasks were (1) retrieving a piece of tissue from a tissue box on a desk; (2) picking up a meal container with a flexible handler from a desk and putting it down at a predefined new location; (3) opening a microwave oven by pushing the door button; (4) retrieving a plastic cup and moving it close enough for the user to drink; and (5) retrieving a straw and putting it into a plastic cup, and picking up the cup and moving it close enough for the user to drink with the straw. Task completion times were recorded. An interview of preference of operation modes was conducted after finishing the tasks. Although the results showed that teleoperation mode was much faster, in the interview after performing all tasks participants indicated they preferred to operate PerMMA independently.

In summary, the evaluation tasks with ICF codes and measurements in these studies are presented in Table 4. Task completion time is the most commonly used measurement in the evaluation of user interfaces. Improvement may be caused by faster robot speed, shorter trajectories, learning effect, easier tasks, lager target size, easier grasp orientation, reduced mode change error, better computer access methods, or better sight of view etc. Therefore, it may not be directly related to deficiency or perceived difficulty in the movement. For example, grasping a bottle on the table without many objects around is much easier than grasping it deep inside the refrigerator. This integrated outcome measure would not be specific enough for researchers to determine which part makes the task difficult. Thus, to determine the performance of human–robot interaction, the following concepts may need to be taken into consideration.

Table 4.

List of the selected studies with outcome measures, ADL tasks, and related ICF codes

Study	Measurements	ADL tasks	ICF code
Corker26	Completion time of peg-in-hole test	Peg-in-hole test	d4452 Reaching
	Fitts Index of Difficulty		d4300 Lifting
	Usage in training		d4305 Putting down objects
	Number of control commands		d4400 Picking up
	Critical Incident Technique interview	Bring a stand and a book on the lap tray where is reachable with a mouth-stick	d4452 Reaching
			d4300 Lifting
			d4305 Putting down objects
			d4400 Picking up
Hammel et al.27	In-house designed pre- and post-questionnaires	Prepare a meal	d630 Preparing meals
	Interviews,	Feed themselves	d550 Eating
	Observer assessments		d560 Drinking
	Task completion time	Wash their face	d5100 Washing body parts
	3-point scale	Shave, and brush their teeth	d5202 Caring for hair
Bach et al.28	Time to finish a meal	Finish a meal	d550 Eating
	Reduced caregiving time		d560 Drinking
	Time of daily usage
Buhler29, Bühler et al.30	First test: Successful rate	Driving to a work position and building a tower of three wooden pieces within requested time	d2100 Undertaking a simple task
			d430 Lifting and carrying objects
			d4600 Moving around within the home
	Second test:	Take care of oneself (e.g. shaving)	d5202 Caring for hair
	Number of tasks finished	Eat, drink, and pour out	d550 Eating
	Interview		d560 Drinking
		Open doors and drawers	d4401 Grasping
			d445 Hand and arm use
			d4450 Pulling
			d640 Doing housework
		Grab and handle objects	d4401 Grasping
			d445 Hand and arm use
		Get papers out of a file	N/A
		Lift up objects from the floor/ground	d4300 Lifting
			d4305 Putting down objects
			d4400 Picking up
			d4452 Reaching
Chen et al.31	Completion time	Fitts’ movement task: to touch two targets on a board in front	d445 Hand and arm use
	Fitts’ Index of Difficulty (Id)		d4452 Reaching
	Linear error	Page-turning task: turned five pages of a large book	d2100 Undertaking a simple task
		To draw two diagonal straight lines by following an X mark on the paper	d3352 Producing drawings and photographs
Schuyler and Mahoney25	Jebsen Hand Test	Simulate feeding	d4453 Turning or twisting the hands or arms
	Block and Box Test (BBT) Minnesota Rate of Manipulation Test (MRMT)		d550 Eating
		Card turning	d4453 Turning or twisting the hands or arms
		Small object	d4300 Lifting
		Stacking checkers	d4305 Putting down objects
		Retrieve a can	d4400 Picking up
		BBT	d4452 Reaching
		MRMT place test	d4453 Turning or twisting the hands or arms
Römer et al.32,,33	Participation in ADL tasks observation	Eating	d550 Eating
	Average daily usage	Drinking	d560 Drinking
	Assistance time was reported.	Washing	d5100 Washing body parts
		Brushing teeth	d5201 Caring for teeth
		Removing objects from the floor or out of a cupboard	d4300 Lifting
			d4400 Picking up
			d4452 Reaching
		Feeding pets	d6506 Taking care of animals
		Operating typical devices such as a VCR	d2100 Undertaking a simple task
Driessen et al.17	Capability of finish the task	To retrieve a colored cup located at the location not seen by the users	d4300 Lifting
			d4400 Picking up
			d4452 Reaching
Tijsma et al.34	Number of mode switches	Pick up an upside down cup and put into another cup on the table, then pick up a pen and insert into the piled cups	d430 Lifting and carrying objects
	Task time		d4300 Lifting
	Rating Scale of Mental Effort (RSME)		d4305 Putting down objects
	Interview	Put two blocks in a box in front of the user	d4400 Picking up
			d4452 Reaching
		Pick up two pens located out of user's sight
Romer et al.33	Number of mode switches	To pick up an upside down cup and put into another cup on the table, then pick up a pen and insert into the piled cups	d430 Lifting and carrying objects
	Task time		d4300 Lifting
	RSME		d4400 Picking up
	Interviews		d4452 Reaching
			d4453 Turning or twisting the hands or arms
		To put two blocks in a box in front of the user	d430 Lifting and carrying objects
			d4300 Lifting
			d4400 Picking up
			d4452 Reaching
		To pick up two pens located out of user's sight	d430 Lifting and carrying objects
			d4300 Lifting
			d4400 Picking up
			d4452 Reaching
Laffont et al., 1	Global success rate	Retrieve an object	d4300 Lifting
	Completion time in object selection		d4400 Picking up
	Number of clicks Satisfaction.		d4452 Reaching
Tsui et al.; Tsui and Yanco; Matsumoto et al.2,36,40	Mood rating scales	Select an object by researcher's choice and the robotic manipulator recognized and retrieved the selected object from a shelf	d430 Lifting and carrying objects
	A post-session questionnaire		d4300 Lifting
	PIADS		d4400 Picking up
	Time for user selection, gross motion, visual alignment, object identification, fine motion, grasping, and return of the object to the participant		d4452 Reaching
	System success rate
Routhier and Archambault38	Success rate	Grasping a bottle located on the left side on the table	d430 Lifting and carrying objects
	Perceived mental loading		d4300 Lifting
	Importance		d4400 Picking up
	Socio-demographic questionnaire		d4452 Reaching
		Grasping a bottle located on the right on a surface near the ground and bringing it on the table	d430 Lifting and carrying objects
			d4300 Lifting
			d4305 Putting down objects
			d4400 Picking up
			d4452 Reaching
		Pushing the buttons of a calculator	d3601 Using writing machines
		Taking a tissue from a box on the table	d4400 Picking up
			d4452 Reaching
		Taking a straw in a glass on the table	d4400 Picking up
			d4452 Reaching
		Pouring water from a bottle into a glass	d560 Drinking
Maheu et al.18	Success rate	Same task as36	Same as above
	Easiness
	Importance
	Estimated time saved in eating, drinking, preparing meal, dressing, and washing
Kim et al.35	Task completion time	Pick-and-place task	d430 Lifting and carrying objects
	Number of clicks		d4300 Lifting
	Command inefficiency		d4305 Putting down objects
	Planning inefficiency		d4400 Picking up
	Modified PIADS		d4452 Reaching
	Interview
Cooper et al.22	Task completion time	Retrieve a piece of tissue from a tissue box on a desk	d4400 Picking up
	Interview of preference about operation modes		d4452 Reaching
		Pick up a meal container with a flexible handler from a desk and put it down at a predefined new location	d430 Lifting and carrying objects
			d4300 Lifting
			d4305 Putting down objects
			d4400 Picking up
			d4452 Reaching
		Open a microwave oven by pushing the door button	d640 Doing housework
		Retrieve a plastic cup and move it close enough for the user to drink	d430 Lifting and carrying objects
			d560 Drinking
		Retrieve a straw and put it into a plastic cup, and pick up the cup and move it close enough for the user to drink with straw	d430 Lifting and carrying objects
			d560 Drinking
			d630 Preparing meals

First, standardized ADL tasks are needed so that different research groups can replicate tasks. These standardized ADL tasks consist specific starting and ending positions, and different target sizes, etc. The specific starting and ending positions constrain the distance of the tasks. In this way, the idealized trajectories can be defined as the straight line from starting position to the ending position. These idealized trajectories are the most efficient path for completing the tasks. Different-sized targets facilitate the computation of Fitts’ parameter and difficulties in using different user interfaces.³⁹ In addition, the standardized ADL tasks can be utilized as a reference for training and evaluation procedures once the assistive robotic manipulators are prescribed.

Secondly, the ICF codes and domains help the researchers identify and develop the functioning of evaluation tasks. Although the ICF codes do not list all the tasks in detail, the domains and codes can be used as the basic functioning activities to be evaluated with assistive robotic manipulator in order to increase independence. Majority of the ICF codes in the evaluation tasks from reviewed studies are picking up, reaching, putting down, or lifting in the Mobility domain (Table 5). In a study that monitored an able-bodied person for 5 days, Lifting and putting down objects are the most frequent activities.⁴⁰ Accelerating and facilitating the control for these highly frequent motions would show significant improvement of entire task performance. On the other hand, simulated eating and drinking tasks in the Self-Care domain were also evaluated in some studies. These are the basic movements for completing complex work or tasks. In comparison with the priority list from per- and post-development and non-users, some highly rated activities in the Activity and Participation domain have not yet been explicitly explored. Those more complex tasks can be included for future assistive robotic manipulator studies.

Table 5.

List of number of reviewed articles that performed evaluation of the certain ICF code

ICF code	Number of reviewed articles (%)
d4452 Reaching	20 (18.02)
d4400 Picking up	18 (16.22)
d4300 Lifting	16 (14.41)
d430 Lifting and carrying objects	11 (9.91)
d4305 Putting down objects	8 (7.21)
d560 Drinking	7 (6.31)
d550 Eating	5 (4.50)
d4453 Turning or twisting the hands or arms	4 (3.60)
d2100 Undertaking a simple task	3 (2.70)
d445 Hand and arm use	3 (2.70)
d4401 Grasping	2 (1.80)
d5100 Washing body parts	2 (1.80)
d5202 Caring for hair	2 (1.80)
d630 Preparing meals	2 (1.80)
d640 Doing housework	2 (1.80)
d3352 Producing drawings and photographs	1 (0.90)
d3601 Using writing machines	1 (0.90)
d4450 Pulling	1 (0.90)
d4600 Moving around within the home	1 (0.90)
d5201 Caring for teeth	1 (0.90)
d6506 Taking care of animals	1 (0.90)

Finally, using widely accepted valid and reliable functional assessment tests help provide stronger clinical evidence in performance evaluation. In the reviewed studies, several function assessment tests²⁵ with modified protocols or subtests are evaluated but the robotic manipulator performed significantly slower with large variance. This might be due to the small sample size (n = 9), different types of user interface, cognitive ability, or lack of feedback and DOF limit in user interface in the assistive robotic manipulators. For example, in the modified MRMT, the subject has to flip the checker and align it into another hole. This motion is the combination of pick-and-place and peg-in-hole tasks that human can perform in seconds by taking the advantage of visual and tactile feedback, and movement with simultaneous rotation and translation. However, robotic manipulators do not provide force feedback, simultaneous movements in translation and rotation. Subjects can only perform the test based on the visual feedback and switching between different control modes, which make the test more difficult. Therefore, it is important to choose appropriate functional assessment tests for performance evaluation with taking the constraints of robotic manipulators into consideration.

Most studies follow the concept of user-centered design or “consumer in the loop” design. However, best practices would be not only to interact with the real end-user, but also the extended users such as family members, therapists, physicians, administrators, caregivers, and others who would influence the usage of the newly developed technologies. Assistive robotic manipulators may interact with tangible objects most of the time; however, intangible interaction with these indirect users and discussion of topics such as social aspect or aesthetics would give researchers broader point of views for their design or development.²³

Conclusion and future work

This review article addressed the development and evaluation of assistive robotic manipulators including desktop- and wheelchair-mounted robotic arms. The list of desired tasks from target population during the pre- and post-development, non-users like family members and caregivers, and end-users in long-term use studies was discussed and provided with associated ICF codes. The performance evaluation measurements were discussed. The associated ICF codes of the tasks used for evaluation are also reported.

Using the ICF codes as a reference reveals the insufficiency of evaluation tasks in current assistive robotic manipulators studies. Further improvement can be shifted of focus on functioning activities, such as drinking, vocational-related work, or simple housework other than pick-and-place tasks or evaluations using clinically valid and reliable outcome measures that are relevant to these ICF codes.

Technological development has made assistive robotic manipulators more efficient and simpler to use, but there is still much room for further development and improvement. One major improvement would be to develop a two-way user interface between higher dexterity WMRMs that could be operated by fewer DOF from end-users. Vision-based interfaces with autonomous path planning demonstrate tremendously reduction in user loading of adjusting and twisting the end-effector to the desired grasping position. It would also reduce the required DOF for pick-and-place tasks, because only few clicks are needed to finish a complex sequential task. However, users’ participation in accomplishing the task was also taken away, which leads to lower satisfaction.

Task completion time is not a specific outcome measure to performance due to influence by many factors. Another improvement could be the development of standardized ADL tasks for the quantitative and qualitative evaluation of task efficiency and performance, which can be compared with other research groups. It is important to choose clinical functional assessment tests with consideration of constraint in assistive robotic manipulator user interfaces. In addition, the reliable and valid outcome measures will help physicians and therapists build standardized tools while prescribing and assessing assistive robotic manipulators.