Testing the usability of a voice control system for assistive robotic arms in people with neurological conditions

Abstract

The acceptance and effective use of assistive robotic arms by people with neurological conditions strongly depend on the usability of the control interface. This study evaluated the usability of a voice control system for an assistive robotic arm in both healthy participants and individuals with neurological disorders, and compared its applicability and effectiveness with conventional joystick control. A voice control module based on automatic speech recognition and text-to-speech feedback was developed for a commercially available 6-degree-of-freedom assistive robotic arm. The system was implemented using the Robot Operating System (ROS) and an Italian speech-recognition knowledge base. Voice control was tested in 20 healthy subjects and 20 individuals with varying levels of upper-limb impairment due to neurological diseases. Participants performed representative activities of daily living, including (i) pressing an elevator button, (ii) picking up a TV remote control, and (iii) pouring water into a glass. Task completion time, usability, and speech recognition accuracy were assessed and compared between voice and joystick control. All participants except one with neurological impairment were able to successfully use the voice interface, whereas only 11 could operate the joystick. Voice control achieved a recognition accuracy of 87% and a System Usability Scale (SUS) score corresponding to a “Good” adjective rating. Overall, these results indicate that voice control represents a promising and inclusive access modality, with the potential to improve the usability of assistive robotic arms, particularly for individuals with severe motor impairments.

Introduction

The analysis of the Global Burden of Disease 2004 data for the World Report on Disability estimates that approximately 978 million people (15.3% of the world population in 2004) had “moderate or severe disability”, while 2.9% or about 185 million experienced “severe disability” [1]. Robots and assistive technologies are increasingly used to assist people with neurological conditions (PwNC) to improve upper-limb function [2–4]. Assistive devices for the upper limb can play a crucial role in the life of PwNC by enabling them to perform activities of daily living (ADLs) they could not manage without support. Consequently, they can regain autonomy as well as social and personal improvements, such as increased self-esteem, increased social participation, and overall improved quality of life [5]. This creates a positive feedback loop, potentially leading to direct and indirect cost savings for national health services [6].

Assistive devices can be divided into two main categories: (i) those that support the user’s function and (ii) those that substitute the user’s limb in accomplishing tasks. Upper-limb exoskeletons and orthoses that, through different mechanisms, overcome muscle weakness and support arm and hand function belong to the first category. Examples include:

1. The LIGHTarm, a passive gravity-compensated exoskeleton for upper-limb rehabilitation and assistance [7, 8], which was customized to assist people with multiple sclerosis [9];

2. The MyoPro, an upper-limb orthosis that uses volitionally generated weak electromyographic signals from paretic muscles to assist movement of the impaired arm [10];

3. The A-Arm, an experimental active planar arm support controlled by multimodal interfaces for people with Duchenne muscular dystrophy [11].

These examples were selected because they are distinguished by the different control strategies they adopt. In the first case, the exoskeleton is passive and does not rely on an active control strategy; the LIGHTarm supports the weight of the upper arm and forearm through a counterbalance weight and a spring, respectively, thereby facilitating movements against gravity. In the second case, MyoPro exploits residual muscular activity to trigger the motor and assist movement. Both systems are suitable for individuals with moderate to severe impairment who retain minimal motor activity and control capability.

Finally, the A-Arm aims to increase usability by tailoring the control interface to the user’s residual upper-limb motor control capability. Specifically, it is equipped with both force- and EMG-based control interfaces, in which residual EMG activity or estimated volitional force is used to assist movement in severely impaired individuals. The main advantage of this category of assistive devices is that it enables users to continue employing their own arms, thereby enhancing proprioception and reducing the risk of learned non-use.

When no residual motor control capability is present and/or the upper limb cannot be used due to spasticity and fixed postures, one of the few viable solutions is to use systems belonging to the second category. Examples include:

1. The Assistive Robotic Manipulator (ARM), also known as Manus;

2. The robotic manipulator arm JACO (Kinova Technology, Canada).

Manus is a 6-degrees-of-freedom (DOF) robotic arm equipped with a two-finger gripper that enables four-point grasping [6]. The system is mounted on a wheelchair and allows users to perform a variety of ADL tasks at home, at work, and outdoors. Users can operate it via a keypad (sixteen buttons arranged in a 44 grid) or a joystick (e.g. , the wheelchair joystick). Additional control modalities ( e.g. , a laser pointer) can be tailored to the user’s needs.

is a 6-degrees-of-freedom robotic manipulator equipped with a three-finger gripper. It was originally designed for table-mounted use but has more recently been mounted on a wheelchair [12, 13]. Users operate the system through a three-axis joystick (left/right, forward/backward, and twist); the controller also includes five independent push buttons and four external auxiliary inputs.

A common limitation of commercially available assistive robotic manipulators is the difficulty that PwNC experience in operating the system, as control devices often require specific hand dexterity. In recent years, new access methods, such as laser pointers, computer-vision-based interfaces, and voice control, have been proposed to improve accessibility [14]. However, despite these developments, a strong need remains to further increase usability, particularly for users with severe upper-limb impairment (see Sect. Related works).

In this context, we developed a voice control system that enables PwNC to operate robotic manipulators. In this study, we evaluated the functionality and usability of the proposed voice control system in a group of healthy subjects and PwNC while performing ADLs with the JACO robot.

Related works

A key factor for the success of assistive technology is the choice of the control method, which should be tailored to the user’s needs [15] and, more specifically, to their residual functional capabilities. In PwNC, neurological diseases can lead to severe upper-limb impairment, up to the complete inability to control gross movements of the arm, hand, and fingers. However, even when PwNC retain some residual motor capabilities, these may be insufficient to operate traditional keypads, joysticks, and buttons and/or knobs, which are typically integrated into control systems to fully exploit robot functionalities. Therefore, several solutions have been investigated to enable PwNC to operate robotic manipulators without, or only partially using, their upper limbs [16–18].

Early studies were conducted in the early 2000s using a laser pointer to control the 3-DOF motion of a robotic arm [16]. By exploiting voice recognition, the authors implemented a control strategy to switch between position and orientation modes, enabling users to operate a 6-DOF manipulator by changing the control mode via voice commands. Laser-based control was later combined with computer vision, object detection, and grasp-detection algorithms [19, 20]. In [19], users pointed to the object to be grasped using a 5 Hz pulsed laser. The system estimated the target position by comparing consecutive frames acquired with an RGB-D camera. A grasp-detection algorithm then generated candidate gripper configurations and selected the most suitable one based on predefined conditions and a cost function. Although this approach enabled grasping of unknown objects, the system exhibited low usability, as grasp execution could take up to 2 min, potentially causing user frustration. In [20], to overcome the limited real-time performance of laser-based approaches, the authors developed an advanced laser-point detection method combined with an improved pose-estimation algorithm. They implemented and tested the proposed algorithms on a Kinova JACO robotic arm, demonstrating that the grasp-pose generation time could be reduced to less than 5 s. However, the method performed reliably only in indoor environments and was limited to grasping known objects.

In recent years, assistive robotic manipulation has increasingly adopted multimodal and semi-autonomous control paradigms, integrating gaze, EMG, haptics, computer vision, or shared autonomy to reduce user workload and increase accessibility [14, 18]. These approaches aim to improve robustness and reduce dependence on precise voluntary motor control.

Within this context, voice control has emerged as a natural and low-fatigue interaction modality for users with preserved speech abilities. Ka and colleagues developed the Assistive Robotic Manipulation Assistance (ARoMA-V2), a semi-autonomous voice-based control system for operating a robotic arm, combined with computer vision for object recognition [21]. They implemented two sets of commands: (i) direction-based ( e.g. , “move up”, “move down”, “move left”, “move right”) and (ii) task-based ( e.g. , “open hand”, “close hand”). The authors evaluated the setup using a JACO robotic arm [22], but only healthy subjects participated in the experiments. Victores and colleagues developed a voice-control modality within a multimodal interface enabling PwNC to operate an assistive robot [23]. Users interacted with the robot through voice commands (CMU Pocketsphinx and GStreamer packages) and augmented-reality 3D vision glasses. Usability evaluation using the System Usability Scale (SUS) [24] indicated that the system was easy to learn but often required expert assistance, thereby limiting its applicability. No experimental validation was performed with PwNC.

More recently, major advances in cloud-based Automatic Speech Recognition (ASR) services and end-to-end neural speech-recognition models ( e.g. , RNN-T, CTC/attention encoder–decoders, Wav2Vec 2.0, Whisper) have significantly improved recognition accuracy, latency, and robustness in noisy or uncontrolled environments [25]. These developments have renewed interest in voice-based interfaces for assistive manipulation, including continuous or incremental voice-mapping approaches, in which vocal cues modulate robot motion rather than triggering only discrete actions.

In a previous study, we developed a voice control setup and evaluated its usability with two PwNC using a JACO robotic arm [26]. The encouraging results motivated the more comprehensive evaluation presented in this paper.

Aim and outline of the paper

This study assessed the usability of the developed voice-control system in healthy individuals and in people with severe neurological impairments, and compared its applicability and effectiveness with joystick control. In particular, we examined whether the voice-control interface could enable PwNC to operate a robotic arm that they are unable to control using a joystick.

The paper is structured as follows: Section The robotic voice controlsystem briefly presents the developed framework, a detailed description is in [26]; Sect. Usability assessment details the protocol used to assess the usability; results are presented in Sect. Results, in Sect. Discussion, there is the discussion, while the conclusions are drawn in Sect. Conclusions.

The robotic voice control system

This section describes the system setup, the software architecture of the voice-control system, and its functioning in operating the robotic platform.

The setup

The setup consists of a 6-DOF service robot, a laptop, a microphone, and a speaker (Fig. 1). As reported in Subsection The software architecture, we developed the voice-control system within the Robot Operating System (ROS) framework; therefore, it can be used to operate any collaborative robot supporting ROS (e.g. KUKA, Universal Robots, ABB).

Fig. 1.

Robotic voice control system setup

For the experiments, we used the JACO (Gen2) robotic manipulator by Kinova Technology, a commercially available assistive robotic arm designed for individuals with upper-limb mobility impairments. The device is lightweight, portable, compliant with safety standards, and operates at 24 V, making it suitable for home and wheelchair-mounted use. The robot can be placed on a table (e.g. a dining table) or on a wheelchair.

We operated the robotic arm using the standard Kinova JACO joystick controller provided by the manufacturer. This device represents the default commercial control interface typically adopted in clinical and assistive contexts and was therefore selected as the baseline condition for comparison with the proposed voice-control system. We installed the custom-developed software on a laptop, although it can also be installed on a mini-computer for improved portability.

The microphone records the user’s voice commands. Considering the application context, we preferred a wireless Bluetooth microphone. We used a speaker to provide vocal feedback and inform the user of commands in which the robot does not move (e.g. changing mode, Start/Stop). The system is compact, portable, and can also be integrated into a wheelchair.

Voice commands

The voice-control system is built upon a predefined set of commands. Predefined commands were preferred over natural-language input to ensure high robustness, low latency, and predictable system behavior, which are essential when operating an assistive robotic device. Although modern natural-language processing and end-to-end ASR models have advanced considerably in recent years, they still require greater computational resources and are generally less deterministic in noisy or clinical environments.

For this reason, a structured command set was selected. Preliminary tests with healthy users were performed to identify the most reliable and quickly recognized commands. Some actions were intentionally assigned redundant commands to improve ease of interaction and increase usability and task-completion rates. For example, to rotate the robot’s wrist, two commands are available, “turn” or “tilt” (“gira” or “inclina” in Italian, Fig. 2c). Multiple commands allow users who have difficulty pronouncing one command to use an alternative. The set of commands was implemented in both English and Italian.

Fig. 2.

vocal commands for robot manipulation

Two types of voice commands were implemented: direction-based and task-based commands.

Direction-based commands are used to move the robot hand in the 3-D Cartesian space - 6 directions (up, down, right, left, front, and back) and 2 wrist rotations (clockwise and anti-clockwise).

Task-based commands to perform a specific action:

• De/activate voice commands

• Move to predefined positions and poses.

• Finger open/close movements

• Change in wrist displacement length for direction-based commands

The final task-based commands in the above list allow the user to select the displacement length (Fig. 2d); namely, three modes are available: small (1 cm), medium (3 cm), and large (5 cm). These options help reduce the number of command repetitions required to move the robot arm over long distances and provide greater control during object handling.

Depending on the application, predefined positions can be defined to reduce the user’s effort. For the tests presented in this work, we identified four predefined robot configurations, along with the corresponding commands and final poses. For the complete list of commands, refer to Table 1. Figure 2e shows the four predefined configurations/hand positions, which enable users to perform the tasks more rapidly.

Table 1.

Pre-defined vocal commands for robot manipulation

Robot Action	Commands (English)	Commands (Italian)
Direction-based commands
Move the robotic arm	Up, down right, left front, back	Salire, scendere (abbassare) destra (avanti), sinistra (indietro) vicino, lontano
Rotate the wrist	Rotate clockwise Rotate anti-clockwise Long rotate clockwise Long rotate anti-clockwise	Gira piu (inclina piu) Gira meno (inclina meno) Gira piu grande (inclina piu grande) Gira meno grande (inclina meno grande )
Task-based commands
De/Activate	Command, finish	Pronto (parti), termina
Change displacement	Long, medium, small	Grande, mezzo(medio), piccolo
Finger movements	Open, close	Spri mano, chiudi mano
Pre-defined orientation	Top grasp, side grasp	Sopra presa (presa alta), lato presa
Pre-defined position	Home position,first position Second position,third position Face position	Posizione casa, posizione prima Posizione seconda, posizione terza Posizione faccia

The software architecture

The voice-control software was implemented in ROS, an open-source framework for controlling robots and integrated devices. ROS allowed us to develop both the voice-control system and the control of the JACO robotic arm. We selected ROS because it is open source, compatible with collaborative robots from different manufacturers, and supports libraries for controlling all physical components of the system, thereby reducing interoperability issues. We used the Kinetic version of ROS installed on Ubuntu 16.04.

The software architecture (detailed in our previous work [26]) consists of three modules (Fig. 3):

1. The Speech Recognition module converts the voice recorded by the microphone into keywords. It consists of three components: (i) the Front End, (ii) the Knowledge Base, and (iii) the Decoder. The Front End processes the input data and extracts features. The Knowledge Base provides the information required to compare the extracted features. The language model in the Knowledge Base was trained using speech data from healthy Italian-speaking subjects. The Decoder compares the extracted features with the information stored in the Knowledge Base and generates the final output.

2. The Robotic Arm Control Module defines wrist translation and orientation or finger movements, depending on the keyword received from the Speech Recognition module. The ROS JACO arm stack provides the ROS interface for controlling the robotic arm.

3. The Text-to-Speech Module provides vocal feedback to the user, based on a predefined set of text messages.

Fig. 3.

Software architecture of the voice control system

Usability assessment

Participants

A group of people with upper-limb motor impairment due to Muscular Dystrophy (MD) and a healthy control group with no history of upper-limb injury were enrolled in this study.

Inclusion criteria

• Availability of the patient and/or caregiver to sign the informed consent for participation in the study

• Defined diagnosis of MD (Duchenne, Becker, limb girdle type 2, facioscapulohumeral and congenital) based on genetic and/or histopathological and clinical criteria

• Being wheelchair bound

• Weakness in the muscular areas of the shoulder girdle and the proximal portion of the upper limb

• Cognitive skills that allow the understanding and management of the robot

• > 10 years.

Exclusion criteria

• Presence of significant comorbidities (epilepsy, dependence 24/24 hours on non-invasive or invasive ventilotherapy)

• Behavioral and psychiatric disorders (e.g. , emotional problems, depression, etc.)

Outline of the study

The voice-control usability assessment was based on two different evaluations: (i) objective tests in which each subject operated the JACO robotic arm using either voice control or the joystick (see Sect. The voice-control testingprotocol); and (ii) the subject’s opinion, collected through a set of questions and a survey (see Sect. Usability assessment measures).

The voice-control testing protocol

Participants had to perform several activities of daily living tasks with the JACO robotic arm using both voice control and the joystick. Half of the participants first performed the tasks using voice control, while the other half first used the joystick to avoid order-related biases. Participants were randomly assigned to one of the two groups.

Before starting the experiment, each subject received instructions on how to control the robot using both the JACO2 joystick and the developed voice-control system. Specifically, the operation of the voice control (i.e. the available command set) was described in detail. After completing the training, participants were asked to perform unstructured tasks (i.e. grasp objects of different shapes and sizes arranged randomly on the table and move them closer or stack them). Once they demonstrated confidence with both control systems, testing began, and participants were asked to perform three specific tasks (see Sect. The tasks). All subjects performed the tasks in the same order, with increasing difficulty.

When using voice control, participants could exploit task-based commands (e.g. predefined positions and orientations) to facilitate task execution. In contrast, the joystick interface was intentionally used in its standard commercial configuration, without additional task-specific mappings, to reflect typical real-world usage of the system. Task execution was video-recorded, and task completion times were measured for comparison.

The study was approved by the Ethics Committee of IRCCS Medea (CE Nr. 73/18, approval date 16/07/2018). All participants, or their parents/legal guardians in the case of minors, signed a written informed consent. The study was registered on clinicaltrials.gov (NCT04313049).

The tasks

The experiment consisted of three tasks, which were selected based on the results of a survey administered to PwNC [26]:

1. Task1, push an elevator button: starting from the home position, the user had to reach the button (a circle drawn on a displaceable polystyrene pad) and push it (i.e. touch and displace the padding) (see Fig. 4).

2. Task2, take a TV remote: starting from the home position, the user had to reach and grasp a TV remote controller and place it on the table (see Fig. 4).

3. Task3, pour/drink water: starting from the home position, the user had to reach and grasp a bottle, move it close to a glass, pour the content into the glass, grasp the glass, and bring it in front of the mouth (see Fig. 4).

When using the voice control, participants could exploit task-based commands (e.g., predefined positions and orientations) to facilitate task execution. In contrast, the joystick interface was intentionally used in its standard commercial configuration, without additional task-specific mappings, to reflect typical real-world usage of the system.

Fig. 4.

Tasks performed during the experiment; all tasks start from the home position (faded frame).Task1: move the robot to the "elevator button" (2nd frame) and push it; Task2: move the robot to the remote control, grasp it (2nd frame) and put it on the table; Task3: move the robot to the bottle, grasp it, pour "water" in the glass, take the glass in front of the mouth

Functional assessment

Voice-control recognition accuracy was assessed by counting, in post-processing, the percentage of errors over the total number of commands. Two types of errors were considered: (i) the robot did not react to a command; (ii) the robot reacted to a command by performing an incorrect action. Regarding error (i), cases in which the robot did not react due to robot constraints were not counted as errors, as they were not attributable to a malfunction of the voice-control system.

Usability assessment measures

User assessment of voice-control usability relied on three steps:

• Administration of a validated survey, i.e., the SUS [27];

• Administration of context-based questions;

• Conversion of SUS results into an adjective scale [28].

The SUS score

SUS is a ten-item Likert scale in which respondents indicate their degree of agreement or disagreement with a set of statements on a 5-point scale. It consists of ten items, providing a global view of the subjective assessment of usability. Positive and negative items are alternated to prevent response biases. Respondents must answer all items. If a respondent feels unable to respond to a particular item, the neutral score of 3 is assigned. Each item score contribution ranges from 0 to 4. For items 1, 3, 5, 7, and 9, the score contribution is the scale position minus 1. For items 2, 4, 6, 8, and 10, the contribution is 5 minus the scale position. The sum of the scores is then multiplied by 2.5 to obtain the overall SUS score. The final score ranges from 0 to 100 [24].

Ad-hoc questionnaire

Some questions were prepared to collect additional qualitative information, which, together with the SUS score, may help understand the system’s usefulness and guide further development. Specifically, participants were asked: “On a scale from 1 to 5, how likely are you to use the system for: 1) eating, 2) scratching yourself, 3) drinking with a straw (i.e.bringing the glass to the mouth to drink with a straw), 4) moving your own arm, 5) answering the phone, 6) ADLs at home, and 7) activities at work?”

Dependent measures

The primary outcome measures for intragroup comparison were the difference in the percentage of completed tasks and execution times using the two control systems. The secondary outcome measure for intragroup comparison was the difference in the SUS scores using the two control systems.

Data analysis

Statistical analysis was performed using IBM SPSS Statistics 21.0. Baseline differences between healthy participants and patients were assessed using the Mann–Whitney U test for age and the chi-square test for gender.

Within-group comparisons among the three tasks (Task1, Task2, Task3) were first assessed using the Friedman test. When a significant effect was detected, post hoc pairwise comparisons were performed using Wilcoxon matched-pairs signed-rank tests. In this case, a Bonferroni correction was applied to control for multiple comparisons within each family of tests (three pairwise comparisons: Task1 vs Task2, Task1 vs Task3, Task2 vs Task3), resulting in a corrected significance threshold of .

Planned comparisons between control systems (joystick vs. voice) within groups were performed using the Wilcoxon matched-pairs signed-rank test. Comparisons of task performance between groups were carried out using the Mann–Whitney U test.

The nominal -error significance level was set to 0.05.

Results

We recruited 20 healthy participants (12 males, median age 32 (12) years) and 20 participants with MD (16 males, median age 27 (8) years). The two groups are matched by gender (chi-square test: , ) but not by age (Mann–Whitney U test: , ). The demographic data are reported in Table 2.

Table 2.

demographic data of participants

	HP	PMD	p-value
Male/Female	12/8	16/4	.135
Age (years)	32(12)*	27(8)*	.043
Diagnosis
(CMD/ DMD/	–	3/10/5/1/1	–
LGMD/ FSHD/ D)

p-values in bold are statistically significant

HP: healthy participants; PMD: participants with MD

CMD: congenital MD; DMD: Duchenne MD

LGMD: Limb-girdle MD; D: Dystrophynopaty

FSHD: Facioscapulohumeral MD dystrophy

* Median and interquartile range are reported

Rate of success and task durations

Table 3 shows the execution times of the 3 tasks performed by individuals in the two groups.

Table 3.

Times of task execution in seconds of healthy subjects and patients

Healthy subject group							Muscular dystrophy group
	Joystick			Voice				Joystick			Voice
ID	Task1: Push Button	Task2: Collect Object	Task3: Drink Glass	Task1: Push Button	Task2: Collect Object	Task3: Drink Glass	ID	Task1: Push Button	Task2: Collect Object	Task3: Drink Glass	Task1: Push Button	Task2: Collect Object	Task3: Drink Glass
HS01(M)	35	59	209	64	66	234	MD01(F)	43	170	355	92	118	321
HS02(M)	29	85	189	96	109	284	MD02(M)	–	–	–	97	114	176
HS03(M)	28	60	187	52	78	291	MD03(F)	40	–	378	83	80	259
HS04(M)	37	101	243	89	64	321	MD04(M)	29	159	320	65	99	364
HS05(M)	19	63	166	38	98	322	MD05(F)	50	298	500	–	–	–
HS06(M)	27	51	217	90	74	348	MD06(M)	–	–	–	197	150	419
HS07(M)	20	48	131	61	104	247	MD07(M)	11	126	280	162	92	314
HS08(M)	35	54	385	80	63	456	MD08(M)	–	–	–	70	101	292
HS09(M)	50	67	200	59	75	276	MD09(M)	–	–	–	49	78	262
HS10(F)	32	78	327	56	-	279	MD10(M)	65	120	228	101	107	222
HS11(F)	51	83	-*	70	61	268	MD11(M)	105	243	432	82	123	482
HS12(F)	55	122	213	48	121	284	MD12(M)	–	–	–	160	127	404
HS13(F)	67	126	467	73	137	459	MD13(M)	20	130	211	56	63	248
HS14(M)	51	119	219	82	69	328	MD14(M)	20	268	298	167	235	204
HS15(F)	47	187	364	91	82	475	MD15(F)	22	232	376	81	125	344
HS16(M)	38	149	217	106	83	506	MD16(M)		–	–	44	–	300
HS17(M)	23	150	278	195	–	267	MD17(M)	–	–	–	69	108	295
HS18(F)	48	92	241	82	122	507	MD18(M)	–	–	–	68	124	305
HS19(F)	59	108	270	78	111	350	MD19(M)	34	237	243	64	146	226
HS20(F)	22	73	265	67	89	298	MD20(M)	–	–	–	72	121	312
med.	36	84	219	76	83	310	med.	34	201	320	81	116	300
iqr	(23)	(58)	(70)	(29)	(38)	(98)	iqr	(26)	(104)	(116)	(33)	(25)	(79)
Nr.Acc.Tasks	20	20	20	20	18	20	Nr.Acc.Tasks	11	10	11	19	18	19
(% Acc.Tasks)	(100%)	(100%)	(100%)	(100%)	(90%)	(100%)	(% Acc.Tasks)	(55%)	(50%)	(55%)	(95%)	(90%)	(95%)

Nr. and % Acc. Tasks number and percentage of accomplished tasks. Times are in seconds

*Subject HS11 accomplished task3, but there was a problem with recording and, therefore, it was not possible to time the task

(F) Female; (M) Male

In the healthy subject group, the rate of individuals who completed the tasks is for all tasks when using the joystick; HS11 completed Task3, but the execution time is not available because the video recording failed. The rate of healthy subjects who completed the tasks using voice control is greater than or equal to . Two individuals could not complete Task2 using voice control: a female participant (HS10), who did not show any problems with word articulation, and a male participant (HS17), who showed difficulties in articulating several words.

In the MD group, the rate of individuals who completed the tasks is less than or equal to for all tasks when using the joystick (9 individuals could not complete Task1 and Task3, and 10 could not complete Task2) and greater than or equal to when using voice control (1 female participant (MD5) could not complete the three tasks, and 1 male participant (MD16) could not complete Task2).

Table 4 shows the median execution time (and interquartile range) for each task using the joystick and voice control in the healthy and MD groups. In the healthy group, the durations of the three tasks are statistically different when using the joystick and voice control (Friedman test, ). Specifically, the duration of Task3 is the longest and is statistically different from that of Task1 and Task2 (Wilcoxon test, for both control systems). Conversely, the duration of Task1 is statistically significantly shorter than that of Task2 when using the joystick (Wilcoxon test, ), whereas the two tasks show comparable durations when using voice control.

Table 4.

Comparison of task durations between groups and control systems

	Joystick control(J)			p-val	Voice control(V)			p-val	p-val J vs. V
	Task1	Task2	Task3		Task1	Task2	Task3		Task1	Task2	Task3
HP			219(70)				310(98)			.616
PMD			320(116)				300(79)				.838
	.605				.365		.270

p-values of the Friedman test among the three tasks within groups

statistically significant difference at the post hoc test, Wilcoxon with Bonferroni correction, between Task1 and Task3 duration and between  Task2 and Task3 duration. p-values of post-hoc tests are in the text

statistically significant difference at the post-hoc test, Wilcoxon with Bonferroni correction, between Task1 and Task2 duration.  p-values of post-hoc tests are in the text

p-values of the Wilcoxon test (between control systems analysis)

p-values of the Mann-Whitney test (between groups analysis)

Participants in the MD group who could accomplish the tasks show similar statistical results. The Friedman test highlights significant differences among the three tasks () for both control systems. The duration of Task3 is the longest and is statistically different from that of Task1 ( using the joystick and using voice control) and Task2 (Wilcoxon test, using the joystick and using voice control). Unlike the healthy group, Task1 and Task2 execution times are comparable in the MD group for both control systems.

When comparing the two control systems, results show that, in the healthy group, the durations of Task1 and Task3 are significantly shorter when using the joystick (Wilcoxon test, for both tasks), whereas the duration of Task2 is comparable between the two control systems. Similarly, participants with MD who could accomplish the tasks perform Task1 faster using the joystick (Wilcoxon test, ) but perform Task2 slower when using the joystick (Wilcoxon test, ). No statistically significant differences are observed in the execution time of Task3 between the two control systems.

When comparing the two groups, healthy participants are statistically significantly faster than participants with MD in performing Task3 using the joystick and Task2 using both control systems. In all other cases–Task1 using both control systems and Task3 using voice control–the two groups show comparable execution times.

The voice control recognition accuracy

Considering the performance of voice control in participants with MD, it worked correctly in 87% of cases, showing good to high recognition accuracy. The few malfunctions were mainly due to cases in which “the robot did not react after a voice command” (10%), while only in 1% of cases “the robot performed the wrong action” after a voice command. Furthermore, cases in which “the robot did not respond due to robot constraints” occurred in 2% of cases.

The voice control usability

Considering the usability assessment (see Table 5), SUS is generally higher for voice control than for the joystick, particularly in participants with MD, for whom usability scores show a statistically significant difference. With regard to voice control, SUS indicates “Good” usability (Adjective Rating [28]) in participants with MD and is statistically significantly higher compared to healthy subjects.

Table 5.

Comparison of the system usability between control systems and groups

	SUS	SUS	p-value
HP	67.5(18.8)	71.3(15)	.061
PMD	57.5(16.3)	82.5(16.3)	.032
p-value	.131	.035

p-values in bold are statistically significant

SUS is the score of SUS related to joystick control

SUS is related to voice control

HP: Healthy Participants

PMD: Participants with Muscular Dystrophy

p-values of Wilcoxon test between control systems

p-values of the Mann-Whitney test between groups

These results refer to participants who could accomplish at least 1 of the 3 tasks: participants with MD who completed at least one task were 11 when using the joystick and 19 when using voice control. Overall, participants with MD completed 32 tasks out of 60 (20 subjects 3 tasks), corresponding to 53% of the tasks on average, when using the joystick, whereas they completed 56 tasks out of 60, corresponding to 93% of the tasks on average, when using voice control (see Table 3).

Regarding the ad-hoc questionnaire, results show that the preferred interface is voice control, with 14 preferences among 20 participants with MD. Considering the question “how much could you use the robot for the following activities”, participants with MD report a median score of 3 (1) out of 5 points, averaged across the 7 activities. The most preferred activities to be performed with the robot are “drinking with a straw” and “moving the arm and performing activities of daily living at home”, with a median value of 4 out of 5 points. The activities that participants believe the robot is less useful for are “eating” and “answering the phone”, with a median value of 2 out of 5 points.

Discussion

The aim of this study was to compare the performance and usability of voice control versus joystick control in both healthy participants and individuals with MD. The results revealed several important findings.

First, the rate of success in completing the tasks differed between the two groups and between the control systems. In the healthy participant group, the completion rate was consistently high for all tasks when using the joystick, while it was slightly lower but still above 90% when using voice control. In contrast, the MD group showed a significantly lower completion rate when using the joystick, with only around 55% of individuals able to complete the tasks. However, when using voice control, the completion rate increased to above 90%. These findings suggest that voice control can be more effective for individuals with MD, potentially compensating for their motor limitations, compared to joystick control.

Regarding task-execution duration, it varied across tasks, control systems, and participant groups. The most relevant results concern the comparison between the two control systems. Specifically, healthy participants completed Task1 and Task3 significantly faster using the joystick compared to voice control, whereas Task2 durations were comparable between the two control systems. Among participants with MD who completed the tasks, Task1 was performed faster using the joystick, while Task2 was completed more slowly using the joystick. Task3 showed no significant differences in execution time between the two control systems. These results suggest that joystick control may offer advantages in terms of execution speed for certain tasks in both healthy participants and individuals with MD who are able to complete these tasks, whereas voice control may be more advantageous for other tasks in individuals with MD.

Overall, for people with adequate motor control, the joystick allows higher execution speeds than voice control with the available set of voice commands. However, execution speed in tasks requiring fine positioning or longer motion sequences could be improved by expanding the available voice commands and, more critically, by increasing the number of predefined robot orientations and end-effector positions accessible through voice control.

The voice-control system demonstrated good to high recognition accuracy, with an overall success rate of 87% in participants with MD. Most malfunctions were attributable to cases in which the robot did not respond after a voice command, while instances in which the robot performed an incorrect action were rare. These findings indicate that the voice-control system was generally reliable and effective in understanding and executing commands. It is also worth noting that the speech-recognition module used in this study is based on a traditional ASR pipeline (Front End, Knowledge Base, Decoder) and a predefined vocabulary. Despite not relying on modern end-to-end ASR architectures, which currently provide higher robustness and noise tolerance [25], the system still achieved high accuracy in real functional tasks. This reinforces the value of the presented results, as more advanced ASR technologies could reasonably yield even better performance in the future, particularly in noisy environments or in users with speech alterations. Another important aspect is that the entire ASR pipeline runs completely offline, without requiring internet connectivity. This ensures reliability, privacy, and consistent performance in clinical or home settings where network conditions may be unreliable.

Usability assessment, as measured by the System Usability Scale (SUS), indicated that participants–particularly those with MD–found the voice-control system more usable than the joystick. SUS scores were significantly higher for voice control in the MD group compared to healthy participants. This suggests that individuals with MD who could use both control modalities perceived the voice-control system as more intuitive and user-friendly, which is relevant for enhancing overall user experience and acceptance of robotic assistance.

In addition, and importantly, it should be recalled that 9 subjects out of 20 could not use the joystick at all, and one additional subject could complete only two tasks. Although the specific causes underlying the inability to use the joystick were not formally quantified during task execution, they were consistently confirmed through direct observation during the experimental sessions and video recordings. In all observed cases, difficulties in using the joystick were attributable to severe upper-limb motor impairments, such as insufficient residual strength, limited range of motion, or reduced motor control, rather than to cognitive factors. Indeed, all participants met the cognitive inclusion criteria and were able to understand the experimental procedures and task instructions.

This observation further indicates that joystick-based interfaces, even in their standard commercial form, may remain inaccessible to users with very severe motor impairments despite preserved cognitive abilities. These findings clearly demonstrate that voice control increases usability compared to joystick control. For users with limited or absent upper-limb strength, voice commands offer a low-fatigue interaction modality, unlike joysticks or simplified manual controllers that still require some degree of motor control.

Finally, the ad-hoc questionnaire confirmed the usability results, revealing a preference for the voice-control interface, with a majority of participants with MD (14 out of 20) selecting it as their preferred method. Participants with MD rated the question “how much could you use the robot for the following activities” with a median score of 3 (1) out of 5 points, averaged across the 7 activities, indicating a positive perception of its potential usefulness. The most preferred activities included drinking with a straw and performing daily life activities at home, whereas activities such as eating and answering the phone were perceived as less useful. Overall, participants with MD reported that they liked the voice-control system and perceived that this assistive technology could enhance their quality of life.

MD is characterized by progressive muscular weakness, eventually leading to loss of ambulation and impaired arm function. Considering current life expectancy, people with MD may live with impaired upper-limb function for an extended period [29], during which they may experience substantial limitations in ADLs and social participation if not adequately supported. Therefore, the system presented in this work represents a potential solution to improve the quality of life of these patients.

The added value of voice control compared to other solutions, such as modified or simplified joysticks (e.g. [30]), lies in its independence from the use of the impaired limb, making it suitable even for individuals with very severe upper-limb impairments. Another advantage compared to other control systems (e.g. gaze-based control [31] or tongue-based interfaces [32]) is that voice control offers a natural interaction modality. However, future studies should compare the usability of voice control with other access methods in individuals with different neurological conditions.

One limitation of this work is that tests were limited to participants with MD, and no other neurological conditions were included in the study. However, it is clear that individuals with very severe upper-limb impairment, or even paralysis, as in the case of spinal cord injury, could not use the joystick. For these individuals, voice control is undoubtedly a viable option. Future work should therefore verify the actual usability of the voice-control system and the potential usefulness of the robotic arm in groups of individuals with different neurological conditions.

An additional limitation of this study is that, although the healthy and MD groups were matched for gender, the comparison between control modalities (voice vs. joystick) involved subgroups with different gender distributions. Although gender imbalance was not an aspect specifically addressed in the design of the present study, its potential impact on interaction with assistive technologies cannot be excluded. In the present sample, one healthy female participant was unable to complete Task2 using voice control, and one female participant with MD was unable to complete any of the tasks. Although these isolated cases do not allow generalization, they highlight the need to further investigate whether gender-related factors (e.g., vocal characteristics affecting speech recognition) may contribute to performance variability.

The gender distribution in the MD group was markedly skewed toward male participants, reflecting the epidemiology of the underlying conditions, which predominantly affect male individuals. Such an imbalance may also have contributed to more favorable usability outcomes for the voice-control modality, particularly if speech-recognition performance is not equivalent across genders. The voice-control system adopted in this study was designed as a unisex interface. However, even if the observed gender imbalance had contributed to more favorable outcomes for voice control–potentially due to differences in speech-recognition performance across genders–this effect would likely reflect characteristics of the specific voice-recognition system adopted in this study, rather than constituting an inherent limitation of voice-based interaction as a control modality. In this context, future developments could explore training strategies explicitly aimed at improving recognition performance for female voices, for example by including more balanced datasets or female-focused training approaches, potentially moving beyond a strictly unisex recognition model, although the extent of such improvements cannot be determined based on the present data.

A further consideration is that the joystick-based condition relied on the standard commercially available Kinova JACO2 joystick controller. While alternative joystick form factors (e.g., thumb-controlled or wheelchair-integrated microjoysticks) may require less proximal upper-limb movement and could lead to different usability and performance outcomes, the adopted joystick represents the default baseline interface typically available in clinical and assistive deployments of the Kinova platform. Overall, the present findings should be interpreted accordingly.

Another limitation is that participants were not formally screened for their level of technological proficiency or prior experience with joystick-based or voice-controlled interfaces. While baseline familiarity with joystick-like controls is likely in wheelchair users, individual proficiency may vary and was not quantified in the present study. This factor may have contributed to inter-subject variability and should be considered when interpreting comparisons between control modalities.

Finally, a limitation of this study is that task-based commands (e.g., moving the robot to predefined positions) were available only in the voice-control condition, resulting in an asymmetry between control modalities. However, this reflects an intrinsic characteristic of voice-based interaction, in which verbal commands can naturally serve as shortcuts to complex actions. Implementing equivalent shortcuts in the joystick condition would require additional physical interfaces (e.g., buttons, switches, or mode selectors). This was not supported by the commercial joystick used in this study and would be challenging even with a custom-designed joystick, as it would substantially increase interface complexity and cognitive load. For these reasons, the joystick was intentionally used in its standard configuration. In this setup, the manufacturer-provided return-to-Home function was the only shortcut available in the joystick condition. This design choice may have contributed to the observed differences in task efficiency and should be considered when interpreting the results.

Conclusions

This study investigated the performance and usability of voice control versus joystick control in healthy participants and individuals with MD. The findings provide valuable insights into the effectiveness and user acceptance of the two control systems. In particular, voice control demonstrated high recognition accuracy, with few malfunctions or incorrect actions. Despite being based on a traditional ASR pipeline with a predefined vocabulary, the system showed reliable performance in real functional tasks and appeared suitable for individuals with limited or absent upper-limb motor control. The usability analysis confirmed that the voice-control system was generally more intuitive and accessible, especially for participants with MD, many of whom could not operate the joystick.

Participants with MD expressed a clear preference for the voice interface, indicating a positive user experience and acceptance of robotic assistance. Overall, this study emphasizes the potential benefits of voice control as a highly accessible interaction modality for individuals with severe motor impairments. The results further support the consideration and integration of voice interfaces into assistive robotic technologies to enhance usability, promote independence, and improve participation in activities of daily living.

Future research should focus on evaluating the system in home environments to assess long-term usability, ecological validity, and actual usefulness in everyday life. Additional work could explore adaptation of the system to other neurological conditions and investigate the integration of more advanced ASR technologies, which may improve robustness in noisy environments and further enhance user performance. Expanding the range of tasks, refining the command set, and comparing voice control with other alternative interfaces would also contribute to improving the versatility and effectiveness of assistive robotic solutions.

Author contributions

MC and EB conceived and supervised the overall project. TP developed the software framework, implemented the voice-control system, performed the preliminary technical tests, and analysed the technical results. EB coordinated the interaction with the hospital and clinical team, supervised the experimental activities, and drafted the clinical sections, including the description and interpretation of clinical results. MGD supervised the study clinically and handled patient selection and recruitment. ED carried out the experimental sessions, administered the usability assessments, and collected the data. MC integrated all contributions and prepared the consolidated manuscript draft. All authors contributed to data interpretation, revised the manuscript critically, and approved the final version.

Funding

This research was partially financially supported by the Italian Lombardy Region within the "Future Home For Future Communities /FHfFC" project (Third Framework Agreement between the CNR and the Lombardy Region), by Fondazione Cariplo within the "EMpowerment del PAzienTe In cAsa (EMPATIA@Lecco) project (Emblematici Cariplo Grants), and by the Italian Ministry of Health (RC 2025/2026 to E. Biffi), and by the Italian Ministry for Universities and Research (MUR) under the FIT4MEDROB grant (PNC0000007).

Data availability

Data supporting the findings of this study are available from the corresponding author upon reasonable request

Declarations

Ethics approval and consent to participate

The study was approved by the Ethics Committee of the IRCCS Medea (CE Nr. 73/18, approval date 16/07/2018). All the participants or their parents/legal guardians, in case of minors, signed a written informed consent.

Consent for publication

All participants gave their written consent.

Competing interests

The authors declare no competing interests.